Jul 21 20:58:42.078577 ip-10-0-140-97 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory Jul 21 20:58:42.078589 ip-10-0-140-97 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory Jul 21 20:58:42.078595 ip-10-0-140-97 systemd[1]: kubelet.service: Failed with result 'resources'. Jul 21 20:58:42.078801 ip-10-0-140-97 systemd[1]: Failed to start Kubernetes Kubelet. Jul 21 20:58:52.125465 ip-10-0-140-97 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found. Jul 21 20:58:52.125480 ip-10-0-140-97 systemd[1]: kubelet.service: Failed with result 'resources'. -- Boot 4ec9e206aa074276b1a6a4fbfeb6f7cf -- Jul 21 21:00:41.960819 ip-10-0-140-97 systemd[1]: Starting Kubernetes Kubelet... Jul 21 21:00:42.347497 ip-10-0-140-97 kubenswrapper[2575]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 21 21:00:42.347497 ip-10-0-140-97 kubenswrapper[2575]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Jul 21 21:00:42.347497 ip-10-0-140-97 kubenswrapper[2575]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 21 21:00:42.347497 ip-10-0-140-97 kubenswrapper[2575]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Jul 21 21:00:42.347497 ip-10-0-140-97 kubenswrapper[2575]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 21 21:00:42.347497 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:42.322106 2575 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Jul 21 21:00:42.347497 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:00:42.328919 2575 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 21 21:00:42.347497 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:00:42.328932 2575 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 21 21:00:42.347497 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:00:42.328936 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 21 21:00:42.347497 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:00:42.328939 2575 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 21 21:00:42.347497 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:00:42.328942 2575 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 21 21:00:42.347497 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:00:42.328945 2575 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 21 21:00:42.347497 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:00:42.328949 2575 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 21 21:00:42.347497 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:00:42.328953 2575 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 21 21:00:42.348969 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:00:42.328956 2575 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 21 21:00:42.348969 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:00:42.328959 2575 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 21 21:00:42.348969 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:00:42.328961 2575 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 21 21:00:42.348969 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:00:42.328964 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 21 21:00:42.348969 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:00:42.328966 2575 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 21 21:00:42.348969 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:00:42.328969 2575 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 21 21:00:42.348969 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:00:42.328972 2575 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 21 21:00:42.348969 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:00:42.328974 2575 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 21 21:00:42.348969 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:00:42.328977 2575 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 21 21:00:42.348969 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:00:42.328979 2575 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 21 21:00:42.348969 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:00:42.328982 2575 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 21 21:00:42.348969 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:00:42.328984 2575 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 21 21:00:42.348969 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:00:42.328987 2575 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 21 21:00:42.348969 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:00:42.328990 2575 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 21 21:00:42.348969 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:00:42.328992 2575 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 21 21:00:42.348969 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:00:42.328995 2575 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 21 21:00:42.348969 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:00:42.328997 2575 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 21 21:00:42.348969 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:00:42.329000 2575 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 21 21:00:42.348969 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:00:42.329003 2575 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 21 21:00:42.348969 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:00:42.329005 2575 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 21 21:00:42.353978 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:00:42.329008 2575 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 21 21:00:42.353978 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:00:42.329010 2575 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 21 21:00:42.353978 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:00:42.329012 2575 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 21 21:00:42.353978 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:00:42.329016 2575 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 21 21:00:42.353978 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:00:42.329018 2575 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 21 21:00:42.353978 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:00:42.329021 2575 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 21 21:00:42.353978 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:00:42.329024 2575 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 21 21:00:42.353978 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:00:42.329027 2575 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 21 21:00:42.353978 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:00:42.329030 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 21 21:00:42.353978 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:00:42.329032 2575 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 21 21:00:42.353978 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:00:42.329035 2575 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 21 21:00:42.353978 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:00:42.329037 2575 feature_gate.go:328] unrecognized feature gate: Example2 Jul 21 21:00:42.353978 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:00:42.329040 2575 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 21 21:00:42.353978 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:00:42.329042 2575 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 21 21:00:42.353978 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:00:42.329045 2575 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 21 21:00:42.353978 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:00:42.329047 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 21 21:00:42.353978 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:00:42.329050 2575 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 21 21:00:42.353978 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:00:42.329052 2575 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 21 21:00:42.353978 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:00:42.329055 2575 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 21 21:00:42.353978 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:00:42.329057 2575 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 21 21:00:42.355004 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:00:42.329060 2575 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 21 21:00:42.355004 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:00:42.329062 2575 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 21 21:00:42.355004 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:00:42.329065 2575 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 21 21:00:42.355004 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:00:42.329068 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 21 21:00:42.355004 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:00:42.329070 2575 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 21 21:00:42.355004 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:00:42.329073 2575 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 21 21:00:42.355004 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:00:42.329075 2575 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 21 21:00:42.355004 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:00:42.329078 2575 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 21 21:00:42.355004 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:00:42.329080 2575 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 21 21:00:42.355004 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:00:42.329083 2575 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 21 21:00:42.355004 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:00:42.329085 2575 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 21 21:00:42.355004 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:00:42.329088 2575 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 21 21:00:42.355004 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:00:42.329090 2575 feature_gate.go:328] unrecognized feature gate: Example Jul 21 21:00:42.355004 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:00:42.329093 2575 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 21 21:00:42.355004 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:00:42.329095 2575 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 21 21:00:42.355004 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:00:42.329098 2575 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 21 21:00:42.355004 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:00:42.329101 2575 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 21 21:00:42.355004 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:00:42.329104 2575 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 21 21:00:42.355004 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:00:42.329107 2575 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 21 21:00:42.355004 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:00:42.329109 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 21 21:00:42.356018 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:00:42.329112 2575 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 21 21:00:42.356018 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:00:42.329114 2575 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 21 21:00:42.356018 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:00:42.329117 2575 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 21 21:00:42.356018 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:00:42.329119 2575 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 21 21:00:42.356018 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:00:42.329121 2575 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 21 21:00:42.356018 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:00:42.329124 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 21 21:00:42.356018 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:00:42.329126 2575 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 21 21:00:42.356018 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:00:42.329128 2575 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 21 21:00:42.356018 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:00:42.329131 2575 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 21 21:00:42.356018 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:00:42.329133 2575 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 21 21:00:42.356018 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:00:42.329136 2575 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 21 21:00:42.356018 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:00:42.329138 2575 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 21 21:00:42.356018 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:00:42.329141 2575 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 21 21:00:42.356018 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:00:42.329143 2575 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 21 21:00:42.356018 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:00:42.329146 2575 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 21 21:00:42.356018 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:00:42.329151 2575 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 21 21:00:42.356018 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:00:42.329154 2575 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 21 21:00:42.356018 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:00:42.329157 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 21 21:00:42.356018 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:00:42.329160 2575 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 21 21:00:42.356018 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:00:42.329162 2575 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 21 21:00:42.358814 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:00:42.329165 2575 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 21 21:00:42.358814 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:00:42.329169 2575 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 21 21:00:42.358814 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:00:42.329171 2575 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 21 21:00:42.358814 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:00:42.329175 2575 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 21 21:00:42.358814 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:00:42.329178 2575 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 21 21:00:42.358814 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:00:42.329182 2575 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 21 21:00:42.358814 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:00:42.329184 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 21 21:00:42.358814 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:00:42.329188 2575 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 21 21:00:42.358814 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:00:42.329192 2575 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 21 21:00:42.358814 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:00:42.329195 2575 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 21 21:00:42.358814 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:00:42.329198 2575 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 21 21:00:42.358814 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:00:42.329917 2575 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 21 21:00:42.358814 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:00:42.329925 2575 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 21 21:00:42.358814 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:00:42.329939 2575 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 21 21:00:42.358814 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:00:42.329942 2575 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 21 21:00:42.358814 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:00:42.329946 2575 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 21 21:00:42.358814 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:00:42.329949 2575 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 21 21:00:42.358814 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:00:42.329952 2575 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 21 21:00:42.358814 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:00:42.329955 2575 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 21 21:00:42.359826 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:00:42.329958 2575 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 21 21:00:42.359826 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:00:42.329961 2575 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 21 21:00:42.359826 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:00:42.329964 2575 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 21 21:00:42.359826 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:00:42.329967 2575 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 21 21:00:42.359826 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:00:42.329969 2575 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 21 21:00:42.359826 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:00:42.329972 2575 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 21 21:00:42.359826 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:00:42.329975 2575 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 21 21:00:42.359826 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:00:42.329978 2575 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 21 21:00:42.359826 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:00:42.329980 2575 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 21 21:00:42.359826 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:00:42.329984 2575 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 21 21:00:42.359826 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:00:42.329988 2575 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 21 21:00:42.359826 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:00:42.329991 2575 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 21 21:00:42.359826 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:00:42.329994 2575 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 21 21:00:42.359826 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:00:42.329997 2575 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 21 21:00:42.359826 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:00:42.330000 2575 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 21 21:00:42.359826 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:00:42.330003 2575 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 21 21:00:42.359826 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:00:42.330005 2575 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 21 21:00:42.359826 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:00:42.330009 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 21 21:00:42.359826 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:00:42.330011 2575 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 21 21:00:42.363505 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:00:42.330014 2575 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 21 21:00:42.363505 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:00:42.330016 2575 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 21 21:00:42.363505 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:00:42.330019 2575 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 21 21:00:42.363505 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:00:42.330021 2575 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 21 21:00:42.363505 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:00:42.330024 2575 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 21 21:00:42.363505 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:00:42.330028 2575 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 21 21:00:42.363505 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:00:42.330032 2575 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 21 21:00:42.363505 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:00:42.330035 2575 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 21 21:00:42.363505 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:00:42.330039 2575 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 21 21:00:42.363505 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:00:42.330042 2575 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 21 21:00:42.363505 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:00:42.330044 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 21 21:00:42.363505 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:00:42.330047 2575 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 21 21:00:42.363505 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:00:42.330050 2575 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 21 21:00:42.363505 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:00:42.330053 2575 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 21 21:00:42.363505 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:00:42.330055 2575 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 21 21:00:42.363505 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:00:42.330058 2575 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 21 21:00:42.363505 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:00:42.330061 2575 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 21 21:00:42.363505 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:00:42.330063 2575 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 21 21:00:42.363505 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:00:42.330066 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 21 21:00:42.364553 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:00:42.330070 2575 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 21 21:00:42.364553 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:00:42.330073 2575 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 21 21:00:42.364553 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:00:42.330076 2575 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 21 21:00:42.364553 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:00:42.330079 2575 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 21 21:00:42.364553 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:00:42.330083 2575 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 21 21:00:42.364553 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:00:42.330086 2575 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 21 21:00:42.364553 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:00:42.330089 2575 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 21 21:00:42.364553 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:00:42.330092 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 21 21:00:42.364553 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:00:42.330095 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 21 21:00:42.364553 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:00:42.330097 2575 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 21 21:00:42.364553 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:00:42.330100 2575 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 21 21:00:42.364553 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:00:42.330102 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 21 21:00:42.364553 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:00:42.330105 2575 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 21 21:00:42.364553 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:00:42.330107 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 21 21:00:42.364553 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:00:42.330110 2575 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 21 21:00:42.364553 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:00:42.330113 2575 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 21 21:00:42.364553 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:00:42.330118 2575 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 21 21:00:42.364553 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:00:42.330121 2575 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 21 21:00:42.364553 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:00:42.330123 2575 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 21 21:00:42.364553 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:00:42.330126 2575 feature_gate.go:328] unrecognized feature gate: Example Jul 21 21:00:42.474246 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:00:42.330128 2575 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 21 21:00:42.474246 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:00:42.330131 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 21 21:00:42.474246 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:00:42.330134 2575 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 21 21:00:42.474246 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:00:42.330136 2575 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 21 21:00:42.474246 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:00:42.330139 2575 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 21 21:00:42.474246 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:00:42.330141 2575 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 21 21:00:42.474246 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:00:42.330144 2575 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 21 21:00:42.474246 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:00:42.330146 2575 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 21 21:00:42.474246 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:00:42.330148 2575 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 21 21:00:42.474246 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:00:42.330151 2575 feature_gate.go:328] unrecognized feature gate: Example2 Jul 21 21:00:42.474246 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:00:42.330153 2575 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 21 21:00:42.474246 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:00:42.330156 2575 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 21 21:00:42.474246 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:00:42.330158 2575 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 21 21:00:42.474246 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:00:42.330161 2575 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 21 21:00:42.474246 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:00:42.330163 2575 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 21 21:00:42.474246 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:00:42.330165 2575 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 21 21:00:42.474246 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:00:42.330168 2575 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 21 21:00:42.474246 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:00:42.330171 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 21 21:00:42.474246 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:00:42.330174 2575 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 21 21:00:42.474246 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:00:42.330176 2575 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 21 21:00:42.383497 ip-10-0-140-97 systemd[1]: Started Kubernetes Kubelet. Jul 21 21:00:42.475425 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:00:42.330179 2575 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 21 21:00:42.475425 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:00:42.330181 2575 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 21 21:00:42.475425 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:00:42.330184 2575 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 21 21:00:42.475425 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:00:42.330186 2575 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 21 21:00:42.475425 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:00:42.330189 2575 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 21 21:00:42.475425 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:00:42.330191 2575 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 21 21:00:42.475425 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:00:42.330194 2575 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 21 21:00:42.475425 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:00:42.330196 2575 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 21 21:00:42.475425 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:00:42.330199 2575 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 21 21:00:42.475425 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:00:42.330201 2575 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 21 21:00:42.475425 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:00:42.330204 2575 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 21 21:00:42.475425 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:00:42.330206 2575 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 21 21:00:42.475425 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:00:42.330209 2575 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 21 21:00:42.475425 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:00:42.330595 2575 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 21 21:00:42.475425 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:00:42.330601 2575 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 21 21:00:42.475425 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:00:42.330604 2575 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 21 21:00:42.475425 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:00:42.330607 2575 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 21 21:00:42.475425 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:00:42.330610 2575 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 21 21:00:42.475425 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:00:42.330613 2575 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 21 21:00:42.484775 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:00:42.330615 2575 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 21 21:00:42.484775 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:00:42.330618 2575 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 21 21:00:42.484775 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:00:42.330620 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 21 21:00:42.484775 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:00:42.330623 2575 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 21 21:00:42.484775 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:00:42.330626 2575 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 21 21:00:42.484775 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:00:42.330628 2575 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 21 21:00:42.484775 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:00:42.330630 2575 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 21 21:00:42.484775 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:00:42.330633 2575 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 21 21:00:42.484775 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:00:42.330636 2575 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 21 21:00:42.484775 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:00:42.330638 2575 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 21 21:00:42.484775 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:00:42.330645 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 21 21:00:42.484775 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:00:42.330648 2575 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 21 21:00:42.484775 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:00:42.330651 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 21 21:00:42.484775 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:00:42.330653 2575 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 21 21:00:42.484775 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:00:42.330656 2575 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 21 21:00:42.484775 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:00:42.330660 2575 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 21 21:00:42.484775 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:00:42.330663 2575 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 21 21:00:42.484775 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:00:42.330666 2575 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 21 21:00:42.484775 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:00:42.330668 2575 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 21 21:00:42.486213 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:00:42.330671 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 21 21:00:42.486213 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:00:42.330673 2575 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 21 21:00:42.486213 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:00:42.330676 2575 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 21 21:00:42.486213 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:00:42.330680 2575 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 21 21:00:42.486213 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:00:42.330684 2575 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 21 21:00:42.486213 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:00:42.330687 2575 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 21 21:00:42.486213 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:00:42.330689 2575 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 21 21:00:42.486213 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:00:42.330691 2575 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 21 21:00:42.486213 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:00:42.330694 2575 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 21 21:00:42.486213 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:00:42.330697 2575 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 21 21:00:42.486213 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:00:42.330702 2575 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 21 21:00:42.486213 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:00:42.330705 2575 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 21 21:00:42.486213 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:00:42.330708 2575 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 21 21:00:42.486213 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:00:42.330710 2575 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 21 21:00:42.486213 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:00:42.330713 2575 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 21 21:00:42.486213 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:00:42.330715 2575 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 21 21:00:42.486213 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:00:42.330718 2575 feature_gate.go:328] unrecognized feature gate: Example2 Jul 21 21:00:42.486213 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:00:42.330720 2575 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 21 21:00:42.486213 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:00:42.330723 2575 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 21 21:00:42.486213 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:00:42.330725 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 21 21:00:42.487295 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:00:42.330728 2575 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 21 21:00:42.487295 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:00:42.330731 2575 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 21 21:00:42.487295 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:00:42.330733 2575 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 21 21:00:42.487295 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:00:42.330736 2575 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 21 21:00:42.487295 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:00:42.330739 2575 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 21 21:00:42.487295 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:00:42.330741 2575 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 21 21:00:42.487295 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:00:42.330744 2575 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 21 21:00:42.487295 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:00:42.330746 2575 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 21 21:00:42.487295 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:00:42.330749 2575 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 21 21:00:42.487295 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:00:42.330751 2575 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 21 21:00:42.487295 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:00:42.330754 2575 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 21 21:00:42.487295 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:00:42.330756 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 21 21:00:42.487295 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:00:42.330759 2575 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 21 21:00:42.487295 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:00:42.330762 2575 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 21 21:00:42.487295 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:00:42.330765 2575 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 21 21:00:42.487295 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:00:42.330767 2575 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 21 21:00:42.487295 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:00:42.330770 2575 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 21 21:00:42.487295 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:00:42.330772 2575 feature_gate.go:328] unrecognized feature gate: Example Jul 21 21:00:42.487295 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:00:42.330775 2575 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 21 21:00:42.487295 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:00:42.330777 2575 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 21 21:00:42.489168 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:00:42.330780 2575 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 21 21:00:42.489168 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:00:42.330783 2575 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 21 21:00:42.489168 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:00:42.330788 2575 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 21 21:00:42.489168 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:00:42.330790 2575 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 21 21:00:42.489168 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:00:42.330792 2575 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 21 21:00:42.489168 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:00:42.330795 2575 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 21 21:00:42.489168 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:00:42.330797 2575 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 21 21:00:42.489168 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:00:42.330800 2575 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 21 21:00:42.489168 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:00:42.330803 2575 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 21 21:00:42.489168 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:00:42.330805 2575 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 21 21:00:42.489168 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:00:42.330808 2575 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 21 21:00:42.489168 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:00:42.330811 2575 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 21 21:00:42.489168 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:00:42.330813 2575 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 21 21:00:42.489168 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:00:42.330815 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 21 21:00:42.489168 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:00:42.330819 2575 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 21 21:00:42.489168 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:00:42.330822 2575 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 21 21:00:42.489168 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:00:42.330825 2575 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 21 21:00:42.489168 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:00:42.330829 2575 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 21 21:00:42.489168 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:00:42.330833 2575 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 21 21:00:42.490190 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:00:42.330835 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 21 21:00:42.490190 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:00:42.330838 2575 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 21 21:00:42.490190 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:00:42.330840 2575 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 21 21:00:42.490190 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:00:42.330843 2575 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 21 21:00:42.490190 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:00:42.330845 2575 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 21 21:00:42.490190 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:00:42.330848 2575 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 21 21:00:42.490190 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:00:42.330850 2575 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 21 21:00:42.490190 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:00:42.330853 2575 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 21 21:00:42.490190 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:00:42.330855 2575 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 21 21:00:42.490190 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:00:42.330858 2575 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 21 21:00:42.490190 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:00:42.330860 2575 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 21 21:00:42.490190 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:00:42.330863 2575 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 21 21:00:42.490190 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:00:42.330865 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 21 21:00:42.490190 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:00:42.330868 2575 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 21 21:00:42.490190 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:00:42.330870 2575 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 21 21:00:42.490190 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:42.330940 2575 flags.go:64] FLAG: --address="0.0.0.0" Jul 21 21:00:42.490190 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:42.330947 2575 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Jul 21 21:00:42.490190 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:42.330953 2575 flags.go:64] FLAG: --anonymous-auth="true" Jul 21 21:00:42.490190 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:42.330957 2575 flags.go:64] FLAG: --application-metrics-count-limit="100" Jul 21 21:00:42.490190 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:42.330962 2575 flags.go:64] FLAG: --authentication-token-webhook="false" Jul 21 21:00:42.490190 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:42.330965 2575 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Jul 21 21:00:42.588120 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:42.330971 2575 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Jul 21 21:00:42.588120 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:42.330975 2575 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Jul 21 21:00:42.588120 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:42.330978 2575 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Jul 21 21:00:42.588120 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:42.330981 2575 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Jul 21 21:00:42.588120 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:42.330985 2575 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Jul 21 21:00:42.588120 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:42.330988 2575 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Jul 21 21:00:42.588120 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:42.330991 2575 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Jul 21 21:00:42.588120 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:42.330994 2575 flags.go:64] FLAG: --cgroup-root="" Jul 21 21:00:42.588120 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:42.330996 2575 flags.go:64] FLAG: --cgroups-per-qos="true" Jul 21 21:00:42.588120 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:42.330999 2575 flags.go:64] FLAG: --client-ca-file="" Jul 21 21:00:42.588120 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:42.331003 2575 flags.go:64] FLAG: --cloud-provider="external" Jul 21 21:00:42.588120 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:42.331006 2575 flags.go:64] FLAG: --cluster-dns="[]" Jul 21 21:00:42.588120 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:42.331010 2575 flags.go:64] FLAG: --cluster-domain="" Jul 21 21:00:42.588120 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:42.331012 2575 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Jul 21 21:00:42.588120 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:42.331015 2575 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Jul 21 21:00:42.588120 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:42.331019 2575 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Jul 21 21:00:42.588120 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:42.331022 2575 flags.go:64] FLAG: --container-log-max-files="5" Jul 21 21:00:42.588120 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:42.331025 2575 flags.go:64] FLAG: --container-log-max-size="10Mi" Jul 21 21:00:42.588120 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:42.331028 2575 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Jul 21 21:00:42.588120 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:42.331031 2575 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Jul 21 21:00:42.588120 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:42.331034 2575 flags.go:64] FLAG: --containerd-namespace="k8s.io" Jul 21 21:00:42.588120 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:42.331037 2575 flags.go:64] FLAG: --contention-profiling="false" Jul 21 21:00:42.588120 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:42.331040 2575 flags.go:64] FLAG: --cpu-cfs-quota="true" Jul 21 21:00:42.588120 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:42.331043 2575 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Jul 21 21:00:42.589501 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:42.331046 2575 flags.go:64] FLAG: --cpu-manager-policy="none" Jul 21 21:00:42.589501 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:42.331049 2575 flags.go:64] FLAG: --cpu-manager-policy-options="" Jul 21 21:00:42.589501 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:42.331054 2575 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Jul 21 21:00:42.589501 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:42.331057 2575 flags.go:64] FLAG: --enable-controller-attach-detach="true" Jul 21 21:00:42.589501 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:42.331060 2575 flags.go:64] FLAG: --enable-debugging-handlers="true" Jul 21 21:00:42.589501 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:42.331063 2575 flags.go:64] FLAG: --enable-load-reader="false" Jul 21 21:00:42.589501 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:42.331066 2575 flags.go:64] FLAG: --enable-server="true" Jul 21 21:00:42.589501 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:42.331069 2575 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Jul 21 21:00:42.589501 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:42.331074 2575 flags.go:64] FLAG: --event-burst="100" Jul 21 21:00:42.589501 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:42.331077 2575 flags.go:64] FLAG: --event-qps="50" Jul 21 21:00:42.589501 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:42.331080 2575 flags.go:64] FLAG: --event-storage-age-limit="default=0" Jul 21 21:00:42.589501 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:42.331083 2575 flags.go:64] FLAG: --event-storage-event-limit="default=0" Jul 21 21:00:42.589501 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:42.331086 2575 flags.go:64] FLAG: --eviction-hard="" Jul 21 21:00:42.589501 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:42.331089 2575 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Jul 21 21:00:42.589501 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:42.331092 2575 flags.go:64] FLAG: --eviction-minimum-reclaim="" Jul 21 21:00:42.589501 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:42.331095 2575 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Jul 21 21:00:42.589501 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:42.331098 2575 flags.go:64] FLAG: --eviction-soft="" Jul 21 21:00:42.589501 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:42.331101 2575 flags.go:64] FLAG: --eviction-soft-grace-period="" Jul 21 21:00:42.589501 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:42.331103 2575 flags.go:64] FLAG: --exit-on-lock-contention="false" Jul 21 21:00:42.589501 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:42.331107 2575 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Jul 21 21:00:42.589501 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:42.331110 2575 flags.go:64] FLAG: --experimental-mounter-path="" Jul 21 21:00:42.589501 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:42.331112 2575 flags.go:64] FLAG: --fail-cgroupv1="false" Jul 21 21:00:42.589501 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:42.331115 2575 flags.go:64] FLAG: --fail-swap-on="true" Jul 21 21:00:42.589501 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:42.331118 2575 flags.go:64] FLAG: --feature-gates="" Jul 21 21:00:42.589501 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:42.331122 2575 flags.go:64] FLAG: --file-check-frequency="20s" Jul 21 21:00:42.591457 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:42.331124 2575 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Jul 21 21:00:42.591457 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:42.331127 2575 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Jul 21 21:00:42.591457 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:42.331130 2575 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Jul 21 21:00:42.591457 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:42.331133 2575 flags.go:64] FLAG: --healthz-port="10248" Jul 21 21:00:42.591457 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:42.331136 2575 flags.go:64] FLAG: --help="false" Jul 21 21:00:42.591457 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:42.331139 2575 flags.go:64] FLAG: --hostname-override="ip-10-0-140-97.ec2.internal" Jul 21 21:00:42.591457 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:42.331142 2575 flags.go:64] FLAG: --housekeeping-interval="10s" Jul 21 21:00:42.591457 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:42.331144 2575 flags.go:64] FLAG: --http-check-frequency="20s" Jul 21 21:00:42.591457 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:42.331147 2575 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Jul 21 21:00:42.591457 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:42.331152 2575 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Jul 21 21:00:42.591457 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:42.331155 2575 flags.go:64] FLAG: --image-gc-high-threshold="85" Jul 21 21:00:42.591457 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:42.331158 2575 flags.go:64] FLAG: --image-gc-low-threshold="80" Jul 21 21:00:42.591457 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:42.331161 2575 flags.go:64] FLAG: --image-service-endpoint="" Jul 21 21:00:42.591457 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:42.331164 2575 flags.go:64] FLAG: --kernel-memcg-notification="false" Jul 21 21:00:42.591457 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:42.331166 2575 flags.go:64] FLAG: --kube-api-burst="100" Jul 21 21:00:42.591457 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:42.331171 2575 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Jul 21 21:00:42.591457 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:42.331174 2575 flags.go:64] FLAG: --kube-api-qps="50" Jul 21 21:00:42.591457 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:42.331177 2575 flags.go:64] FLAG: --kube-reserved="" Jul 21 21:00:42.591457 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:42.331180 2575 flags.go:64] FLAG: --kube-reserved-cgroup="" Jul 21 21:00:42.591457 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:42.331183 2575 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Jul 21 21:00:42.591457 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:42.331185 2575 flags.go:64] FLAG: --kubelet-cgroups="" Jul 21 21:00:42.591457 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:42.331188 2575 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Jul 21 21:00:42.591457 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:42.331191 2575 flags.go:64] FLAG: --lock-file="" Jul 21 21:00:42.591457 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:42.331194 2575 flags.go:64] FLAG: --log-cadvisor-usage="false" Jul 21 21:00:42.594289 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:42.331197 2575 flags.go:64] FLAG: --log-flush-frequency="5s" Jul 21 21:00:42.594289 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:42.331199 2575 flags.go:64] FLAG: --log-json-info-buffer-size="0" Jul 21 21:00:42.594289 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:42.331205 2575 flags.go:64] FLAG: --log-json-split-stream="false" Jul 21 21:00:42.594289 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:42.331208 2575 flags.go:64] FLAG: --log-text-info-buffer-size="0" Jul 21 21:00:42.594289 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:42.331211 2575 flags.go:64] FLAG: --log-text-split-stream="false" Jul 21 21:00:42.594289 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:42.331214 2575 flags.go:64] FLAG: --logging-format="text" Jul 21 21:00:42.594289 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:42.331217 2575 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Jul 21 21:00:42.594289 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:42.331220 2575 flags.go:64] FLAG: --make-iptables-util-chains="true" Jul 21 21:00:42.594289 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:42.331223 2575 flags.go:64] FLAG: --manifest-url="" Jul 21 21:00:42.594289 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:42.331226 2575 flags.go:64] FLAG: --manifest-url-header="" Jul 21 21:00:42.594289 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:42.331230 2575 flags.go:64] FLAG: --max-housekeeping-interval="15s" Jul 21 21:00:42.594289 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:42.331232 2575 flags.go:64] FLAG: --max-open-files="1000000" Jul 21 21:00:42.594289 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:42.331236 2575 flags.go:64] FLAG: --max-pods="110" Jul 21 21:00:42.594289 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:42.331239 2575 flags.go:64] FLAG: --maximum-dead-containers="-1" Jul 21 21:00:42.594289 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:42.331242 2575 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Jul 21 21:00:42.594289 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:42.331245 2575 flags.go:64] FLAG: --memory-manager-policy="None" Jul 21 21:00:42.594289 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:42.331248 2575 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Jul 21 21:00:42.594289 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:42.331252 2575 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Jul 21 21:00:42.594289 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:42.331255 2575 flags.go:64] FLAG: --node-ip="0.0.0.0" Jul 21 21:00:42.594289 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:42.331258 2575 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Jul 21 21:00:42.594289 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:42.331266 2575 flags.go:64] FLAG: --node-status-max-images="50" Jul 21 21:00:42.594289 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:42.331269 2575 flags.go:64] FLAG: --node-status-update-frequency="10s" Jul 21 21:00:42.594289 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:42.331271 2575 flags.go:64] FLAG: --oom-score-adj="-999" Jul 21 21:00:42.594289 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:42.331276 2575 flags.go:64] FLAG: --pod-cidr="" Jul 21 21:00:42.595250 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:42.331278 2575 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:4017e99d76d7b91598d3a3430cd0eba6eda8b33653addd3838fdfe3cca69c5a0" Jul 21 21:00:42.595250 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:42.331283 2575 flags.go:64] FLAG: --pod-manifest-path="" Jul 21 21:00:42.595250 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:42.331286 2575 flags.go:64] FLAG: --pod-max-pids="-1" Jul 21 21:00:42.595250 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:42.331289 2575 flags.go:64] FLAG: --pods-per-core="0" Jul 21 21:00:42.595250 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:42.331292 2575 flags.go:64] FLAG: --port="10250" Jul 21 21:00:42.595250 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:42.331295 2575 flags.go:64] FLAG: --protect-kernel-defaults="false" Jul 21 21:00:42.595250 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:42.331297 2575 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-08993ae86a5067cd0" Jul 21 21:00:42.595250 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:42.331300 2575 flags.go:64] FLAG: --qos-reserved="" Jul 21 21:00:42.595250 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:42.331303 2575 flags.go:64] FLAG: --read-only-port="10255" Jul 21 21:00:42.595250 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:42.331306 2575 flags.go:64] FLAG: --register-node="true" Jul 21 21:00:42.595250 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:42.331309 2575 flags.go:64] FLAG: --register-with-taints="" Jul 21 21:00:42.595250 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:42.331313 2575 flags.go:64] FLAG: --registry-burst="10" Jul 21 21:00:42.595250 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:42.331316 2575 flags.go:64] FLAG: --registry-qps="5" Jul 21 21:00:42.595250 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:42.331319 2575 flags.go:64] FLAG: --reserved-cpus="" Jul 21 21:00:42.595250 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:42.331321 2575 flags.go:64] FLAG: --reserved-memory="" Jul 21 21:00:42.595250 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:42.331325 2575 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Jul 21 21:00:42.595250 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:42.331328 2575 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Jul 21 21:00:42.595250 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:42.331331 2575 flags.go:64] FLAG: --rotate-certificates="false" Jul 21 21:00:42.595250 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:42.331333 2575 flags.go:64] FLAG: --rotate-server-certificates="false" Jul 21 21:00:42.595250 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:42.331336 2575 flags.go:64] FLAG: --runonce="false" Jul 21 21:00:42.595250 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:42.331339 2575 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Jul 21 21:00:42.595250 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:42.331342 2575 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Jul 21 21:00:42.595250 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:42.331345 2575 flags.go:64] FLAG: --seccomp-default="false" Jul 21 21:00:42.595250 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:42.331348 2575 flags.go:64] FLAG: --serialize-image-pulls="true" Jul 21 21:00:42.595250 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:42.331350 2575 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Jul 21 21:00:42.596207 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:42.331354 2575 flags.go:64] FLAG: --storage-driver-db="cadvisor" Jul 21 21:00:42.596207 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:42.331358 2575 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Jul 21 21:00:42.596207 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:42.331361 2575 flags.go:64] FLAG: --storage-driver-password="root" Jul 21 21:00:42.596207 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:42.331364 2575 flags.go:64] FLAG: --storage-driver-secure="false" Jul 21 21:00:42.596207 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:42.331367 2575 flags.go:64] FLAG: --storage-driver-table="stats" Jul 21 21:00:42.596207 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:42.331369 2575 flags.go:64] FLAG: --storage-driver-user="root" Jul 21 21:00:42.596207 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:42.331373 2575 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Jul 21 21:00:42.596207 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:42.331377 2575 flags.go:64] FLAG: --sync-frequency="1m0s" Jul 21 21:00:42.596207 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:42.331379 2575 flags.go:64] FLAG: --system-cgroups="" Jul 21 21:00:42.596207 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:42.331382 2575 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Jul 21 21:00:42.596207 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:42.331387 2575 flags.go:64] FLAG: --system-reserved-cgroup="" Jul 21 21:00:42.596207 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:42.331390 2575 flags.go:64] FLAG: --tls-cert-file="" Jul 21 21:00:42.596207 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:42.331392 2575 flags.go:64] FLAG: --tls-cipher-suites="[]" Jul 21 21:00:42.596207 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:42.331397 2575 flags.go:64] FLAG: --tls-min-version="" Jul 21 21:00:42.596207 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:42.331399 2575 flags.go:64] FLAG: --tls-private-key-file="" Jul 21 21:00:42.596207 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:42.331402 2575 flags.go:64] FLAG: --topology-manager-policy="none" Jul 21 21:00:42.596207 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:42.331405 2575 flags.go:64] FLAG: --topology-manager-policy-options="" Jul 21 21:00:42.596207 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:42.331408 2575 flags.go:64] FLAG: --topology-manager-scope="container" Jul 21 21:00:42.596207 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:42.331411 2575 flags.go:64] FLAG: --v="2" Jul 21 21:00:42.596207 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:42.331414 2575 flags.go:64] FLAG: --version="false" Jul 21 21:00:42.596207 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:42.331419 2575 flags.go:64] FLAG: --vmodule="" Jul 21 21:00:42.596207 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:42.331422 2575 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Jul 21 21:00:42.596207 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:42.331425 2575 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Jul 21 21:00:42.596207 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:00:42.331557 2575 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 21 21:00:42.596207 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:00:42.331562 2575 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 21 21:00:42.596857 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:00:42.331565 2575 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 21 21:00:42.596857 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:00:42.331568 2575 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 21 21:00:42.596857 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:00:42.331570 2575 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 21 21:00:42.596857 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:00:42.331573 2575 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 21 21:00:42.596857 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:00:42.331576 2575 feature_gate.go:328] unrecognized feature gate: Example Jul 21 21:00:42.596857 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:00:42.331579 2575 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 21 21:00:42.596857 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:00:42.331582 2575 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 21 21:00:42.596857 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:00:42.331585 2575 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 21 21:00:42.596857 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:00:42.331589 2575 feature_gate.go:328] unrecognized feature gate: Example2 Jul 21 21:00:42.596857 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:00:42.331592 2575 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 21 21:00:42.596857 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:00:42.331594 2575 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 21 21:00:42.596857 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:00:42.331597 2575 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 21 21:00:42.596857 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:00:42.331600 2575 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 21 21:00:42.596857 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:00:42.331604 2575 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 21 21:00:42.596857 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:00:42.331606 2575 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 21 21:00:42.596857 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:00:42.331609 2575 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 21 21:00:42.596857 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:00:42.331611 2575 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 21 21:00:42.596857 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:00:42.331614 2575 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 21 21:00:42.596857 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:00:42.331616 2575 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 21 21:00:42.596857 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:00:42.331619 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 21 21:00:42.597414 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:00:42.331621 2575 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 21 21:00:42.597414 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:00:42.331624 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 21 21:00:42.597414 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:00:42.331627 2575 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 21 21:00:42.597414 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:00:42.331629 2575 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 21 21:00:42.597414 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:00:42.331632 2575 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 21 21:00:42.597414 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:00:42.331634 2575 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 21 21:00:42.597414 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:00:42.331637 2575 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 21 21:00:42.597414 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:00:42.331639 2575 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 21 21:00:42.597414 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:00:42.331642 2575 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 21 21:00:42.597414 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:00:42.331646 2575 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 21 21:00:42.597414 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:00:42.331650 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 21 21:00:42.597414 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:00:42.331653 2575 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 21 21:00:42.597414 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:00:42.331655 2575 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 21 21:00:42.597414 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:00:42.331658 2575 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 21 21:00:42.597414 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:00:42.331661 2575 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 21 21:00:42.597414 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:00:42.331663 2575 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 21 21:00:42.597414 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:00:42.331666 2575 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 21 21:00:42.597414 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:00:42.331668 2575 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 21 21:00:42.597414 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:00:42.331671 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 21 21:00:42.597414 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:00:42.331674 2575 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 21 21:00:42.597908 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:00:42.331678 2575 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 21 21:00:42.597908 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:00:42.331680 2575 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 21 21:00:42.597908 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:00:42.331683 2575 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 21 21:00:42.597908 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:00:42.331685 2575 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 21 21:00:42.597908 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:00:42.331688 2575 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 21 21:00:42.597908 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:00:42.331692 2575 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 21 21:00:42.597908 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:00:42.331695 2575 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 21 21:00:42.597908 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:00:42.331697 2575 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 21 21:00:42.597908 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:00:42.331700 2575 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 21 21:00:42.597908 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:00:42.331703 2575 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 21 21:00:42.597908 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:00:42.331708 2575 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 21 21:00:42.597908 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:00:42.331710 2575 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 21 21:00:42.597908 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:00:42.331713 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 21 21:00:42.597908 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:00:42.331716 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 21 21:00:42.597908 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:00:42.331720 2575 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 21 21:00:42.597908 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:00:42.331723 2575 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 21 21:00:42.597908 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:00:42.331726 2575 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 21 21:00:42.597908 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:00:42.331728 2575 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 21 21:00:42.597908 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:00:42.331731 2575 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 21 21:00:42.598371 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:00:42.331734 2575 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 21 21:00:42.598371 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:00:42.331737 2575 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 21 21:00:42.598371 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:00:42.331740 2575 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 21 21:00:42.598371 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:00:42.331742 2575 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 21 21:00:42.598371 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:00:42.331744 2575 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 21 21:00:42.598371 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:00:42.331747 2575 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 21 21:00:42.598371 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:00:42.331750 2575 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 21 21:00:42.598371 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:00:42.331752 2575 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 21 21:00:42.598371 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:00:42.331755 2575 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 21 21:00:42.598371 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:00:42.331757 2575 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 21 21:00:42.598371 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:00:42.331760 2575 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 21 21:00:42.598371 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:00:42.331762 2575 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 21 21:00:42.598371 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:00:42.331765 2575 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 21 21:00:42.598371 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:00:42.331769 2575 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 21 21:00:42.598371 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:00:42.331772 2575 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 21 21:00:42.598371 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:00:42.331774 2575 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 21 21:00:42.598371 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:00:42.331777 2575 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 21 21:00:42.598371 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:00:42.331780 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 21 21:00:42.598371 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:00:42.331783 2575 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 21 21:00:42.598840 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:00:42.331786 2575 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 21 21:00:42.598840 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:00:42.331788 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 21 21:00:42.598840 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:00:42.331791 2575 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 21 21:00:42.598840 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:00:42.331794 2575 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 21 21:00:42.598840 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:00:42.331796 2575 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 21 21:00:42.598840 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:00:42.331799 2575 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 21 21:00:42.598840 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:00:42.331801 2575 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 21 21:00:42.598840 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:00:42.331804 2575 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 21 21:00:42.598840 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:00:42.331806 2575 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 21 21:00:42.598840 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:00:42.331809 2575 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 21 21:00:42.598840 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:00:42.331811 2575 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 21 21:00:42.598840 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:00:42.331814 2575 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 21 21:00:42.598840 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:00:42.331816 2575 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 21 21:00:42.598840 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:00:42.331818 2575 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 21 21:00:42.598840 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:00:42.331821 2575 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 21 21:00:42.598840 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:00:42.331824 2575 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 21 21:00:42.598840 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:00:42.331826 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 21 21:00:42.598840 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:00:42.331829 2575 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 21 21:00:42.598840 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:00:42.331831 2575 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 21 21:00:42.599298 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:42.332505 2575 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 21 21:00:42.599298 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:42.338950 2575 server.go:529] "Kubelet version" kubeletVersion="v1.34.9" Jul 21 21:00:42.599298 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:42.338966 2575 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Jul 21 21:00:42.599298 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:42.338984 2575 watchdog_linux.go:95] "Systemd watchdog is not enabled" Jul 21 21:00:42.599298 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:42.338990 2575 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Jul 21 21:00:42.599298 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:00:42.339045 2575 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 21 21:00:42.599298 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:00:42.339050 2575 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 21 21:00:42.599298 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:00:42.339054 2575 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 21 21:00:42.599298 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:00:42.339057 2575 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 21 21:00:42.599298 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:00:42.339060 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 21 21:00:42.599298 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:00:42.339062 2575 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 21 21:00:42.599298 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:00:42.339065 2575 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 21 21:00:42.599298 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:00:42.339067 2575 feature_gate.go:328] unrecognized feature gate: Example Jul 21 21:00:42.599298 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:00:42.339070 2575 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 21 21:00:42.599648 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:00:42.339075 2575 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 21 21:00:42.599648 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:00:42.339078 2575 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 21 21:00:42.599648 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:00:42.339081 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 21 21:00:42.599648 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:00:42.339084 2575 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 21 21:00:42.599648 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:00:42.339087 2575 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 21 21:00:42.599648 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:00:42.339090 2575 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 21 21:00:42.599648 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:00:42.339093 2575 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 21 21:00:42.599648 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:00:42.339096 2575 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 21 21:00:42.599648 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:00:42.339099 2575 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 21 21:00:42.599648 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:00:42.339102 2575 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 21 21:00:42.599648 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:00:42.339105 2575 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 21 21:00:42.599648 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:00:42.339107 2575 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 21 21:00:42.599648 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:00:42.339110 2575 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 21 21:00:42.599648 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:00:42.339113 2575 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 21 21:00:42.599648 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:00:42.339116 2575 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 21 21:00:42.599648 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:00:42.339118 2575 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 21 21:00:42.599648 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:00:42.339121 2575 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 21 21:00:42.599648 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:00:42.339123 2575 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 21 21:00:42.599648 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:00:42.339126 2575 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 21 21:00:42.600166 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:00:42.339129 2575 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 21 21:00:42.600166 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:00:42.339132 2575 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 21 21:00:42.600166 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:00:42.339134 2575 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 21 21:00:42.600166 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:00:42.339137 2575 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 21 21:00:42.600166 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:00:42.339139 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 21 21:00:42.600166 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:00:42.339142 2575 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 21 21:00:42.600166 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:00:42.339144 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 21 21:00:42.600166 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:00:42.339147 2575 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 21 21:00:42.600166 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:00:42.339149 2575 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 21 21:00:42.600166 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:00:42.339153 2575 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 21 21:00:42.600166 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:00:42.339156 2575 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 21 21:00:42.600166 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:00:42.339160 2575 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 21 21:00:42.600166 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:00:42.339162 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 21 21:00:42.600166 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:00:42.339165 2575 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 21 21:00:42.600166 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:00:42.339168 2575 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 21 21:00:42.600166 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:00:42.339170 2575 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 21 21:00:42.600166 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:00:42.339173 2575 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 21 21:00:42.600166 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:00:42.339175 2575 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 21 21:00:42.600166 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:00:42.339178 2575 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 21 21:00:42.600166 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:00:42.339180 2575 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 21 21:00:42.600696 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:00:42.339183 2575 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 21 21:00:42.600696 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:00:42.339186 2575 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 21 21:00:42.600696 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:00:42.339188 2575 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 21 21:00:42.600696 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:00:42.339191 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 21 21:00:42.600696 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:00:42.339193 2575 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 21 21:00:42.600696 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:00:42.339198 2575 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 21 21:00:42.600696 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:00:42.339201 2575 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 21 21:00:42.600696 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:00:42.339203 2575 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 21 21:00:42.600696 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:00:42.339206 2575 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 21 21:00:42.600696 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:00:42.339208 2575 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 21 21:00:42.600696 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:00:42.339211 2575 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 21 21:00:42.600696 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:00:42.339214 2575 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 21 21:00:42.600696 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:00:42.339216 2575 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 21 21:00:42.600696 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:00:42.339219 2575 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 21 21:00:42.600696 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:00:42.339222 2575 feature_gate.go:328] unrecognized feature gate: Example2 Jul 21 21:00:42.600696 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:00:42.339224 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 21 21:00:42.600696 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:00:42.339227 2575 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 21 21:00:42.600696 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:00:42.339229 2575 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 21 21:00:42.600696 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:00:42.339232 2575 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 21 21:00:42.600696 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:00:42.339235 2575 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 21 21:00:42.601179 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:00:42.339237 2575 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 21 21:00:42.601179 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:00:42.339240 2575 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 21 21:00:42.601179 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:00:42.339242 2575 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 21 21:00:42.601179 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:00:42.339245 2575 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 21 21:00:42.601179 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:00:42.339248 2575 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 21 21:00:42.601179 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:00:42.339251 2575 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 21 21:00:42.601179 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:00:42.339255 2575 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 21 21:00:42.601179 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:00:42.339257 2575 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 21 21:00:42.601179 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:00:42.339260 2575 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 21 21:00:42.601179 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:00:42.339263 2575 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 21 21:00:42.601179 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:00:42.339266 2575 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 21 21:00:42.601179 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:00:42.339269 2575 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 21 21:00:42.601179 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:00:42.339271 2575 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 21 21:00:42.601179 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:00:42.339274 2575 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 21 21:00:42.601179 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:00:42.339276 2575 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 21 21:00:42.601179 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:00:42.339279 2575 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 21 21:00:42.601179 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:00:42.339282 2575 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 21 21:00:42.601179 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:00:42.339285 2575 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 21 21:00:42.601179 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:00:42.339288 2575 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 21 21:00:42.601665 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:00:42.339290 2575 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 21 21:00:42.601665 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:00:42.339293 2575 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 21 21:00:42.601665 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:00:42.339296 2575 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 21 21:00:42.601665 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:00:42.339298 2575 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 21 21:00:42.601665 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:00:42.339301 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 21 21:00:42.601665 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:00:42.339303 2575 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 21 21:00:42.601665 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:00:42.339306 2575 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 21 21:00:42.601665 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:00:42.339308 2575 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 21 21:00:42.601665 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:00:42.339311 2575 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 21 21:00:42.601665 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:00:42.339314 2575 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 21 21:00:42.601665 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:00:42.339316 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 21 21:00:42.601665 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:00:42.339319 2575 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 21 21:00:42.601665 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:42.339324 2575 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 21 21:00:42.601665 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:00:42.339433 2575 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 21 21:00:42.602008 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:00:42.339437 2575 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 21 21:00:42.602008 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:00:42.339441 2575 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 21 21:00:42.602008 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:00:42.339444 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 21 21:00:42.602008 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:00:42.339447 2575 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 21 21:00:42.602008 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:00:42.339450 2575 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 21 21:00:42.602008 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:00:42.339452 2575 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 21 21:00:42.602008 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:00:42.339455 2575 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 21 21:00:42.602008 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:00:42.339457 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 21 21:00:42.602008 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:00:42.339460 2575 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 21 21:00:42.602008 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:00:42.339465 2575 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 21 21:00:42.602008 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:00:42.339467 2575 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 21 21:00:42.602008 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:00:42.339470 2575 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 21 21:00:42.602008 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:00:42.339472 2575 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 21 21:00:42.602008 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:00:42.339475 2575 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 21 21:00:42.602008 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:00:42.339478 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 21 21:00:42.602008 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:00:42.339480 2575 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 21 21:00:42.602008 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:00:42.339482 2575 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 21 21:00:42.602008 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:00:42.339486 2575 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 21 21:00:42.602008 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:00:42.339489 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 21 21:00:42.602008 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:00:42.339491 2575 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 21 21:00:42.602490 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:00:42.339494 2575 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 21 21:00:42.602490 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:00:42.339497 2575 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 21 21:00:42.602490 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:00:42.339499 2575 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 21 21:00:42.602490 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:00:42.339502 2575 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 21 21:00:42.602490 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:00:42.339506 2575 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 21 21:00:42.602490 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:00:42.339509 2575 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 21 21:00:42.602490 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:00:42.339512 2575 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 21 21:00:42.602490 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:00:42.339515 2575 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 21 21:00:42.602490 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:00:42.339518 2575 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 21 21:00:42.602490 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:00:42.339520 2575 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 21 21:00:42.602490 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:00:42.339523 2575 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 21 21:00:42.602490 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:00:42.339540 2575 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 21 21:00:42.602490 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:00:42.339544 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 21 21:00:42.602490 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:00:42.339547 2575 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 21 21:00:42.602490 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:00:42.339550 2575 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 21 21:00:42.602490 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:00:42.339553 2575 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 21 21:00:42.602490 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:00:42.339555 2575 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 21 21:00:42.602490 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:00:42.339558 2575 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 21 21:00:42.602490 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:00:42.339560 2575 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 21 21:00:42.602992 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:00:42.339563 2575 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 21 21:00:42.602992 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:00:42.339565 2575 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 21 21:00:42.602992 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:00:42.339569 2575 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 21 21:00:42.602992 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:00:42.339572 2575 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 21 21:00:42.602992 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:00:42.339575 2575 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 21 21:00:42.602992 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:00:42.339577 2575 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 21 21:00:42.602992 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:00:42.339580 2575 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 21 21:00:42.602992 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:00:42.339582 2575 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 21 21:00:42.602992 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:00:42.339585 2575 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 21 21:00:42.602992 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:00:42.339587 2575 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 21 21:00:42.602992 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:00:42.339591 2575 feature_gate.go:328] unrecognized feature gate: Example2 Jul 21 21:00:42.602992 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:00:42.339594 2575 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 21 21:00:42.602992 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:00:42.339597 2575 feature_gate.go:328] unrecognized feature gate: Example Jul 21 21:00:42.602992 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:00:42.339599 2575 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 21 21:00:42.602992 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:00:42.339602 2575 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 21 21:00:42.602992 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:00:42.339604 2575 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 21 21:00:42.602992 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:00:42.339607 2575 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 21 21:00:42.602992 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:00:42.339609 2575 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 21 21:00:42.602992 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:00:42.339612 2575 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 21 21:00:42.602992 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:00:42.339614 2575 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 21 21:00:42.603463 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:00:42.339617 2575 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 21 21:00:42.603463 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:00:42.339620 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 21 21:00:42.603463 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:00:42.339623 2575 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 21 21:00:42.603463 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:00:42.339626 2575 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 21 21:00:42.603463 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:00:42.339628 2575 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 21 21:00:42.603463 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:00:42.339631 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 21 21:00:42.603463 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:00:42.339633 2575 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 21 21:00:42.603463 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:00:42.339636 2575 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 21 21:00:42.603463 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:00:42.339638 2575 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 21 21:00:42.603463 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:00:42.339640 2575 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 21 21:00:42.603463 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:00:42.339643 2575 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 21 21:00:42.603463 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:00:42.339645 2575 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 21 21:00:42.603463 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:00:42.339648 2575 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 21 21:00:42.603463 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:00:42.339650 2575 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 21 21:00:42.603463 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:00:42.339654 2575 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 21 21:00:42.603463 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:00:42.339657 2575 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 21 21:00:42.603463 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:00:42.339659 2575 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 21 21:00:42.603463 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:00:42.339662 2575 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 21 21:00:42.603463 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:00:42.339664 2575 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 21 21:00:42.603463 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:00:42.339667 2575 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 21 21:00:42.603983 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:00:42.339669 2575 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 21 21:00:42.603983 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:00:42.339672 2575 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 21 21:00:42.603983 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:00:42.339676 2575 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 21 21:00:42.603983 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:00:42.339678 2575 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 21 21:00:42.603983 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:00:42.339681 2575 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 21 21:00:42.603983 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:00:42.339683 2575 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 21 21:00:42.603983 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:00:42.339687 2575 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 21 21:00:42.603983 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:00:42.339690 2575 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 21 21:00:42.603983 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:00:42.339693 2575 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 21 21:00:42.603983 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:00:42.339695 2575 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 21 21:00:42.603983 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:00:42.339698 2575 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 21 21:00:42.603983 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:00:42.339700 2575 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 21 21:00:42.603983 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:00:42.339703 2575 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 21 21:00:42.603983 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:00:42.339705 2575 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 21 21:00:42.603983 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:00:42.339711 2575 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 21 21:00:42.603983 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:00:42.339715 2575 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 21 21:00:42.603983 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:00:42.339719 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 21 21:00:42.603983 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:00:42.339721 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 21 21:00:42.603983 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:00:42.339724 2575 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 21 21:00:42.604426 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:42.339729 2575 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 21 21:00:42.604426 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:42.339838 2575 server.go:962] "Client rotation is on, will bootstrap in background" Jul 21 21:00:42.604426 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:42.341616 2575 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Jul 21 21:00:42.604426 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:42.342775 2575 server.go:1019] "Starting client certificate rotation" Jul 21 21:00:42.604426 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:42.342869 2575 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 21 21:00:42.604426 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:42.342907 2575 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 21 21:00:42.604426 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:42.344683 2575 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 21 21:00:42.604426 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:42.346194 2575 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 21 21:00:42.604426 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:42.356930 2575 log.go:25] "Validated CRI v1 runtime API" Jul 21 21:00:42.604426 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:42.358839 2575 log.go:25] "Validated CRI v1 image API" Jul 21 21:00:42.604426 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:42.360373 2575 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Jul 21 21:00:42.604426 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:42.362470 2575 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 a2ae45d7-a8d7-45d5-b495-a965fcc5ed8e:/dev/nvme0n1p3 c07113a8-31c8-493c-81e0-3cfedfa966ef:/dev/nvme0n1p4] Jul 21 21:00:42.604739 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:42.362485 2575 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Jul 21 21:00:42.604772 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:42.367675 2575 manager.go:216] Machine: {Timestamp:2026-07-21 21:00:42.366606884 +0000 UTC m=+0.301356542 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3099988 MemoryCapacity:33164492800 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec21c1cafcac8196231dd5d1ccc48755 SystemUUID:ec21c1ca-fcac-8196-231d-d5d1ccc48755 BootID:4ec9e206-aa07-4276-b1a6-a4fbfeb6f7cf Filesystems:[{Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582246400 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582246400 Type:vfs Inodes:4048400 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:45:5a:2a:01:3b Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:45:5a:2a:01:3b Speed:0 Mtu:9001} {Name:ovs-system MacAddress:82:f0:53:38:d8:01 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164492800 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Jul 21 21:00:42.604772 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:42.367785 2575 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Jul 21 21:00:42.604772 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:42.367879 2575 manager.go:232] Version: {KernelVersion:5.14.0-570.128.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260715-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Jul 21 21:00:42.604772 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:42.369320 2575 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Jul 21 21:00:42.604772 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:42.369340 2575 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-140-97.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Jul 21 21:00:42.604772 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:42.369881 2575 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 21 21:00:42.604772 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:42.369950 2575 topology_manager.go:138] "Creating topology manager with none policy" Jul 21 21:00:42.604979 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:42.369964 2575 container_manager_linux.go:309] "Creating device plugin manager" Jul 21 21:00:42.604979 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:42.369974 2575 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 21 21:00:42.604979 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:42.370656 2575 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 21 21:00:42.604979 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:42.371722 2575 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Jul 21 21:00:42.604979 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:42.374777 2575 state_mem.go:36] "Initialized new in-memory state store" Jul 21 21:00:42.604979 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:42.374892 2575 server.go:1240] "Using root directory" path="/var/lib/kubelet" Jul 21 21:00:42.604979 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:42.374920 2575 kubelet.go:486] "Attempting to sync node with API server" Jul 21 21:00:42.604979 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:42.374937 2575 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Jul 21 21:00:42.604979 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:42.374948 2575 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Jul 21 21:00:42.604979 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:42.374956 2575 kubelet.go:398] "Adding apiserver pod source" Jul 21 21:00:42.604979 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:42.374965 2575 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Jul 21 21:00:42.604979 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:42.375270 2575 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 21 21:00:42.604979 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:42.375284 2575 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 21 21:00:42.604979 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:42.376818 2575 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1" Jul 21 21:00:42.604979 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:42.378353 2575 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Jul 21 21:00:42.604979 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:42.378380 2575 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Jul 21 21:00:42.604979 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:42.378996 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Jul 21 21:00:42.604979 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:42.379019 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Jul 21 21:00:42.604979 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:42.379026 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Jul 21 21:00:42.604979 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:42.379032 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Jul 21 21:00:42.604979 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:42.379040 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Jul 21 21:00:42.604979 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:42.379045 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Jul 21 21:00:42.604979 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:42.379053 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Jul 21 21:00:42.604979 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:42.379058 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Jul 21 21:00:42.604979 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:42.379065 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Jul 21 21:00:42.604979 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:42.379070 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Jul 21 21:00:42.604979 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:42.379079 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Jul 21 21:00:42.604979 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:42.379084 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Jul 21 21:00:42.604979 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:42.379607 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Jul 21 21:00:42.604979 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:42.379614 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Jul 21 21:00:42.604979 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:42.382791 2575 server.go:1268] "Started kubelet" Jul 21 21:00:42.604979 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:42.382863 2575 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Jul 21 21:00:42.604979 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:42.382979 2575 server_v1.go:49] "podresources" method="list" useActivePods=true Jul 21 21:00:42.604979 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:42.382911 2575 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Jul 21 21:00:42.604979 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:42.384218 2575 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Jul 21 21:00:42.604979 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:42.388359 2575 server.go:310] "Adding debug handlers to kubelet server" Jul 21 21:00:42.604979 ip-10-0-140-97 kubenswrapper[2575]: E0721 21:00:42.389021 2575 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Jul 21 21:00:42.604979 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:42.391046 2575 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-140-97.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Jul 21 21:00:42.604979 ip-10-0-140-97 kubenswrapper[2575]: E0721 21:00:42.391753 2575 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-140-97.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Jul 21 21:00:42.605908 ip-10-0-140-97 kubenswrapper[2575]: E0721 21:00:42.391754 2575 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Jul 21 21:00:42.605908 ip-10-0-140-97 kubenswrapper[2575]: E0721 21:00:42.391324 2575 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-140-97.ec2.internal.18c469c3e25bc7c9 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-140-97.ec2.internal,UID:ip-10-0-140-97.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-140-97.ec2.internal,},FirstTimestamp:2026-07-21 21:00:42.382763977 +0000 UTC m=+0.317513633,LastTimestamp:2026-07-21 21:00:42.382763977 +0000 UTC m=+0.317513633,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-140-97.ec2.internal,}" Jul 21 21:00:42.605908 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:42.393183 2575 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Jul 21 21:00:42.605908 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:42.393199 2575 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Jul 21 21:00:42.605908 ip-10-0-140-97 kubenswrapper[2575]: E0721 21:00:42.393249 2575 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-97.ec2.internal\" not found" Jul 21 21:00:42.605908 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:42.394187 2575 factory.go:55] Registering systemd factory Jul 21 21:00:42.605908 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:42.394203 2575 factory.go:223] Registration of the systemd container factory successfully Jul 21 21:00:42.605908 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:42.394188 2575 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Jul 21 21:00:42.605908 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:42.394307 2575 volume_manager.go:311] "The desired_state_of_world populator starts" Jul 21 21:00:42.605908 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:42.394328 2575 volume_manager.go:313] "Starting Kubelet Volume Manager" Jul 21 21:00:42.605908 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:42.394440 2575 reconstruct.go:97] "Volume reconstruction finished" Jul 21 21:00:42.605908 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:42.394456 2575 reconciler.go:29] "Reconciler: start to sync state" Jul 21 21:00:42.605908 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:42.394467 2575 factory.go:153] Registering CRI-O factory Jul 21 21:00:42.605908 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:42.394482 2575 factory.go:223] Registration of the crio container factory successfully Jul 21 21:00:42.605908 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:42.394546 2575 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Jul 21 21:00:42.605908 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:42.394569 2575 factory.go:103] Registering Raw factory Jul 21 21:00:42.605908 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:42.394583 2575 manager.go:1196] Started watching for new ooms in manager Jul 21 21:00:42.605908 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:42.395059 2575 manager.go:349] Starting recovery of all containers Jul 21 21:00:42.605908 ip-10-0-140-97 kubenswrapper[2575]: E0721 21:00:42.396408 2575 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-140-97.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Jul 21 21:00:42.605908 ip-10-0-140-97 kubenswrapper[2575]: E0721 21:00:42.396551 2575 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" Jul 21 21:00:42.605908 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:42.403853 2575 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-qxbvq" Jul 21 21:00:42.605908 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:42.405598 2575 manager.go:354] Recovery completed Jul 21 21:00:42.605908 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:42.409275 2575 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 21 21:00:42.605908 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:42.410939 2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-97.ec2.internal" event="NodeHasSufficientMemory" Jul 21 21:00:42.605908 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:42.410967 2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-97.ec2.internal" event="NodeHasNoDiskPressure" Jul 21 21:00:42.605908 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:42.410980 2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-97.ec2.internal" event="NodeHasSufficientPID" Jul 21 21:00:42.605908 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:42.411435 2575 cpu_manager.go:222] "Starting CPU manager" policy="none" Jul 21 21:00:42.605908 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:42.411444 2575 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Jul 21 21:00:42.605908 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:42.411448 2575 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 21 21:00:42.605908 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:42.411460 2575 state_mem.go:36] "Initialized new in-memory state store" Jul 21 21:00:42.605908 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:42.412083 2575 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-qxbvq" Jul 21 21:00:42.606693 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:42.412153 2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-97.ec2.internal" event="NodeHasSufficientMemory" Jul 21 21:00:42.606693 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:42.412181 2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-97.ec2.internal" event="NodeHasNoDiskPressure" Jul 21 21:00:42.606693 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:42.412192 2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-97.ec2.internal" event="NodeHasSufficientPID" Jul 21 21:00:42.606693 ip-10-0-140-97 kubenswrapper[2575]: E0721 21:00:42.413134 2575 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-140-97.ec2.internal.18c469c3e409e092 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-140-97.ec2.internal,UID:ip-10-0-140-97.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-140-97.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-140-97.ec2.internal,},FirstTimestamp:2026-07-21 21:00:42.410950802 +0000 UTC m=+0.345700460,LastTimestamp:2026-07-21 21:00:42.410950802 +0000 UTC m=+0.345700460,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-140-97.ec2.internal,}" Jul 21 21:00:42.606693 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:42.413231 2575 policy_none.go:49] "None policy: Start" Jul 21 21:00:42.606693 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:42.413246 2575 memory_manager.go:187] "Starting memorymanager" policy="None" Jul 21 21:00:42.606693 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:42.413259 2575 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Jul 21 21:00:42.606693 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:42.414198 2575 policy_none.go:47] "Start" Jul 21 21:00:42.606693 ip-10-0-140-97 kubenswrapper[2575]: E0721 21:00:42.493515 2575 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-97.ec2.internal\" not found" Jul 21 21:00:42.606693 ip-10-0-140-97 kubenswrapper[2575]: E0721 21:00:42.593805 2575 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-97.ec2.internal\" not found" Jul 21 21:00:42.606693 ip-10-0-140-97 kubenswrapper[2575]: E0721 21:00:42.602838 2575 nodelease.go:49] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"ip-10-0-140-97.ec2.internal\" not found" node="ip-10-0-140-97.ec2.internal" Jul 21 21:00:42.628100 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:42.628077 2575 manager.go:337] "Starting Device Plugin manager" Jul 21 21:00:42.628190 ip-10-0-140-97 kubenswrapper[2575]: E0721 21:00:42.628113 2575 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Jul 21 21:00:42.628190 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:42.628122 2575 server.go:87] "Starting device plugin registration server" Jul 21 21:00:42.628521 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:42.628382 2575 eviction_manager.go:189] "Eviction manager: starting control loop" Jul 21 21:00:42.628606 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:42.628551 2575 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Jul 21 21:00:42.628672 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:42.628652 2575 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Jul 21 21:00:42.629187 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:42.628738 2575 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Jul 21 21:00:42.629187 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:42.628750 2575 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Jul 21 21:00:42.629328 ip-10-0-140-97 kubenswrapper[2575]: E0721 21:00:42.629272 2575 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Jul 21 21:00:42.629328 ip-10-0-140-97 kubenswrapper[2575]: E0721 21:00:42.629320 2575 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-140-97.ec2.internal\" not found" Jul 21 21:00:42.728998 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:42.728968 2575 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 21 21:00:42.729864 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:42.729847 2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-97.ec2.internal" event="NodeHasSufficientMemory" Jul 21 21:00:42.729927 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:42.729878 2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-97.ec2.internal" event="NodeHasNoDiskPressure" Jul 21 21:00:42.729927 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:42.729887 2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-97.ec2.internal" event="NodeHasSufficientPID" Jul 21 21:00:42.729927 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:42.729912 2575 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-140-97.ec2.internal" Jul 21 21:00:42.733078 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:42.733050 2575 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Jul 21 21:00:42.734157 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:42.734140 2575 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Jul 21 21:00:42.734235 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:42.734175 2575 status_manager.go:244] "Starting to sync pod status with apiserver" Jul 21 21:00:42.734235 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:42.734195 2575 kubelet.go:2448] "Starting kubelet main sync loop" Jul 21 21:00:42.734235 ip-10-0-140-97 kubenswrapper[2575]: E0721 21:00:42.734223 2575 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Jul 21 21:00:42.736310 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:42.736266 2575 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 21 21:00:42.737540 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:42.737512 2575 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-140-97.ec2.internal" Jul 21 21:00:42.737621 ip-10-0-140-97 kubenswrapper[2575]: E0721 21:00:42.737549 2575 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-140-97.ec2.internal\": node \"ip-10-0-140-97.ec2.internal\" not found" Jul 21 21:00:42.770805 ip-10-0-140-97 kubenswrapper[2575]: E0721 21:00:42.770790 2575 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-97.ec2.internal\" not found" Jul 21 21:00:42.834307 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:42.834284 2575 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-97.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-140-97.ec2.internal"] Jul 21 21:00:42.834395 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:42.834346 2575 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 21 21:00:42.835738 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:42.835723 2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-97.ec2.internal" event="NodeHasSufficientMemory" Jul 21 21:00:42.835804 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:42.835750 2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-97.ec2.internal" event="NodeHasNoDiskPressure" Jul 21 21:00:42.835804 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:42.835760 2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-97.ec2.internal" event="NodeHasSufficientPID" Jul 21 21:00:42.837062 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:42.837052 2575 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 21 21:00:42.837172 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:42.837152 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-97.ec2.internal" Jul 21 21:00:42.837257 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:42.837187 2575 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 21 21:00:42.837701 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:42.837683 2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-97.ec2.internal" event="NodeHasSufficientMemory" Jul 21 21:00:42.837701 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:42.837689 2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-97.ec2.internal" event="NodeHasSufficientMemory" Jul 21 21:00:42.837813 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:42.837707 2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-97.ec2.internal" event="NodeHasNoDiskPressure" Jul 21 21:00:42.837813 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:42.837721 2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-97.ec2.internal" event="NodeHasSufficientPID" Jul 21 21:00:42.837813 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:42.837722 2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-97.ec2.internal" event="NodeHasNoDiskPressure" Jul 21 21:00:42.837813 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:42.837742 2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-97.ec2.internal" event="NodeHasSufficientPID" Jul 21 21:00:42.838910 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:42.838893 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-97.ec2.internal" Jul 21 21:00:42.838974 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:42.838921 2575 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 21 21:00:42.839539 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:42.839515 2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-97.ec2.internal" event="NodeHasSufficientMemory" Jul 21 21:00:42.839603 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:42.839552 2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-97.ec2.internal" event="NodeHasNoDiskPressure" Jul 21 21:00:42.839603 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:42.839561 2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-97.ec2.internal" event="NodeHasSufficientPID" Jul 21 21:00:42.863987 ip-10-0-140-97 kubenswrapper[2575]: E0721 21:00:42.863970 2575 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-140-97.ec2.internal\" not found" node="ip-10-0-140-97.ec2.internal" Jul 21 21:00:42.868151 ip-10-0-140-97 kubenswrapper[2575]: E0721 21:00:42.868136 2575 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-140-97.ec2.internal\" not found" node="ip-10-0-140-97.ec2.internal" Jul 21 21:00:42.871045 ip-10-0-140-97 kubenswrapper[2575]: E0721 21:00:42.871031 2575 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-97.ec2.internal\" not found" Jul 21 21:00:42.897035 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:42.897016 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/df77ae6a34653e84fea42c7c1fb02209-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-140-97.ec2.internal\" (UID: \"df77ae6a34653e84fea42c7c1fb02209\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-97.ec2.internal" Jul 21 21:00:42.897115 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:42.897037 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/df77ae6a34653e84fea42c7c1fb02209-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-140-97.ec2.internal\" (UID: \"df77ae6a34653e84fea42c7c1fb02209\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-97.ec2.internal" Jul 21 21:00:42.897115 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:42.897054 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/810679f65980473b59e3cca08ad4f0fc-config\") pod \"kube-apiserver-proxy-ip-10-0-140-97.ec2.internal\" (UID: \"810679f65980473b59e3cca08ad4f0fc\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-140-97.ec2.internal" Jul 21 21:00:42.971475 ip-10-0-140-97 kubenswrapper[2575]: E0721 21:00:42.971454 2575 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-97.ec2.internal\" not found" Jul 21 21:00:42.997681 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:42.997640 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/df77ae6a34653e84fea42c7c1fb02209-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-140-97.ec2.internal\" (UID: \"df77ae6a34653e84fea42c7c1fb02209\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-97.ec2.internal" Jul 21 21:00:42.997681 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:42.997673 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/df77ae6a34653e84fea42c7c1fb02209-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-140-97.ec2.internal\" (UID: \"df77ae6a34653e84fea42c7c1fb02209\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-97.ec2.internal" Jul 21 21:00:42.997798 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:42.997692 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/810679f65980473b59e3cca08ad4f0fc-config\") pod \"kube-apiserver-proxy-ip-10-0-140-97.ec2.internal\" (UID: \"810679f65980473b59e3cca08ad4f0fc\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-140-97.ec2.internal" Jul 21 21:00:42.997798 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:42.997735 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/df77ae6a34653e84fea42c7c1fb02209-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-140-97.ec2.internal\" (UID: \"df77ae6a34653e84fea42c7c1fb02209\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-97.ec2.internal" Jul 21 21:00:42.997798 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:42.997752 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/df77ae6a34653e84fea42c7c1fb02209-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-140-97.ec2.internal\" (UID: \"df77ae6a34653e84fea42c7c1fb02209\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-97.ec2.internal" Jul 21 21:00:42.997885 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:42.997797 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/810679f65980473b59e3cca08ad4f0fc-config\") pod \"kube-apiserver-proxy-ip-10-0-140-97.ec2.internal\" (UID: \"810679f65980473b59e3cca08ad4f0fc\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-140-97.ec2.internal" Jul 21 21:00:43.071658 ip-10-0-140-97 kubenswrapper[2575]: E0721 21:00:43.071633 2575 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-97.ec2.internal\" not found" Jul 21 21:00:43.166928 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:43.166908 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-97.ec2.internal" Jul 21 21:00:43.171399 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:43.171385 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-97.ec2.internal" Jul 21 21:00:43.172411 ip-10-0-140-97 kubenswrapper[2575]: E0721 21:00:43.172397 2575 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-97.ec2.internal\" not found" Jul 21 21:00:43.273002 ip-10-0-140-97 kubenswrapper[2575]: E0721 21:00:43.272945 2575 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-97.ec2.internal\" not found" Jul 21 21:00:43.315326 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:43.315308 2575 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Jul 21 21:00:43.320934 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:43.320919 2575 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 21 21:00:43.342687 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:43.342662 2575 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Jul 21 21:00:43.342790 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:43.342772 2575 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 21 21:00:43.342825 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:43.342813 2575 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 21 21:00:43.342858 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:43.342816 2575 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 21 21:00:43.373827 ip-10-0-140-97 kubenswrapper[2575]: E0721 21:00:43.373806 2575 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-97.ec2.internal\" not found" Jul 21 21:00:43.415896 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:43.415871 2575 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-20 20:55:42 +0000 UTC" deadline="2027-12-22 06:06:17.705037516 +0000 UTC" Jul 21 21:00:43.415896 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:43.415893 2575 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="12441h5m34.289146797s" Jul 21 21:00:43.432964 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:43.432949 2575 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Jul 21 21:00:43.494204 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:43.494191 2575 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-97.ec2.internal" Jul 21 21:00:43.504254 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:43.504234 2575 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 21 21:00:43.505002 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:43.504990 2575 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-97.ec2.internal" Jul 21 21:00:43.512909 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:43.512898 2575 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 21 21:00:43.757481 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:00:43.757453 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddf77ae6a34653e84fea42c7c1fb02209.slice/crio-6c615e8d1f562a4cedeca4bf8630c76392a82b916a2c35e0154a38e909bd486c WatchSource:0}: Error finding container 6c615e8d1f562a4cedeca4bf8630c76392a82b916a2c35e0154a38e909bd486c: Status 404 returned error can't find the container with id 6c615e8d1f562a4cedeca4bf8630c76392a82b916a2c35e0154a38e909bd486c Jul 21 21:00:43.761602 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:43.761584 2575 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 21 21:00:43.818257 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:00:43.818234 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod810679f65980473b59e3cca08ad4f0fc.slice/crio-bc5b3d9e4ebaf24ab318f972e45c39d7ddcd9f8538c7e8d7c1beeffc1d53a0b6 WatchSource:0}: Error finding container bc5b3d9e4ebaf24ab318f972e45c39d7ddcd9f8538c7e8d7c1beeffc1d53a0b6: Status 404 returned error can't find the container with id bc5b3d9e4ebaf24ab318f972e45c39d7ddcd9f8538c7e8d7c1beeffc1d53a0b6 Jul 21 21:00:44.376745 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:44.376717 2575 apiserver.go:52] "Watching apiserver" Jul 21 21:00:44.386716 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:44.385218 2575 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Jul 21 21:00:44.386716 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:44.386028 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/konnectivity-agent-cq6dd","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x9lqv","openshift-dns/node-resolver-vdkfn","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-97.ec2.internal","openshift-network-diagnostics/network-check-target-4nvs6","openshift-network-operator/iptables-alerter-2czn5","openshift-ovn-kubernetes/ovnkube-node-m794j","kube-system/global-pull-secret-syncer-5694f","kube-system/kube-apiserver-proxy-ip-10-0-140-97.ec2.internal","openshift-cluster-node-tuning-operator/tuned-5pbk4","openshift-image-registry/node-ca-jhz69","openshift-multus/multus-additional-cni-plugins-952f6","openshift-multus/multus-lphp9","openshift-multus/network-metrics-daemon-6fx5v"] Jul 21 21:00:44.387828 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:44.387806 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-5pbk4" Jul 21 21:00:44.389488 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:44.389470 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-lphp9" Jul 21 21:00:44.390526 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:44.390508 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Jul 21 21:00:44.390682 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:44.390667 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-vdkfn" Jul 21 21:00:44.391804 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:44.391789 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6fx5v" Jul 21 21:00:44.391891 ip-10-0-140-97 kubenswrapper[2575]: E0721 21:00:44.391840 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6fx5v" podUID="67b28cbf-d3f3-4d32-b87f-f1e586e2faca" Jul 21 21:00:44.392288 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:44.392269 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Jul 21 21:00:44.392357 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:44.392269 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Jul 21 21:00:44.392944 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:44.392926 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4nvs6" Jul 21 21:00:44.393015 ip-10-0-140-97 kubenswrapper[2575]: E0721 21:00:44.392973 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4nvs6" podUID="7a92ce6a-922d-4196-819d-c03a22e230d1" Jul 21 21:00:44.393457 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:44.393436 2575 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Jul 21 21:00:44.394144 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:44.393996 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Jul 21 21:00:44.394144 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:44.394004 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Jul 21 21:00:44.394144 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:44.394083 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Jul 21 21:00:44.394320 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:44.394302 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-zwpcz\"" Jul 21 21:00:44.394418 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:44.394403 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Jul 21 21:00:44.394655 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:44.394558 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-2czn5" Jul 21 21:00:44.395300 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:44.395284 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-7k5kv\"" Jul 21 21:00:44.395627 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:44.395612 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5694f" Jul 21 21:00:44.395689 ip-10-0-140-97 kubenswrapper[2575]: E0721 21:00:44.395659 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5694f" podUID="e3dcf4f7-a30b-46f5-9c5d-a77f61d88c51" Jul 21 21:00:44.397155 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:44.397141 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-m794j" Jul 21 21:00:44.398745 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:44.398730 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x9lqv" Jul 21 21:00:44.399101 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:44.399084 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Jul 21 21:00:44.399302 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:44.399287 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Jul 21 21:00:44.399391 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:44.399308 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Jul 21 21:00:44.399391 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:44.399379 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Jul 21 21:00:44.399851 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:44.399837 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Jul 21 21:00:44.399957 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:44.399938 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Jul 21 21:00:44.400060 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:44.400047 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-cq6dd" Jul 21 21:00:44.400527 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:44.400506 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Jul 21 21:00:44.400656 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:44.400587 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Jul 21 21:00:44.400722 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:44.400700 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-5tpsj\"" Jul 21 21:00:44.401330 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:44.401316 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-jhz69" Jul 21 21:00:44.401783 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:44.401769 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-k9h4q\"" Jul 21 21:00:44.402386 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:44.402369 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6bb08697-52d4-4ebf-b058-5771faf60563-etc-kubernetes\") pod \"multus-lphp9\" (UID: \"6bb08697-52d4-4ebf-b058-5771faf60563\") " pod="openshift-multus/multus-lphp9" Jul 21 21:00:44.402434 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:44.402400 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/3877fdb0-b49b-444a-8f8c-597d3f935383-hosts-file\") pod \"node-resolver-vdkfn\" (UID: \"3877fdb0-b49b-444a-8f8c-597d3f935383\") " pod="openshift-dns/node-resolver-vdkfn" Jul 21 21:00:44.402434 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:44.402425 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/67b28cbf-d3f3-4d32-b87f-f1e586e2faca-metrics-certs\") pod \"network-metrics-daemon-6fx5v\" (UID: \"67b28cbf-d3f3-4d32-b87f-f1e586e2faca\") " pod="openshift-multus/network-metrics-daemon-6fx5v" Jul 21 21:00:44.402497 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:44.402461 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/1fd6b467-f1bf-45b5-8d30-96201ffccd36-iptables-alerter-script\") pod \"iptables-alerter-2czn5\" (UID: \"1fd6b467-f1bf-45b5-8d30-96201ffccd36\") " pod="openshift-network-operator/iptables-alerter-2czn5" Jul 21 21:00:44.402555 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:44.402495 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/6bb08697-52d4-4ebf-b058-5771faf60563-host-var-lib-cni-bin\") pod \"multus-lphp9\" (UID: \"6bb08697-52d4-4ebf-b058-5771faf60563\") " pod="openshift-multus/multus-lphp9" Jul 21 21:00:44.402555 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:44.402546 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sgtlj\" (UniqueName: \"kubernetes.io/projected/6bb08697-52d4-4ebf-b058-5771faf60563-kube-api-access-sgtlj\") pod \"multus-lphp9\" (UID: \"6bb08697-52d4-4ebf-b058-5771faf60563\") " pod="openshift-multus/multus-lphp9" Jul 21 21:00:44.402641 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:44.402572 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3877fdb0-b49b-444a-8f8c-597d3f935383-tmp-dir\") pod \"node-resolver-vdkfn\" (UID: \"3877fdb0-b49b-444a-8f8c-597d3f935383\") " pod="openshift-dns/node-resolver-vdkfn" Jul 21 21:00:44.402641 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:44.402604 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/b69ef6b6-fca5-47e8-abc5-566b34e77058-etc-tuned\") pod \"tuned-5pbk4\" (UID: \"b69ef6b6-fca5-47e8-abc5-566b34e77058\") " pod="openshift-cluster-node-tuning-operator/tuned-5pbk4" Jul 21 21:00:44.402704 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:44.402656 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b69ef6b6-fca5-47e8-abc5-566b34e77058-tmp\") pod \"tuned-5pbk4\" (UID: \"b69ef6b6-fca5-47e8-abc5-566b34e77058\") " pod="openshift-cluster-node-tuning-operator/tuned-5pbk4" Jul 21 21:00:44.402704 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:44.402663 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-952f6" Jul 21 21:00:44.402704 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:44.402684 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6bb08697-52d4-4ebf-b058-5771faf60563-system-cni-dir\") pod \"multus-lphp9\" (UID: \"6bb08697-52d4-4ebf-b058-5771faf60563\") " pod="openshift-multus/multus-lphp9" Jul 21 21:00:44.402814 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:44.402709 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6bb08697-52d4-4ebf-b058-5771faf60563-multus-cni-dir\") pod \"multus-lphp9\" (UID: \"6bb08697-52d4-4ebf-b058-5771faf60563\") " pod="openshift-multus/multus-lphp9" Jul 21 21:00:44.402814 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:44.402731 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6bb08697-52d4-4ebf-b058-5771faf60563-host-var-lib-kubelet\") pod \"multus-lphp9\" (UID: \"6bb08697-52d4-4ebf-b058-5771faf60563\") " pod="openshift-multus/multus-lphp9" Jul 21 21:00:44.402814 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:44.402752 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/b69ef6b6-fca5-47e8-abc5-566b34e77058-etc-systemd\") pod \"tuned-5pbk4\" (UID: \"b69ef6b6-fca5-47e8-abc5-566b34e77058\") " pod="openshift-cluster-node-tuning-operator/tuned-5pbk4" Jul 21 21:00:44.402814 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:44.402774 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/b69ef6b6-fca5-47e8-abc5-566b34e77058-lib-modules\") pod \"tuned-5pbk4\" (UID: \"b69ef6b6-fca5-47e8-abc5-566b34e77058\") " pod="openshift-cluster-node-tuning-operator/tuned-5pbk4" Jul 21 21:00:44.402982 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:44.402795 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/6bb08697-52d4-4ebf-b058-5771faf60563-os-release\") pod \"multus-lphp9\" (UID: \"6bb08697-52d4-4ebf-b058-5771faf60563\") " pod="openshift-multus/multus-lphp9" Jul 21 21:00:44.402982 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:44.402836 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/6bb08697-52d4-4ebf-b058-5771faf60563-multus-socket-dir-parent\") pod \"multus-lphp9\" (UID: \"6bb08697-52d4-4ebf-b058-5771faf60563\") " pod="openshift-multus/multus-lphp9" Jul 21 21:00:44.402982 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:44.402860 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/6bb08697-52d4-4ebf-b058-5771faf60563-host-run-k8s-cni-cncf-io\") pod \"multus-lphp9\" (UID: \"6bb08697-52d4-4ebf-b058-5771faf60563\") " pod="openshift-multus/multus-lphp9" Jul 21 21:00:44.402982 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:44.402897 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/6bb08697-52d4-4ebf-b058-5771faf60563-hostroot\") pod \"multus-lphp9\" (UID: \"6bb08697-52d4-4ebf-b058-5771faf60563\") " pod="openshift-multus/multus-lphp9" Jul 21 21:00:44.402982 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:44.402930 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vkwg6\" (UniqueName: \"kubernetes.io/projected/3877fdb0-b49b-444a-8f8c-597d3f935383-kube-api-access-vkwg6\") pod \"node-resolver-vdkfn\" (UID: \"3877fdb0-b49b-444a-8f8c-597d3f935383\") " pod="openshift-dns/node-resolver-vdkfn" Jul 21 21:00:44.402982 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:44.402955 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/b69ef6b6-fca5-47e8-abc5-566b34e77058-etc-sysconfig\") pod \"tuned-5pbk4\" (UID: \"b69ef6b6-fca5-47e8-abc5-566b34e77058\") " pod="openshift-cluster-node-tuning-operator/tuned-5pbk4" Jul 21 21:00:44.402982 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:44.402979 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b69ef6b6-fca5-47e8-abc5-566b34e77058-sys\") pod \"tuned-5pbk4\" (UID: \"b69ef6b6-fca5-47e8-abc5-566b34e77058\") " pod="openshift-cluster-node-tuning-operator/tuned-5pbk4" Jul 21 21:00:44.403311 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:44.403002 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/6bb08697-52d4-4ebf-b058-5771faf60563-cni-binary-copy\") pod \"multus-lphp9\" (UID: \"6bb08697-52d4-4ebf-b058-5771faf60563\") " pod="openshift-multus/multus-lphp9" Jul 21 21:00:44.403311 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:44.403024 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/6bb08697-52d4-4ebf-b058-5771faf60563-multus-daemon-config\") pod \"multus-lphp9\" (UID: \"6bb08697-52d4-4ebf-b058-5771faf60563\") " pod="openshift-multus/multus-lphp9" Jul 21 21:00:44.403311 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:44.403048 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-djhxf\" (UniqueName: \"kubernetes.io/projected/7a92ce6a-922d-4196-819d-c03a22e230d1-kube-api-access-djhxf\") pod \"network-check-target-4nvs6\" (UID: \"7a92ce6a-922d-4196-819d-c03a22e230d1\") " pod="openshift-network-diagnostics/network-check-target-4nvs6" Jul 21 21:00:44.403311 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:44.403095 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9hd59\" (UniqueName: \"kubernetes.io/projected/1fd6b467-f1bf-45b5-8d30-96201ffccd36-kube-api-access-9hd59\") pod \"iptables-alerter-2czn5\" (UID: \"1fd6b467-f1bf-45b5-8d30-96201ffccd36\") " pod="openshift-network-operator/iptables-alerter-2czn5" Jul 21 21:00:44.403311 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:44.403120 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b69ef6b6-fca5-47e8-abc5-566b34e77058-etc-kubernetes\") pod \"tuned-5pbk4\" (UID: \"b69ef6b6-fca5-47e8-abc5-566b34e77058\") " pod="openshift-cluster-node-tuning-operator/tuned-5pbk4" Jul 21 21:00:44.403311 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:44.403142 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/b69ef6b6-fca5-47e8-abc5-566b34e77058-etc-sysctl-conf\") pod \"tuned-5pbk4\" (UID: \"b69ef6b6-fca5-47e8-abc5-566b34e77058\") " pod="openshift-cluster-node-tuning-operator/tuned-5pbk4" Jul 21 21:00:44.403311 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:44.403170 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f45zk\" (UniqueName: \"kubernetes.io/projected/b69ef6b6-fca5-47e8-abc5-566b34e77058-kube-api-access-f45zk\") pod \"tuned-5pbk4\" (UID: \"b69ef6b6-fca5-47e8-abc5-566b34e77058\") " pod="openshift-cluster-node-tuning-operator/tuned-5pbk4" Jul 21 21:00:44.403710 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:44.403658 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/6bb08697-52d4-4ebf-b058-5771faf60563-host-run-multus-certs\") pod \"multus-lphp9\" (UID: \"6bb08697-52d4-4ebf-b058-5771faf60563\") " pod="openshift-multus/multus-lphp9" Jul 21 21:00:44.403710 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:44.403696 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/b69ef6b6-fca5-47e8-abc5-566b34e77058-etc-modprobe-d\") pod \"tuned-5pbk4\" (UID: \"b69ef6b6-fca5-47e8-abc5-566b34e77058\") " pod="openshift-cluster-node-tuning-operator/tuned-5pbk4" Jul 21 21:00:44.403805 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:44.403717 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/b69ef6b6-fca5-47e8-abc5-566b34e77058-run\") pod \"tuned-5pbk4\" (UID: \"b69ef6b6-fca5-47e8-abc5-566b34e77058\") " pod="openshift-cluster-node-tuning-operator/tuned-5pbk4" Jul 21 21:00:44.403805 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:44.403741 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/6bb08697-52d4-4ebf-b058-5771faf60563-cnibin\") pod \"multus-lphp9\" (UID: \"6bb08697-52d4-4ebf-b058-5771faf60563\") " pod="openshift-multus/multus-lphp9" Jul 21 21:00:44.403805 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:44.403782 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/6bb08697-52d4-4ebf-b058-5771faf60563-multus-conf-dir\") pod \"multus-lphp9\" (UID: \"6bb08697-52d4-4ebf-b058-5771faf60563\") " pod="openshift-multus/multus-lphp9" Jul 21 21:00:44.403907 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:44.403820 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wvsds\" (UniqueName: \"kubernetes.io/projected/67b28cbf-d3f3-4d32-b87f-f1e586e2faca-kube-api-access-wvsds\") pod \"network-metrics-daemon-6fx5v\" (UID: \"67b28cbf-d3f3-4d32-b87f-f1e586e2faca\") " pod="openshift-multus/network-metrics-daemon-6fx5v" Jul 21 21:00:44.403907 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:44.403842 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b69ef6b6-fca5-47e8-abc5-566b34e77058-var-lib-kubelet\") pod \"tuned-5pbk4\" (UID: \"b69ef6b6-fca5-47e8-abc5-566b34e77058\") " pod="openshift-cluster-node-tuning-operator/tuned-5pbk4" Jul 21 21:00:44.403907 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:44.403856 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/b69ef6b6-fca5-47e8-abc5-566b34e77058-host\") pod \"tuned-5pbk4\" (UID: \"b69ef6b6-fca5-47e8-abc5-566b34e77058\") " pod="openshift-cluster-node-tuning-operator/tuned-5pbk4" Jul 21 21:00:44.404037 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:44.403906 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/1fd6b467-f1bf-45b5-8d30-96201ffccd36-host-slash\") pod \"iptables-alerter-2czn5\" (UID: \"1fd6b467-f1bf-45b5-8d30-96201ffccd36\") " pod="openshift-network-operator/iptables-alerter-2czn5" Jul 21 21:00:44.404037 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:44.403932 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/b69ef6b6-fca5-47e8-abc5-566b34e77058-etc-sysctl-d\") pod \"tuned-5pbk4\" (UID: \"b69ef6b6-fca5-47e8-abc5-566b34e77058\") " pod="openshift-cluster-node-tuning-operator/tuned-5pbk4" Jul 21 21:00:44.404037 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:44.403953 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/6bb08697-52d4-4ebf-b058-5771faf60563-host-run-netns\") pod \"multus-lphp9\" (UID: \"6bb08697-52d4-4ebf-b058-5771faf60563\") " pod="openshift-multus/multus-lphp9" Jul 21 21:00:44.404037 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:44.403983 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/6bb08697-52d4-4ebf-b058-5771faf60563-host-var-lib-cni-multus\") pod \"multus-lphp9\" (UID: \"6bb08697-52d4-4ebf-b058-5771faf60563\") " pod="openshift-multus/multus-lphp9" Jul 21 21:00:44.404236 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:44.404216 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Jul 21 21:00:44.404354 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:44.404236 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-gmfpv\"" Jul 21 21:00:44.404354 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:44.404265 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Jul 21 21:00:44.404354 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:44.404274 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Jul 21 21:00:44.404354 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:44.404281 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-2rmvn\"" Jul 21 21:00:44.404658 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:44.404471 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-zlzzs\"" Jul 21 21:00:44.404658 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:44.404477 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Jul 21 21:00:44.404658 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:44.404569 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Jul 21 21:00:44.404830 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:44.404750 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Jul 21 21:00:44.404936 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:44.404921 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Jul 21 21:00:44.413988 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:44.413972 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Jul 21 21:00:44.414056 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:44.414025 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Jul 21 21:00:44.417344 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:44.417263 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-77hx6\"" Jul 21 21:00:44.417344 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:44.417282 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Jul 21 21:00:44.417344 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:44.417341 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-dcsnq\"" Jul 21 21:00:44.417549 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:44.417361 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Jul 21 21:00:44.417549 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:44.417341 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Jul 21 21:00:44.427341 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:44.427319 2575 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 21 21:00:44.495488 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:44.495468 2575 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Jul 21 21:00:44.504145 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:44.504128 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/1fd6b467-f1bf-45b5-8d30-96201ffccd36-host-slash\") pod \"iptables-alerter-2czn5\" (UID: \"1fd6b467-f1bf-45b5-8d30-96201ffccd36\") " pod="openshift-network-operator/iptables-alerter-2czn5" Jul 21 21:00:44.504257 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:44.504156 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/6bb08697-52d4-4ebf-b058-5771faf60563-host-var-lib-cni-multus\") pod \"multus-lphp9\" (UID: \"6bb08697-52d4-4ebf-b058-5771faf60563\") " pod="openshift-multus/multus-lphp9" Jul 21 21:00:44.504257 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:44.504183 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/3877fdb0-b49b-444a-8f8c-597d3f935383-hosts-file\") pod \"node-resolver-vdkfn\" (UID: \"3877fdb0-b49b-444a-8f8c-597d3f935383\") " pod="openshift-dns/node-resolver-vdkfn" Jul 21 21:00:44.504257 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:44.504200 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/e3dcf4f7-a30b-46f5-9c5d-a77f61d88c51-dbus\") pod \"global-pull-secret-syncer-5694f\" (UID: \"e3dcf4f7-a30b-46f5-9c5d-a77f61d88c51\") " pod="kube-system/global-pull-secret-syncer-5694f" Jul 21 21:00:44.504257 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:44.504216 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/6bb08697-52d4-4ebf-b058-5771faf60563-host-var-lib-cni-bin\") pod \"multus-lphp9\" (UID: \"6bb08697-52d4-4ebf-b058-5771faf60563\") " pod="openshift-multus/multus-lphp9" Jul 21 21:00:44.504257 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:44.504233 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sgtlj\" (UniqueName: \"kubernetes.io/projected/6bb08697-52d4-4ebf-b058-5771faf60563-kube-api-access-sgtlj\") pod \"multus-lphp9\" (UID: \"6bb08697-52d4-4ebf-b058-5771faf60563\") " pod="openshift-multus/multus-lphp9" Jul 21 21:00:44.504257 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:44.504240 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/1fd6b467-f1bf-45b5-8d30-96201ffccd36-host-slash\") pod \"iptables-alerter-2czn5\" (UID: \"1fd6b467-f1bf-45b5-8d30-96201ffccd36\") " pod="openshift-network-operator/iptables-alerter-2czn5" Jul 21 21:00:44.504257 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:44.504254 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/edd93c23-9f22-4d2c-9342-7f6c398fd6bb-var-lib-openvswitch\") pod \"ovnkube-node-m794j\" (UID: \"edd93c23-9f22-4d2c-9342-7f6c398fd6bb\") " pod="openshift-ovn-kubernetes/ovnkube-node-m794j" Jul 21 21:00:44.504257 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:44.504253 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/6bb08697-52d4-4ebf-b058-5771faf60563-host-var-lib-cni-multus\") pod \"multus-lphp9\" (UID: \"6bb08697-52d4-4ebf-b058-5771faf60563\") " pod="openshift-multus/multus-lphp9" Jul 21 21:00:44.504556 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:44.504272 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/3877fdb0-b49b-444a-8f8c-597d3f935383-hosts-file\") pod \"node-resolver-vdkfn\" (UID: \"3877fdb0-b49b-444a-8f8c-597d3f935383\") " pod="openshift-dns/node-resolver-vdkfn" Jul 21 21:00:44.504556 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:44.504298 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/6bb08697-52d4-4ebf-b058-5771faf60563-host-var-lib-cni-bin\") pod \"multus-lphp9\" (UID: \"6bb08697-52d4-4ebf-b058-5771faf60563\") " pod="openshift-multus/multus-lphp9" Jul 21 21:00:44.504556 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:44.504302 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-98tgq\" (UniqueName: \"kubernetes.io/projected/6a0eccd2-79bf-4f17-8217-6533fff9fbbb-kube-api-access-98tgq\") pod \"aws-ebs-csi-driver-node-x9lqv\" (UID: \"6a0eccd2-79bf-4f17-8217-6533fff9fbbb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x9lqv" Jul 21 21:00:44.504556 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:44.504343 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/d648b77f-43a8-4540-b59c-63768e78c245-os-release\") pod \"multus-additional-cni-plugins-952f6\" (UID: \"d648b77f-43a8-4540-b59c-63768e78c245\") " pod="openshift-multus/multus-additional-cni-plugins-952f6" Jul 21 21:00:44.504556 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:44.504361 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/b69ef6b6-fca5-47e8-abc5-566b34e77058-etc-tuned\") pod \"tuned-5pbk4\" (UID: \"b69ef6b6-fca5-47e8-abc5-566b34e77058\") " pod="openshift-cluster-node-tuning-operator/tuned-5pbk4" Jul 21 21:00:44.504556 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:44.504378 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6bb08697-52d4-4ebf-b058-5771faf60563-system-cni-dir\") pod \"multus-lphp9\" (UID: \"6bb08697-52d4-4ebf-b058-5771faf60563\") " pod="openshift-multus/multus-lphp9" Jul 21 21:00:44.504556 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:44.504397 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6bb08697-52d4-4ebf-b058-5771faf60563-multus-cni-dir\") pod \"multus-lphp9\" (UID: \"6bb08697-52d4-4ebf-b058-5771faf60563\") " pod="openshift-multus/multus-lphp9" Jul 21 21:00:44.504556 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:44.504437 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6bb08697-52d4-4ebf-b058-5771faf60563-host-var-lib-kubelet\") pod \"multus-lphp9\" (UID: \"6bb08697-52d4-4ebf-b058-5771faf60563\") " pod="openshift-multus/multus-lphp9" Jul 21 21:00:44.504556 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:44.504461 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6bb08697-52d4-4ebf-b058-5771faf60563-system-cni-dir\") pod \"multus-lphp9\" (UID: \"6bb08697-52d4-4ebf-b058-5771faf60563\") " pod="openshift-multus/multus-lphp9" Jul 21 21:00:44.504556 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:44.504477 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/edd93c23-9f22-4d2c-9342-7f6c398fd6bb-host-run-netns\") pod \"ovnkube-node-m794j\" (UID: \"edd93c23-9f22-4d2c-9342-7f6c398fd6bb\") " pod="openshift-ovn-kubernetes/ovnkube-node-m794j" Jul 21 21:00:44.504556 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:44.504502 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/edd93c23-9f22-4d2c-9342-7f6c398fd6bb-node-log\") pod \"ovnkube-node-m794j\" (UID: \"edd93c23-9f22-4d2c-9342-7f6c398fd6bb\") " pod="openshift-ovn-kubernetes/ovnkube-node-m794j" Jul 21 21:00:44.504556 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:44.504519 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/edd93c23-9f22-4d2c-9342-7f6c398fd6bb-ovn-node-metrics-cert\") pod \"ovnkube-node-m794j\" (UID: \"edd93c23-9f22-4d2c-9342-7f6c398fd6bb\") " pod="openshift-ovn-kubernetes/ovnkube-node-m794j" Jul 21 21:00:44.504556 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:44.504516 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6bb08697-52d4-4ebf-b058-5771faf60563-host-var-lib-kubelet\") pod \"multus-lphp9\" (UID: \"6bb08697-52d4-4ebf-b058-5771faf60563\") " pod="openshift-multus/multus-lphp9" Jul 21 21:00:44.504556 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:44.504553 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ce7f9a12-9e5b-4634-83d3-babc1e854a66-host\") pod \"node-ca-jhz69\" (UID: \"ce7f9a12-9e5b-4634-83d3-babc1e854a66\") " pod="openshift-image-registry/node-ca-jhz69" Jul 21 21:00:44.505223 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:44.504590 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/6bb08697-52d4-4ebf-b058-5771faf60563-os-release\") pod \"multus-lphp9\" (UID: \"6bb08697-52d4-4ebf-b058-5771faf60563\") " pod="openshift-multus/multus-lphp9" Jul 21 21:00:44.505223 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:44.504617 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/6bb08697-52d4-4ebf-b058-5771faf60563-multus-socket-dir-parent\") pod \"multus-lphp9\" (UID: \"6bb08697-52d4-4ebf-b058-5771faf60563\") " pod="openshift-multus/multus-lphp9" Jul 21 21:00:44.505223 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:44.504642 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/6bb08697-52d4-4ebf-b058-5771faf60563-hostroot\") pod \"multus-lphp9\" (UID: \"6bb08697-52d4-4ebf-b058-5771faf60563\") " pod="openshift-multus/multus-lphp9" Jul 21 21:00:44.505223 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:44.504663 2575 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Jul 21 21:00:44.505223 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:44.504672 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6bb08697-52d4-4ebf-b058-5771faf60563-multus-cni-dir\") pod \"multus-lphp9\" (UID: \"6bb08697-52d4-4ebf-b058-5771faf60563\") " pod="openshift-multus/multus-lphp9" Jul 21 21:00:44.505223 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:44.504689 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/6bb08697-52d4-4ebf-b058-5771faf60563-hostroot\") pod \"multus-lphp9\" (UID: \"6bb08697-52d4-4ebf-b058-5771faf60563\") " pod="openshift-multus/multus-lphp9" Jul 21 21:00:44.505223 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:44.504680 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/6bb08697-52d4-4ebf-b058-5771faf60563-os-release\") pod \"multus-lphp9\" (UID: \"6bb08697-52d4-4ebf-b058-5771faf60563\") " pod="openshift-multus/multus-lphp9" Jul 21 21:00:44.505223 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:44.504668 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/edd93c23-9f22-4d2c-9342-7f6c398fd6bb-host-cni-netd\") pod \"ovnkube-node-m794j\" (UID: \"edd93c23-9f22-4d2c-9342-7f6c398fd6bb\") " pod="openshift-ovn-kubernetes/ovnkube-node-m794j" Jul 21 21:00:44.505223 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:44.504736 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/b69ef6b6-fca5-47e8-abc5-566b34e77058-etc-sysconfig\") pod \"tuned-5pbk4\" (UID: \"b69ef6b6-fca5-47e8-abc5-566b34e77058\") " pod="openshift-cluster-node-tuning-operator/tuned-5pbk4" Jul 21 21:00:44.505223 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:44.504711 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/6bb08697-52d4-4ebf-b058-5771faf60563-multus-socket-dir-parent\") pod \"multus-lphp9\" (UID: \"6bb08697-52d4-4ebf-b058-5771faf60563\") " pod="openshift-multus/multus-lphp9" Jul 21 21:00:44.505223 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:44.504752 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b69ef6b6-fca5-47e8-abc5-566b34e77058-sys\") pod \"tuned-5pbk4\" (UID: \"b69ef6b6-fca5-47e8-abc5-566b34e77058\") " pod="openshift-cluster-node-tuning-operator/tuned-5pbk4" Jul 21 21:00:44.505223 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:44.504776 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b69ef6b6-fca5-47e8-abc5-566b34e77058-sys\") pod \"tuned-5pbk4\" (UID: \"b69ef6b6-fca5-47e8-abc5-566b34e77058\") " pod="openshift-cluster-node-tuning-operator/tuned-5pbk4" Jul 21 21:00:44.505223 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:44.504797 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/edd93c23-9f22-4d2c-9342-7f6c398fd6bb-run-ovn\") pod \"ovnkube-node-m794j\" (UID: \"edd93c23-9f22-4d2c-9342-7f6c398fd6bb\") " pod="openshift-ovn-kubernetes/ovnkube-node-m794j" Jul 21 21:00:44.505223 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:44.504808 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/b69ef6b6-fca5-47e8-abc5-566b34e77058-etc-sysconfig\") pod \"tuned-5pbk4\" (UID: \"b69ef6b6-fca5-47e8-abc5-566b34e77058\") " pod="openshift-cluster-node-tuning-operator/tuned-5pbk4" Jul 21 21:00:44.505223 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:44.504822 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/edd93c23-9f22-4d2c-9342-7f6c398fd6bb-host-run-ovn-kubernetes\") pod \"ovnkube-node-m794j\" (UID: \"edd93c23-9f22-4d2c-9342-7f6c398fd6bb\") " pod="openshift-ovn-kubernetes/ovnkube-node-m794j" Jul 21 21:00:44.505934 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:44.504854 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/6a0eccd2-79bf-4f17-8217-6533fff9fbbb-etc-selinux\") pod \"aws-ebs-csi-driver-node-x9lqv\" (UID: \"6a0eccd2-79bf-4f17-8217-6533fff9fbbb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x9lqv" Jul 21 21:00:44.505934 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:44.504882 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-djhxf\" (UniqueName: \"kubernetes.io/projected/7a92ce6a-922d-4196-819d-c03a22e230d1-kube-api-access-djhxf\") pod \"network-check-target-4nvs6\" (UID: \"7a92ce6a-922d-4196-819d-c03a22e230d1\") " pod="openshift-network-diagnostics/network-check-target-4nvs6" Jul 21 21:00:44.505934 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:44.504901 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b69ef6b6-fca5-47e8-abc5-566b34e77058-etc-kubernetes\") pod \"tuned-5pbk4\" (UID: \"b69ef6b6-fca5-47e8-abc5-566b34e77058\") " pod="openshift-cluster-node-tuning-operator/tuned-5pbk4" Jul 21 21:00:44.505934 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:44.504933 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b69ef6b6-fca5-47e8-abc5-566b34e77058-etc-kubernetes\") pod \"tuned-5pbk4\" (UID: \"b69ef6b6-fca5-47e8-abc5-566b34e77058\") " pod="openshift-cluster-node-tuning-operator/tuned-5pbk4" Jul 21 21:00:44.505934 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:44.504973 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/b69ef6b6-fca5-47e8-abc5-566b34e77058-etc-sysctl-conf\") pod \"tuned-5pbk4\" (UID: \"b69ef6b6-fca5-47e8-abc5-566b34e77058\") " pod="openshift-cluster-node-tuning-operator/tuned-5pbk4" Jul 21 21:00:44.505934 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:44.504991 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f45zk\" (UniqueName: \"kubernetes.io/projected/b69ef6b6-fca5-47e8-abc5-566b34e77058-kube-api-access-f45zk\") pod \"tuned-5pbk4\" (UID: \"b69ef6b6-fca5-47e8-abc5-566b34e77058\") " pod="openshift-cluster-node-tuning-operator/tuned-5pbk4" Jul 21 21:00:44.505934 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:44.505016 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/6bb08697-52d4-4ebf-b058-5771faf60563-host-run-multus-certs\") pod \"multus-lphp9\" (UID: \"6bb08697-52d4-4ebf-b058-5771faf60563\") " pod="openshift-multus/multus-lphp9" Jul 21 21:00:44.505934 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:44.505037 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7q2jm\" (UniqueName: \"kubernetes.io/projected/ce7f9a12-9e5b-4634-83d3-babc1e854a66-kube-api-access-7q2jm\") pod \"node-ca-jhz69\" (UID: \"ce7f9a12-9e5b-4634-83d3-babc1e854a66\") " pod="openshift-image-registry/node-ca-jhz69" Jul 21 21:00:44.505934 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:44.505054 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d648b77f-43a8-4540-b59c-63768e78c245-cnibin\") pod \"multus-additional-cni-plugins-952f6\" (UID: \"d648b77f-43a8-4540-b59c-63768e78c245\") " pod="openshift-multus/multus-additional-cni-plugins-952f6" Jul 21 21:00:44.505934 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:44.505060 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/b69ef6b6-fca5-47e8-abc5-566b34e77058-etc-sysctl-conf\") pod \"tuned-5pbk4\" (UID: \"b69ef6b6-fca5-47e8-abc5-566b34e77058\") " pod="openshift-cluster-node-tuning-operator/tuned-5pbk4" Jul 21 21:00:44.505934 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:44.505071 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d648b77f-43a8-4540-b59c-63768e78c245-tuning-conf-dir\") pod \"multus-additional-cni-plugins-952f6\" (UID: \"d648b77f-43a8-4540-b59c-63768e78c245\") " pod="openshift-multus/multus-additional-cni-plugins-952f6" Jul 21 21:00:44.505934 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:44.505094 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/6bb08697-52d4-4ebf-b058-5771faf60563-multus-conf-dir\") pod \"multus-lphp9\" (UID: \"6bb08697-52d4-4ebf-b058-5771faf60563\") " pod="openshift-multus/multus-lphp9" Jul 21 21:00:44.505934 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:44.505106 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/6bb08697-52d4-4ebf-b058-5771faf60563-host-run-multus-certs\") pod \"multus-lphp9\" (UID: \"6bb08697-52d4-4ebf-b058-5771faf60563\") " pod="openshift-multus/multus-lphp9" Jul 21 21:00:44.505934 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:44.505137 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d648b77f-43a8-4540-b59c-63768e78c245-cni-binary-copy\") pod \"multus-additional-cni-plugins-952f6\" (UID: \"d648b77f-43a8-4540-b59c-63768e78c245\") " pod="openshift-multus/multus-additional-cni-plugins-952f6" Jul 21 21:00:44.506518 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:44.505152 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/6bb08697-52d4-4ebf-b058-5771faf60563-multus-conf-dir\") pod \"multus-lphp9\" (UID: \"6bb08697-52d4-4ebf-b058-5771faf60563\") " pod="openshift-multus/multus-lphp9" Jul 21 21:00:44.506518 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:44.505192 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/b69ef6b6-fca5-47e8-abc5-566b34e77058-host\") pod \"tuned-5pbk4\" (UID: \"b69ef6b6-fca5-47e8-abc5-566b34e77058\") " pod="openshift-cluster-node-tuning-operator/tuned-5pbk4" Jul 21 21:00:44.506518 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:44.505211 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/b69ef6b6-fca5-47e8-abc5-566b34e77058-host\") pod \"tuned-5pbk4\" (UID: \"b69ef6b6-fca5-47e8-abc5-566b34e77058\") " pod="openshift-cluster-node-tuning-operator/tuned-5pbk4" Jul 21 21:00:44.506518 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:44.505240 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/edd93c23-9f22-4d2c-9342-7f6c398fd6bb-ovnkube-script-lib\") pod \"ovnkube-node-m794j\" (UID: \"edd93c23-9f22-4d2c-9342-7f6c398fd6bb\") " pod="openshift-ovn-kubernetes/ovnkube-node-m794j" Jul 21 21:00:44.506518 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:44.505262 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g69kf\" (UniqueName: \"kubernetes.io/projected/d648b77f-43a8-4540-b59c-63768e78c245-kube-api-access-g69kf\") pod \"multus-additional-cni-plugins-952f6\" (UID: \"d648b77f-43a8-4540-b59c-63768e78c245\") " pod="openshift-multus/multus-additional-cni-plugins-952f6" Jul 21 21:00:44.506518 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:44.505301 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/b69ef6b6-fca5-47e8-abc5-566b34e77058-etc-sysctl-d\") pod \"tuned-5pbk4\" (UID: \"b69ef6b6-fca5-47e8-abc5-566b34e77058\") " pod="openshift-cluster-node-tuning-operator/tuned-5pbk4" Jul 21 21:00:44.506518 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:44.505327 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/6bb08697-52d4-4ebf-b058-5771faf60563-host-run-netns\") pod \"multus-lphp9\" (UID: \"6bb08697-52d4-4ebf-b058-5771faf60563\") " pod="openshift-multus/multus-lphp9" Jul 21 21:00:44.506518 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:44.505358 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/6bb08697-52d4-4ebf-b058-5771faf60563-host-run-netns\") pod \"multus-lphp9\" (UID: \"6bb08697-52d4-4ebf-b058-5771faf60563\") " pod="openshift-multus/multus-lphp9" Jul 21 21:00:44.506518 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:44.505357 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6bb08697-52d4-4ebf-b058-5771faf60563-etc-kubernetes\") pod \"multus-lphp9\" (UID: \"6bb08697-52d4-4ebf-b058-5771faf60563\") " pod="openshift-multus/multus-lphp9" Jul 21 21:00:44.506518 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:44.505382 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/b69ef6b6-fca5-47e8-abc5-566b34e77058-etc-sysctl-d\") pod \"tuned-5pbk4\" (UID: \"b69ef6b6-fca5-47e8-abc5-566b34e77058\") " pod="openshift-cluster-node-tuning-operator/tuned-5pbk4" Jul 21 21:00:44.506518 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:44.505397 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6bb08697-52d4-4ebf-b058-5771faf60563-etc-kubernetes\") pod \"multus-lphp9\" (UID: \"6bb08697-52d4-4ebf-b058-5771faf60563\") " pod="openshift-multus/multus-lphp9" Jul 21 21:00:44.506518 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:44.505388 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/edd93c23-9f22-4d2c-9342-7f6c398fd6bb-host-slash\") pod \"ovnkube-node-m794j\" (UID: \"edd93c23-9f22-4d2c-9342-7f6c398fd6bb\") " pod="openshift-ovn-kubernetes/ovnkube-node-m794j" Jul 21 21:00:44.506518 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:44.505437 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/edd93c23-9f22-4d2c-9342-7f6c398fd6bb-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-m794j\" (UID: \"edd93c23-9f22-4d2c-9342-7f6c398fd6bb\") " pod="openshift-ovn-kubernetes/ovnkube-node-m794j" Jul 21 21:00:44.506518 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:44.505463 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/edd93c23-9f22-4d2c-9342-7f6c398fd6bb-log-socket\") pod \"ovnkube-node-m794j\" (UID: \"edd93c23-9f22-4d2c-9342-7f6c398fd6bb\") " pod="openshift-ovn-kubernetes/ovnkube-node-m794j" Jul 21 21:00:44.507079 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:44.505488 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/67b28cbf-d3f3-4d32-b87f-f1e586e2faca-metrics-certs\") pod \"network-metrics-daemon-6fx5v\" (UID: \"67b28cbf-d3f3-4d32-b87f-f1e586e2faca\") " pod="openshift-multus/network-metrics-daemon-6fx5v" Jul 21 21:00:44.507079 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:44.505511 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/1fd6b467-f1bf-45b5-8d30-96201ffccd36-iptables-alerter-script\") pod \"iptables-alerter-2czn5\" (UID: \"1fd6b467-f1bf-45b5-8d30-96201ffccd36\") " pod="openshift-network-operator/iptables-alerter-2czn5" Jul 21 21:00:44.507079 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:44.505555 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3877fdb0-b49b-444a-8f8c-597d3f935383-tmp-dir\") pod \"node-resolver-vdkfn\" (UID: \"3877fdb0-b49b-444a-8f8c-597d3f935383\") " pod="openshift-dns/node-resolver-vdkfn" Jul 21 21:00:44.507079 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:44.505593 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/edd93c23-9f22-4d2c-9342-7f6c398fd6bb-etc-openvswitch\") pod \"ovnkube-node-m794j\" (UID: \"edd93c23-9f22-4d2c-9342-7f6c398fd6bb\") " pod="openshift-ovn-kubernetes/ovnkube-node-m794j" Jul 21 21:00:44.507079 ip-10-0-140-97 kubenswrapper[2575]: E0721 21:00:44.505602 2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 21 21:00:44.507079 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:44.505618 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/6a0eccd2-79bf-4f17-8217-6533fff9fbbb-sys-fs\") pod \"aws-ebs-csi-driver-node-x9lqv\" (UID: \"6a0eccd2-79bf-4f17-8217-6533fff9fbbb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x9lqv" Jul 21 21:00:44.507079 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:44.505642 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/aaa2e752-150d-4b89-852d-64473f98f593-agent-certs\") pod \"konnectivity-agent-cq6dd\" (UID: \"aaa2e752-150d-4b89-852d-64473f98f593\") " pod="kube-system/konnectivity-agent-cq6dd" Jul 21 21:00:44.507079 ip-10-0-140-97 kubenswrapper[2575]: E0721 21:00:44.505659 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/67b28cbf-d3f3-4d32-b87f-f1e586e2faca-metrics-certs podName:67b28cbf-d3f3-4d32-b87f-f1e586e2faca nodeName:}" failed. No retries permitted until 2026-07-21 21:00:45.005643364 +0000 UTC m=+2.940393007 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/67b28cbf-d3f3-4d32-b87f-f1e586e2faca-metrics-certs") pod "network-metrics-daemon-6fx5v" (UID: "67b28cbf-d3f3-4d32-b87f-f1e586e2faca") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 21 21:00:44.507079 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:44.505697 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/edd93c23-9f22-4d2c-9342-7f6c398fd6bb-host-cni-bin\") pod \"ovnkube-node-m794j\" (UID: \"edd93c23-9f22-4d2c-9342-7f6c398fd6bb\") " pod="openshift-ovn-kubernetes/ovnkube-node-m794j" Jul 21 21:00:44.507079 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:44.505727 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b69ef6b6-fca5-47e8-abc5-566b34e77058-tmp\") pod \"tuned-5pbk4\" (UID: \"b69ef6b6-fca5-47e8-abc5-566b34e77058\") " pod="openshift-cluster-node-tuning-operator/tuned-5pbk4" Jul 21 21:00:44.507079 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:44.505753 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/edd93c23-9f22-4d2c-9342-7f6c398fd6bb-run-openvswitch\") pod \"ovnkube-node-m794j\" (UID: \"edd93c23-9f22-4d2c-9342-7f6c398fd6bb\") " pod="openshift-ovn-kubernetes/ovnkube-node-m794j" Jul 21 21:00:44.507079 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:44.505779 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/edd93c23-9f22-4d2c-9342-7f6c398fd6bb-ovnkube-config\") pod \"ovnkube-node-m794j\" (UID: \"edd93c23-9f22-4d2c-9342-7f6c398fd6bb\") " pod="openshift-ovn-kubernetes/ovnkube-node-m794j" Jul 21 21:00:44.507079 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:44.505803 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/e3dcf4f7-a30b-46f5-9c5d-a77f61d88c51-kubelet-config\") pod \"global-pull-secret-syncer-5694f\" (UID: \"e3dcf4f7-a30b-46f5-9c5d-a77f61d88c51\") " pod="kube-system/global-pull-secret-syncer-5694f" Jul 21 21:00:44.507079 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:44.505828 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/6a0eccd2-79bf-4f17-8217-6533fff9fbbb-socket-dir\") pod \"aws-ebs-csi-driver-node-x9lqv\" (UID: \"6a0eccd2-79bf-4f17-8217-6533fff9fbbb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x9lqv" Jul 21 21:00:44.507469 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:44.505858 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/6a0eccd2-79bf-4f17-8217-6533fff9fbbb-registration-dir\") pod \"aws-ebs-csi-driver-node-x9lqv\" (UID: \"6a0eccd2-79bf-4f17-8217-6533fff9fbbb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x9lqv" Jul 21 21:00:44.507469 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:44.505867 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3877fdb0-b49b-444a-8f8c-597d3f935383-tmp-dir\") pod \"node-resolver-vdkfn\" (UID: \"3877fdb0-b49b-444a-8f8c-597d3f935383\") " pod="openshift-dns/node-resolver-vdkfn" Jul 21 21:00:44.507469 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:44.505884 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/d648b77f-43a8-4540-b59c-63768e78c245-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-952f6\" (UID: \"d648b77f-43a8-4540-b59c-63768e78c245\") " pod="openshift-multus/multus-additional-cni-plugins-952f6" Jul 21 21:00:44.507469 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:44.505913 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/b69ef6b6-fca5-47e8-abc5-566b34e77058-etc-systemd\") pod \"tuned-5pbk4\" (UID: \"b69ef6b6-fca5-47e8-abc5-566b34e77058\") " pod="openshift-cluster-node-tuning-operator/tuned-5pbk4" Jul 21 21:00:44.507469 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:44.505953 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/b69ef6b6-fca5-47e8-abc5-566b34e77058-lib-modules\") pod \"tuned-5pbk4\" (UID: \"b69ef6b6-fca5-47e8-abc5-566b34e77058\") " pod="openshift-cluster-node-tuning-operator/tuned-5pbk4" Jul 21 21:00:44.507469 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:44.506015 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/6bb08697-52d4-4ebf-b058-5771faf60563-host-run-k8s-cni-cncf-io\") pod \"multus-lphp9\" (UID: \"6bb08697-52d4-4ebf-b058-5771faf60563\") " pod="openshift-multus/multus-lphp9" Jul 21 21:00:44.507469 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:44.506035 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vkwg6\" (UniqueName: \"kubernetes.io/projected/3877fdb0-b49b-444a-8f8c-597d3f935383-kube-api-access-vkwg6\") pod \"node-resolver-vdkfn\" (UID: \"3877fdb0-b49b-444a-8f8c-597d3f935383\") " pod="openshift-dns/node-resolver-vdkfn" Jul 21 21:00:44.507469 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:44.506044 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/b69ef6b6-fca5-47e8-abc5-566b34e77058-etc-systemd\") pod \"tuned-5pbk4\" (UID: \"b69ef6b6-fca5-47e8-abc5-566b34e77058\") " pod="openshift-cluster-node-tuning-operator/tuned-5pbk4" Jul 21 21:00:44.507469 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:44.506054 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/edd93c23-9f22-4d2c-9342-7f6c398fd6bb-systemd-units\") pod \"ovnkube-node-m794j\" (UID: \"edd93c23-9f22-4d2c-9342-7f6c398fd6bb\") " pod="openshift-ovn-kubernetes/ovnkube-node-m794j" Jul 21 21:00:44.507469 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:44.506089 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/b69ef6b6-fca5-47e8-abc5-566b34e77058-lib-modules\") pod \"tuned-5pbk4\" (UID: \"b69ef6b6-fca5-47e8-abc5-566b34e77058\") " pod="openshift-cluster-node-tuning-operator/tuned-5pbk4" Jul 21 21:00:44.507469 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:44.506107 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/edd93c23-9f22-4d2c-9342-7f6c398fd6bb-run-systemd\") pod \"ovnkube-node-m794j\" (UID: \"edd93c23-9f22-4d2c-9342-7f6c398fd6bb\") " pod="openshift-ovn-kubernetes/ovnkube-node-m794j" Jul 21 21:00:44.507469 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:44.506127 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/1fd6b467-f1bf-45b5-8d30-96201ffccd36-iptables-alerter-script\") pod \"iptables-alerter-2czn5\" (UID: \"1fd6b467-f1bf-45b5-8d30-96201ffccd36\") " pod="openshift-network-operator/iptables-alerter-2czn5" Jul 21 21:00:44.507469 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:44.506160 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/6bb08697-52d4-4ebf-b058-5771faf60563-host-run-k8s-cni-cncf-io\") pod \"multus-lphp9\" (UID: \"6bb08697-52d4-4ebf-b058-5771faf60563\") " pod="openshift-multus/multus-lphp9" Jul 21 21:00:44.507469 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:44.506186 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s6lwj\" (UniqueName: \"kubernetes.io/projected/edd93c23-9f22-4d2c-9342-7f6c398fd6bb-kube-api-access-s6lwj\") pod \"ovnkube-node-m794j\" (UID: \"edd93c23-9f22-4d2c-9342-7f6c398fd6bb\") " pod="openshift-ovn-kubernetes/ovnkube-node-m794j" Jul 21 21:00:44.507874 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:44.506214 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e3dcf4f7-a30b-46f5-9c5d-a77f61d88c51-original-pull-secret\") pod \"global-pull-secret-syncer-5694f\" (UID: \"e3dcf4f7-a30b-46f5-9c5d-a77f61d88c51\") " pod="kube-system/global-pull-secret-syncer-5694f" Jul 21 21:00:44.507874 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:44.506255 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/6bb08697-52d4-4ebf-b058-5771faf60563-cni-binary-copy\") pod \"multus-lphp9\" (UID: \"6bb08697-52d4-4ebf-b058-5771faf60563\") " pod="openshift-multus/multus-lphp9" Jul 21 21:00:44.507874 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:44.506279 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/6bb08697-52d4-4ebf-b058-5771faf60563-multus-daemon-config\") pod \"multus-lphp9\" (UID: \"6bb08697-52d4-4ebf-b058-5771faf60563\") " pod="openshift-multus/multus-lphp9" Jul 21 21:00:44.507874 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:44.506305 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/6a0eccd2-79bf-4f17-8217-6533fff9fbbb-device-dir\") pod \"aws-ebs-csi-driver-node-x9lqv\" (UID: \"6a0eccd2-79bf-4f17-8217-6533fff9fbbb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x9lqv" Jul 21 21:00:44.507874 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:44.506365 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d648b77f-43a8-4540-b59c-63768e78c245-system-cni-dir\") pod \"multus-additional-cni-plugins-952f6\" (UID: \"d648b77f-43a8-4540-b59c-63768e78c245\") " pod="openshift-multus/multus-additional-cni-plugins-952f6" Jul 21 21:00:44.507874 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:44.506391 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/d648b77f-43a8-4540-b59c-63768e78c245-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-952f6\" (UID: \"d648b77f-43a8-4540-b59c-63768e78c245\") " pod="openshift-multus/multus-additional-cni-plugins-952f6" Jul 21 21:00:44.507874 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:44.506422 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9hd59\" (UniqueName: \"kubernetes.io/projected/1fd6b467-f1bf-45b5-8d30-96201ffccd36-kube-api-access-9hd59\") pod \"iptables-alerter-2czn5\" (UID: \"1fd6b467-f1bf-45b5-8d30-96201ffccd36\") " pod="openshift-network-operator/iptables-alerter-2czn5" Jul 21 21:00:44.507874 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:44.506447 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/edd93c23-9f22-4d2c-9342-7f6c398fd6bb-env-overrides\") pod \"ovnkube-node-m794j\" (UID: \"edd93c23-9f22-4d2c-9342-7f6c398fd6bb\") " pod="openshift-ovn-kubernetes/ovnkube-node-m794j" Jul 21 21:00:44.507874 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:44.506484 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/6a0eccd2-79bf-4f17-8217-6533fff9fbbb-kubelet-dir\") pod \"aws-ebs-csi-driver-node-x9lqv\" (UID: \"6a0eccd2-79bf-4f17-8217-6533fff9fbbb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x9lqv" Jul 21 21:00:44.507874 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:44.506516 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/b69ef6b6-fca5-47e8-abc5-566b34e77058-etc-modprobe-d\") pod \"tuned-5pbk4\" (UID: \"b69ef6b6-fca5-47e8-abc5-566b34e77058\") " pod="openshift-cluster-node-tuning-operator/tuned-5pbk4" Jul 21 21:00:44.507874 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:44.506556 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/b69ef6b6-fca5-47e8-abc5-566b34e77058-run\") pod \"tuned-5pbk4\" (UID: \"b69ef6b6-fca5-47e8-abc5-566b34e77058\") " pod="openshift-cluster-node-tuning-operator/tuned-5pbk4" Jul 21 21:00:44.507874 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:44.506585 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/6bb08697-52d4-4ebf-b058-5771faf60563-cnibin\") pod \"multus-lphp9\" (UID: \"6bb08697-52d4-4ebf-b058-5771faf60563\") " pod="openshift-multus/multus-lphp9" Jul 21 21:00:44.507874 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:44.506628 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/aaa2e752-150d-4b89-852d-64473f98f593-konnectivity-ca\") pod \"konnectivity-agent-cq6dd\" (UID: \"aaa2e752-150d-4b89-852d-64473f98f593\") " pod="kube-system/konnectivity-agent-cq6dd" Jul 21 21:00:44.507874 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:44.506655 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wvsds\" (UniqueName: \"kubernetes.io/projected/67b28cbf-d3f3-4d32-b87f-f1e586e2faca-kube-api-access-wvsds\") pod \"network-metrics-daemon-6fx5v\" (UID: \"67b28cbf-d3f3-4d32-b87f-f1e586e2faca\") " pod="openshift-multus/network-metrics-daemon-6fx5v" Jul 21 21:00:44.508247 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:44.506669 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/b69ef6b6-fca5-47e8-abc5-566b34e77058-etc-modprobe-d\") pod \"tuned-5pbk4\" (UID: \"b69ef6b6-fca5-47e8-abc5-566b34e77058\") " pod="openshift-cluster-node-tuning-operator/tuned-5pbk4" Jul 21 21:00:44.508247 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:44.506679 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b69ef6b6-fca5-47e8-abc5-566b34e77058-var-lib-kubelet\") pod \"tuned-5pbk4\" (UID: \"b69ef6b6-fca5-47e8-abc5-566b34e77058\") " pod="openshift-cluster-node-tuning-operator/tuned-5pbk4" Jul 21 21:00:44.508247 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:44.506704 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/edd93c23-9f22-4d2c-9342-7f6c398fd6bb-host-kubelet\") pod \"ovnkube-node-m794j\" (UID: \"edd93c23-9f22-4d2c-9342-7f6c398fd6bb\") " pod="openshift-ovn-kubernetes/ovnkube-node-m794j" Jul 21 21:00:44.508247 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:44.506685 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/b69ef6b6-fca5-47e8-abc5-566b34e77058-run\") pod \"tuned-5pbk4\" (UID: \"b69ef6b6-fca5-47e8-abc5-566b34e77058\") " pod="openshift-cluster-node-tuning-operator/tuned-5pbk4" Jul 21 21:00:44.508247 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:44.506727 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/ce7f9a12-9e5b-4634-83d3-babc1e854a66-serviceca\") pod \"node-ca-jhz69\" (UID: \"ce7f9a12-9e5b-4634-83d3-babc1e854a66\") " pod="openshift-image-registry/node-ca-jhz69" Jul 21 21:00:44.508247 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:44.506757 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/6bb08697-52d4-4ebf-b058-5771faf60563-cnibin\") pod \"multus-lphp9\" (UID: \"6bb08697-52d4-4ebf-b058-5771faf60563\") " pod="openshift-multus/multus-lphp9" Jul 21 21:00:44.508247 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:44.506761 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/6bb08697-52d4-4ebf-b058-5771faf60563-cni-binary-copy\") pod \"multus-lphp9\" (UID: \"6bb08697-52d4-4ebf-b058-5771faf60563\") " pod="openshift-multus/multus-lphp9" Jul 21 21:00:44.508247 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:44.506768 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b69ef6b6-fca5-47e8-abc5-566b34e77058-var-lib-kubelet\") pod \"tuned-5pbk4\" (UID: \"b69ef6b6-fca5-47e8-abc5-566b34e77058\") " pod="openshift-cluster-node-tuning-operator/tuned-5pbk4" Jul 21 21:00:44.508247 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:44.506853 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/6bb08697-52d4-4ebf-b058-5771faf60563-multus-daemon-config\") pod \"multus-lphp9\" (UID: \"6bb08697-52d4-4ebf-b058-5771faf60563\") " pod="openshift-multus/multus-lphp9" Jul 21 21:00:44.508247 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:44.508068 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b69ef6b6-fca5-47e8-abc5-566b34e77058-tmp\") pod \"tuned-5pbk4\" (UID: \"b69ef6b6-fca5-47e8-abc5-566b34e77058\") " pod="openshift-cluster-node-tuning-operator/tuned-5pbk4" Jul 21 21:00:44.508247 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:44.508099 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/b69ef6b6-fca5-47e8-abc5-566b34e77058-etc-tuned\") pod \"tuned-5pbk4\" (UID: \"b69ef6b6-fca5-47e8-abc5-566b34e77058\") " pod="openshift-cluster-node-tuning-operator/tuned-5pbk4" Jul 21 21:00:44.514719 ip-10-0-140-97 kubenswrapper[2575]: E0721 21:00:44.514702 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 21 21:00:44.514760 ip-10-0-140-97 kubenswrapper[2575]: E0721 21:00:44.514725 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 21 21:00:44.514760 ip-10-0-140-97 kubenswrapper[2575]: E0721 21:00:44.514748 2575 projected.go:196] Error preparing data for projected volume kube-api-access-djhxf for pod openshift-network-diagnostics/network-check-target-4nvs6: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 21 21:00:44.514818 ip-10-0-140-97 kubenswrapper[2575]: E0721 21:00:44.514801 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7a92ce6a-922d-4196-819d-c03a22e230d1-kube-api-access-djhxf podName:7a92ce6a-922d-4196-819d-c03a22e230d1 nodeName:}" failed. No retries permitted until 2026-07-21 21:00:45.01478577 +0000 UTC m=+2.949535428 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-djhxf" (UniqueName: "kubernetes.io/projected/7a92ce6a-922d-4196-819d-c03a22e230d1-kube-api-access-djhxf") pod "network-check-target-4nvs6" (UID: "7a92ce6a-922d-4196-819d-c03a22e230d1") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 21 21:00:44.516913 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:44.516890 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sgtlj\" (UniqueName: \"kubernetes.io/projected/6bb08697-52d4-4ebf-b058-5771faf60563-kube-api-access-sgtlj\") pod \"multus-lphp9\" (UID: \"6bb08697-52d4-4ebf-b058-5771faf60563\") " pod="openshift-multus/multus-lphp9" Jul 21 21:00:44.517110 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:44.517095 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f45zk\" (UniqueName: \"kubernetes.io/projected/b69ef6b6-fca5-47e8-abc5-566b34e77058-kube-api-access-f45zk\") pod \"tuned-5pbk4\" (UID: \"b69ef6b6-fca5-47e8-abc5-566b34e77058\") " pod="openshift-cluster-node-tuning-operator/tuned-5pbk4" Jul 21 21:00:44.519291 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:44.519267 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vkwg6\" (UniqueName: \"kubernetes.io/projected/3877fdb0-b49b-444a-8f8c-597d3f935383-kube-api-access-vkwg6\") pod \"node-resolver-vdkfn\" (UID: \"3877fdb0-b49b-444a-8f8c-597d3f935383\") " pod="openshift-dns/node-resolver-vdkfn" Jul 21 21:00:44.519730 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:44.519710 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9hd59\" (UniqueName: \"kubernetes.io/projected/1fd6b467-f1bf-45b5-8d30-96201ffccd36-kube-api-access-9hd59\") pod \"iptables-alerter-2czn5\" (UID: \"1fd6b467-f1bf-45b5-8d30-96201ffccd36\") " pod="openshift-network-operator/iptables-alerter-2czn5" Jul 21 21:00:44.520723 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:44.520708 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wvsds\" (UniqueName: \"kubernetes.io/projected/67b28cbf-d3f3-4d32-b87f-f1e586e2faca-kube-api-access-wvsds\") pod \"network-metrics-daemon-6fx5v\" (UID: \"67b28cbf-d3f3-4d32-b87f-f1e586e2faca\") " pod="openshift-multus/network-metrics-daemon-6fx5v" Jul 21 21:00:44.558766 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:44.558744 2575 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-pdfff" Jul 21 21:00:44.565741 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:44.565723 2575 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-pdfff" Jul 21 21:00:44.607969 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:44.607942 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/edd93c23-9f22-4d2c-9342-7f6c398fd6bb-run-ovn\") pod \"ovnkube-node-m794j\" (UID: \"edd93c23-9f22-4d2c-9342-7f6c398fd6bb\") " pod="openshift-ovn-kubernetes/ovnkube-node-m794j" Jul 21 21:00:44.607969 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:44.607969 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/edd93c23-9f22-4d2c-9342-7f6c398fd6bb-host-run-ovn-kubernetes\") pod \"ovnkube-node-m794j\" (UID: \"edd93c23-9f22-4d2c-9342-7f6c398fd6bb\") " pod="openshift-ovn-kubernetes/ovnkube-node-m794j" Jul 21 21:00:44.608133 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:44.607984 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/6a0eccd2-79bf-4f17-8217-6533fff9fbbb-etc-selinux\") pod \"aws-ebs-csi-driver-node-x9lqv\" (UID: \"6a0eccd2-79bf-4f17-8217-6533fff9fbbb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x9lqv" Jul 21 21:00:44.608133 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:44.608020 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7q2jm\" (UniqueName: \"kubernetes.io/projected/ce7f9a12-9e5b-4634-83d3-babc1e854a66-kube-api-access-7q2jm\") pod \"node-ca-jhz69\" (UID: \"ce7f9a12-9e5b-4634-83d3-babc1e854a66\") " pod="openshift-image-registry/node-ca-jhz69" Jul 21 21:00:44.608133 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:44.608042 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d648b77f-43a8-4540-b59c-63768e78c245-cnibin\") pod \"multus-additional-cni-plugins-952f6\" (UID: \"d648b77f-43a8-4540-b59c-63768e78c245\") " pod="openshift-multus/multus-additional-cni-plugins-952f6" Jul 21 21:00:44.608133 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:44.608050 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/edd93c23-9f22-4d2c-9342-7f6c398fd6bb-run-ovn\") pod \"ovnkube-node-m794j\" (UID: \"edd93c23-9f22-4d2c-9342-7f6c398fd6bb\") " pod="openshift-ovn-kubernetes/ovnkube-node-m794j" Jul 21 21:00:44.608133 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:44.608058 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/edd93c23-9f22-4d2c-9342-7f6c398fd6bb-host-run-ovn-kubernetes\") pod \"ovnkube-node-m794j\" (UID: \"edd93c23-9f22-4d2c-9342-7f6c398fd6bb\") " pod="openshift-ovn-kubernetes/ovnkube-node-m794j" Jul 21 21:00:44.608133 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:44.608083 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d648b77f-43a8-4540-b59c-63768e78c245-cnibin\") pod \"multus-additional-cni-plugins-952f6\" (UID: \"d648b77f-43a8-4540-b59c-63768e78c245\") " pod="openshift-multus/multus-additional-cni-plugins-952f6" Jul 21 21:00:44.608133 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:44.608114 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d648b77f-43a8-4540-b59c-63768e78c245-tuning-conf-dir\") pod \"multus-additional-cni-plugins-952f6\" (UID: \"d648b77f-43a8-4540-b59c-63768e78c245\") " pod="openshift-multus/multus-additional-cni-plugins-952f6" Jul 21 21:00:44.608133 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:44.608120 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/6a0eccd2-79bf-4f17-8217-6533fff9fbbb-etc-selinux\") pod \"aws-ebs-csi-driver-node-x9lqv\" (UID: \"6a0eccd2-79bf-4f17-8217-6533fff9fbbb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x9lqv" Jul 21 21:00:44.608467 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:44.608145 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d648b77f-43a8-4540-b59c-63768e78c245-cni-binary-copy\") pod \"multus-additional-cni-plugins-952f6\" (UID: \"d648b77f-43a8-4540-b59c-63768e78c245\") " pod="openshift-multus/multus-additional-cni-plugins-952f6" Jul 21 21:00:44.608467 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:44.608167 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/edd93c23-9f22-4d2c-9342-7f6c398fd6bb-ovnkube-script-lib\") pod \"ovnkube-node-m794j\" (UID: \"edd93c23-9f22-4d2c-9342-7f6c398fd6bb\") " pod="openshift-ovn-kubernetes/ovnkube-node-m794j" Jul 21 21:00:44.608467 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:44.608182 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g69kf\" (UniqueName: \"kubernetes.io/projected/d648b77f-43a8-4540-b59c-63768e78c245-kube-api-access-g69kf\") pod \"multus-additional-cni-plugins-952f6\" (UID: \"d648b77f-43a8-4540-b59c-63768e78c245\") " pod="openshift-multus/multus-additional-cni-plugins-952f6" Jul 21 21:00:44.608467 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:44.608225 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/edd93c23-9f22-4d2c-9342-7f6c398fd6bb-host-slash\") pod \"ovnkube-node-m794j\" (UID: \"edd93c23-9f22-4d2c-9342-7f6c398fd6bb\") " pod="openshift-ovn-kubernetes/ovnkube-node-m794j" Jul 21 21:00:44.608467 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:44.608251 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d648b77f-43a8-4540-b59c-63768e78c245-tuning-conf-dir\") pod \"multus-additional-cni-plugins-952f6\" (UID: \"d648b77f-43a8-4540-b59c-63768e78c245\") " pod="openshift-multus/multus-additional-cni-plugins-952f6" Jul 21 21:00:44.608467 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:44.608254 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/edd93c23-9f22-4d2c-9342-7f6c398fd6bb-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-m794j\" (UID: \"edd93c23-9f22-4d2c-9342-7f6c398fd6bb\") " pod="openshift-ovn-kubernetes/ovnkube-node-m794j" Jul 21 21:00:44.608467 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:44.608288 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/edd93c23-9f22-4d2c-9342-7f6c398fd6bb-host-slash\") pod \"ovnkube-node-m794j\" (UID: \"edd93c23-9f22-4d2c-9342-7f6c398fd6bb\") " pod="openshift-ovn-kubernetes/ovnkube-node-m794j" Jul 21 21:00:44.608467 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:44.608298 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/edd93c23-9f22-4d2c-9342-7f6c398fd6bb-log-socket\") pod \"ovnkube-node-m794j\" (UID: \"edd93c23-9f22-4d2c-9342-7f6c398fd6bb\") " pod="openshift-ovn-kubernetes/ovnkube-node-m794j" Jul 21 21:00:44.608467 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:44.608301 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/edd93c23-9f22-4d2c-9342-7f6c398fd6bb-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-m794j\" (UID: \"edd93c23-9f22-4d2c-9342-7f6c398fd6bb\") " pod="openshift-ovn-kubernetes/ovnkube-node-m794j" Jul 21 21:00:44.608467 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:44.608369 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/edd93c23-9f22-4d2c-9342-7f6c398fd6bb-log-socket\") pod \"ovnkube-node-m794j\" (UID: \"edd93c23-9f22-4d2c-9342-7f6c398fd6bb\") " pod="openshift-ovn-kubernetes/ovnkube-node-m794j" Jul 21 21:00:44.608467 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:44.608381 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/edd93c23-9f22-4d2c-9342-7f6c398fd6bb-etc-openvswitch\") pod \"ovnkube-node-m794j\" (UID: \"edd93c23-9f22-4d2c-9342-7f6c398fd6bb\") " pod="openshift-ovn-kubernetes/ovnkube-node-m794j" Jul 21 21:00:44.608467 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:44.608406 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/6a0eccd2-79bf-4f17-8217-6533fff9fbbb-sys-fs\") pod \"aws-ebs-csi-driver-node-x9lqv\" (UID: \"6a0eccd2-79bf-4f17-8217-6533fff9fbbb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x9lqv" Jul 21 21:00:44.608467 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:44.608429 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/aaa2e752-150d-4b89-852d-64473f98f593-agent-certs\") pod \"konnectivity-agent-cq6dd\" (UID: \"aaa2e752-150d-4b89-852d-64473f98f593\") " pod="kube-system/konnectivity-agent-cq6dd" Jul 21 21:00:44.608467 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:44.608427 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/edd93c23-9f22-4d2c-9342-7f6c398fd6bb-etc-openvswitch\") pod \"ovnkube-node-m794j\" (UID: \"edd93c23-9f22-4d2c-9342-7f6c398fd6bb\") " pod="openshift-ovn-kubernetes/ovnkube-node-m794j" Jul 21 21:00:44.609125 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:44.608455 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/edd93c23-9f22-4d2c-9342-7f6c398fd6bb-host-cni-bin\") pod \"ovnkube-node-m794j\" (UID: \"edd93c23-9f22-4d2c-9342-7f6c398fd6bb\") " pod="openshift-ovn-kubernetes/ovnkube-node-m794j" Jul 21 21:00:44.609125 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:44.608487 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/edd93c23-9f22-4d2c-9342-7f6c398fd6bb-host-cni-bin\") pod \"ovnkube-node-m794j\" (UID: \"edd93c23-9f22-4d2c-9342-7f6c398fd6bb\") " pod="openshift-ovn-kubernetes/ovnkube-node-m794j" Jul 21 21:00:44.609125 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:44.608468 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/6a0eccd2-79bf-4f17-8217-6533fff9fbbb-sys-fs\") pod \"aws-ebs-csi-driver-node-x9lqv\" (UID: \"6a0eccd2-79bf-4f17-8217-6533fff9fbbb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x9lqv" Jul 21 21:00:44.609125 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:44.608501 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/edd93c23-9f22-4d2c-9342-7f6c398fd6bb-run-openvswitch\") pod \"ovnkube-node-m794j\" (UID: \"edd93c23-9f22-4d2c-9342-7f6c398fd6bb\") " pod="openshift-ovn-kubernetes/ovnkube-node-m794j" Jul 21 21:00:44.609125 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:44.608525 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/edd93c23-9f22-4d2c-9342-7f6c398fd6bb-run-openvswitch\") pod \"ovnkube-node-m794j\" (UID: \"edd93c23-9f22-4d2c-9342-7f6c398fd6bb\") " pod="openshift-ovn-kubernetes/ovnkube-node-m794j" Jul 21 21:00:44.609125 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:44.608545 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/edd93c23-9f22-4d2c-9342-7f6c398fd6bb-ovnkube-config\") pod \"ovnkube-node-m794j\" (UID: \"edd93c23-9f22-4d2c-9342-7f6c398fd6bb\") " pod="openshift-ovn-kubernetes/ovnkube-node-m794j" Jul 21 21:00:44.609125 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:44.608574 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/e3dcf4f7-a30b-46f5-9c5d-a77f61d88c51-kubelet-config\") pod \"global-pull-secret-syncer-5694f\" (UID: \"e3dcf4f7-a30b-46f5-9c5d-a77f61d88c51\") " pod="kube-system/global-pull-secret-syncer-5694f" Jul 21 21:00:44.609125 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:44.608594 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/6a0eccd2-79bf-4f17-8217-6533fff9fbbb-socket-dir\") pod \"aws-ebs-csi-driver-node-x9lqv\" (UID: \"6a0eccd2-79bf-4f17-8217-6533fff9fbbb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x9lqv" Jul 21 21:00:44.609125 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:44.608640 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/6a0eccd2-79bf-4f17-8217-6533fff9fbbb-registration-dir\") pod \"aws-ebs-csi-driver-node-x9lqv\" (UID: \"6a0eccd2-79bf-4f17-8217-6533fff9fbbb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x9lqv" Jul 21 21:00:44.609125 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:44.608660 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/e3dcf4f7-a30b-46f5-9c5d-a77f61d88c51-kubelet-config\") pod \"global-pull-secret-syncer-5694f\" (UID: \"e3dcf4f7-a30b-46f5-9c5d-a77f61d88c51\") " pod="kube-system/global-pull-secret-syncer-5694f" Jul 21 21:00:44.609125 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:44.608671 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/d648b77f-43a8-4540-b59c-63768e78c245-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-952f6\" (UID: \"d648b77f-43a8-4540-b59c-63768e78c245\") " pod="openshift-multus/multus-additional-cni-plugins-952f6" Jul 21 21:00:44.609125 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:44.608702 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/edd93c23-9f22-4d2c-9342-7f6c398fd6bb-systemd-units\") pod \"ovnkube-node-m794j\" (UID: \"edd93c23-9f22-4d2c-9342-7f6c398fd6bb\") " pod="openshift-ovn-kubernetes/ovnkube-node-m794j" Jul 21 21:00:44.609125 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:44.608719 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/6a0eccd2-79bf-4f17-8217-6533fff9fbbb-registration-dir\") pod \"aws-ebs-csi-driver-node-x9lqv\" (UID: \"6a0eccd2-79bf-4f17-8217-6533fff9fbbb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x9lqv" Jul 21 21:00:44.609125 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:44.609024 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/edd93c23-9f22-4d2c-9342-7f6c398fd6bb-run-systemd\") pod \"ovnkube-node-m794j\" (UID: \"edd93c23-9f22-4d2c-9342-7f6c398fd6bb\") " pod="openshift-ovn-kubernetes/ovnkube-node-m794j" Jul 21 21:00:44.609792 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:44.608758 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/6a0eccd2-79bf-4f17-8217-6533fff9fbbb-socket-dir\") pod \"aws-ebs-csi-driver-node-x9lqv\" (UID: \"6a0eccd2-79bf-4f17-8217-6533fff9fbbb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x9lqv" Jul 21 21:00:44.609792 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:44.609145 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s6lwj\" (UniqueName: \"kubernetes.io/projected/edd93c23-9f22-4d2c-9342-7f6c398fd6bb-kube-api-access-s6lwj\") pod \"ovnkube-node-m794j\" (UID: \"edd93c23-9f22-4d2c-9342-7f6c398fd6bb\") " pod="openshift-ovn-kubernetes/ovnkube-node-m794j" Jul 21 21:00:44.609792 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:44.609189 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e3dcf4f7-a30b-46f5-9c5d-a77f61d88c51-original-pull-secret\") pod \"global-pull-secret-syncer-5694f\" (UID: \"e3dcf4f7-a30b-46f5-9c5d-a77f61d88c51\") " pod="kube-system/global-pull-secret-syncer-5694f" Jul 21 21:00:44.609792 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:44.609199 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d648b77f-43a8-4540-b59c-63768e78c245-cni-binary-copy\") pod \"multus-additional-cni-plugins-952f6\" (UID: \"d648b77f-43a8-4540-b59c-63768e78c245\") " pod="openshift-multus/multus-additional-cni-plugins-952f6" Jul 21 21:00:44.609792 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:44.608842 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/edd93c23-9f22-4d2c-9342-7f6c398fd6bb-systemd-units\") pod \"ovnkube-node-m794j\" (UID: \"edd93c23-9f22-4d2c-9342-7f6c398fd6bb\") " pod="openshift-ovn-kubernetes/ovnkube-node-m794j" Jul 21 21:00:44.609792 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:44.609226 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/6a0eccd2-79bf-4f17-8217-6533fff9fbbb-device-dir\") pod \"aws-ebs-csi-driver-node-x9lqv\" (UID: \"6a0eccd2-79bf-4f17-8217-6533fff9fbbb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x9lqv" Jul 21 21:00:44.609792 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:44.609261 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d648b77f-43a8-4540-b59c-63768e78c245-system-cni-dir\") pod \"multus-additional-cni-plugins-952f6\" (UID: \"d648b77f-43a8-4540-b59c-63768e78c245\") " pod="openshift-multus/multus-additional-cni-plugins-952f6" Jul 21 21:00:44.609792 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:44.609313 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/d648b77f-43a8-4540-b59c-63768e78c245-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-952f6\" (UID: \"d648b77f-43a8-4540-b59c-63768e78c245\") " pod="openshift-multus/multus-additional-cni-plugins-952f6" Jul 21 21:00:44.609792 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:44.609341 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/edd93c23-9f22-4d2c-9342-7f6c398fd6bb-env-overrides\") pod \"ovnkube-node-m794j\" (UID: \"edd93c23-9f22-4d2c-9342-7f6c398fd6bb\") " pod="openshift-ovn-kubernetes/ovnkube-node-m794j" Jul 21 21:00:44.609792 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:44.609383 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/6a0eccd2-79bf-4f17-8217-6533fff9fbbb-kubelet-dir\") pod \"aws-ebs-csi-driver-node-x9lqv\" (UID: \"6a0eccd2-79bf-4f17-8217-6533fff9fbbb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x9lqv" Jul 21 21:00:44.609792 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:44.609418 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/aaa2e752-150d-4b89-852d-64473f98f593-konnectivity-ca\") pod \"konnectivity-agent-cq6dd\" (UID: \"aaa2e752-150d-4b89-852d-64473f98f593\") " pod="kube-system/konnectivity-agent-cq6dd" Jul 21 21:00:44.609792 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:44.609452 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/edd93c23-9f22-4d2c-9342-7f6c398fd6bb-host-kubelet\") pod \"ovnkube-node-m794j\" (UID: \"edd93c23-9f22-4d2c-9342-7f6c398fd6bb\") " pod="openshift-ovn-kubernetes/ovnkube-node-m794j" Jul 21 21:00:44.609792 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:44.609477 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/edd93c23-9f22-4d2c-9342-7f6c398fd6bb-ovnkube-config\") pod \"ovnkube-node-m794j\" (UID: \"edd93c23-9f22-4d2c-9342-7f6c398fd6bb\") " pod="openshift-ovn-kubernetes/ovnkube-node-m794j" Jul 21 21:00:44.609792 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:44.609312 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/edd93c23-9f22-4d2c-9342-7f6c398fd6bb-ovnkube-script-lib\") pod \"ovnkube-node-m794j\" (UID: \"edd93c23-9f22-4d2c-9342-7f6c398fd6bb\") " pod="openshift-ovn-kubernetes/ovnkube-node-m794j" Jul 21 21:00:44.610414 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:44.609588 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/6a0eccd2-79bf-4f17-8217-6533fff9fbbb-device-dir\") pod \"aws-ebs-csi-driver-node-x9lqv\" (UID: \"6a0eccd2-79bf-4f17-8217-6533fff9fbbb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x9lqv" Jul 21 21:00:44.610414 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:44.609621 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d648b77f-43a8-4540-b59c-63768e78c245-system-cni-dir\") pod \"multus-additional-cni-plugins-952f6\" (UID: \"d648b77f-43a8-4540-b59c-63768e78c245\") " pod="openshift-multus/multus-additional-cni-plugins-952f6" Jul 21 21:00:44.610414 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:44.609647 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/edd93c23-9f22-4d2c-9342-7f6c398fd6bb-host-kubelet\") pod \"ovnkube-node-m794j\" (UID: \"edd93c23-9f22-4d2c-9342-7f6c398fd6bb\") " pod="openshift-ovn-kubernetes/ovnkube-node-m794j" Jul 21 21:00:44.610414 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:44.609751 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/d648b77f-43a8-4540-b59c-63768e78c245-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-952f6\" (UID: \"d648b77f-43a8-4540-b59c-63768e78c245\") " pod="openshift-multus/multus-additional-cni-plugins-952f6" Jul 21 21:00:44.610414 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:44.609863 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/6a0eccd2-79bf-4f17-8217-6533fff9fbbb-kubelet-dir\") pod \"aws-ebs-csi-driver-node-x9lqv\" (UID: \"6a0eccd2-79bf-4f17-8217-6533fff9fbbb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x9lqv" Jul 21 21:00:44.610414 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:44.609984 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/edd93c23-9f22-4d2c-9342-7f6c398fd6bb-run-systemd\") pod \"ovnkube-node-m794j\" (UID: \"edd93c23-9f22-4d2c-9342-7f6c398fd6bb\") " pod="openshift-ovn-kubernetes/ovnkube-node-m794j" Jul 21 21:00:44.610414 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:44.610032 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/ce7f9a12-9e5b-4634-83d3-babc1e854a66-serviceca\") pod \"node-ca-jhz69\" (UID: \"ce7f9a12-9e5b-4634-83d3-babc1e854a66\") " pod="openshift-image-registry/node-ca-jhz69" Jul 21 21:00:44.610414 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:44.609483 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/ce7f9a12-9e5b-4634-83d3-babc1e854a66-serviceca\") pod \"node-ca-jhz69\" (UID: \"ce7f9a12-9e5b-4634-83d3-babc1e854a66\") " pod="openshift-image-registry/node-ca-jhz69" Jul 21 21:00:44.610414 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:44.610090 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/aaa2e752-150d-4b89-852d-64473f98f593-konnectivity-ca\") pod \"konnectivity-agent-cq6dd\" (UID: \"aaa2e752-150d-4b89-852d-64473f98f593\") " pod="kube-system/konnectivity-agent-cq6dd" Jul 21 21:00:44.610414 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:44.610113 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/e3dcf4f7-a30b-46f5-9c5d-a77f61d88c51-dbus\") pod \"global-pull-secret-syncer-5694f\" (UID: \"e3dcf4f7-a30b-46f5-9c5d-a77f61d88c51\") " pod="kube-system/global-pull-secret-syncer-5694f" Jul 21 21:00:44.610414 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:44.610188 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/e3dcf4f7-a30b-46f5-9c5d-a77f61d88c51-dbus\") pod \"global-pull-secret-syncer-5694f\" (UID: \"e3dcf4f7-a30b-46f5-9c5d-a77f61d88c51\") " pod="kube-system/global-pull-secret-syncer-5694f" Jul 21 21:00:44.610414 ip-10-0-140-97 kubenswrapper[2575]: E0721 21:00:44.610192 2575 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 21 21:00:44.610414 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:44.610212 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/edd93c23-9f22-4d2c-9342-7f6c398fd6bb-env-overrides\") pod \"ovnkube-node-m794j\" (UID: \"edd93c23-9f22-4d2c-9342-7f6c398fd6bb\") " pod="openshift-ovn-kubernetes/ovnkube-node-m794j" Jul 21 21:00:44.610414 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:44.610220 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/edd93c23-9f22-4d2c-9342-7f6c398fd6bb-var-lib-openvswitch\") pod \"ovnkube-node-m794j\" (UID: \"edd93c23-9f22-4d2c-9342-7f6c398fd6bb\") " pod="openshift-ovn-kubernetes/ovnkube-node-m794j" Jul 21 21:00:44.610414 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:44.610259 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-98tgq\" (UniqueName: \"kubernetes.io/projected/6a0eccd2-79bf-4f17-8217-6533fff9fbbb-kube-api-access-98tgq\") pod \"aws-ebs-csi-driver-node-x9lqv\" (UID: \"6a0eccd2-79bf-4f17-8217-6533fff9fbbb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x9lqv" Jul 21 21:00:44.611177 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:44.610291 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/d648b77f-43a8-4540-b59c-63768e78c245-os-release\") pod \"multus-additional-cni-plugins-952f6\" (UID: \"d648b77f-43a8-4540-b59c-63768e78c245\") " pod="openshift-multus/multus-additional-cni-plugins-952f6" Jul 21 21:00:44.611177 ip-10-0-140-97 kubenswrapper[2575]: E0721 21:00:44.610329 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e3dcf4f7-a30b-46f5-9c5d-a77f61d88c51-original-pull-secret podName:e3dcf4f7-a30b-46f5-9c5d-a77f61d88c51 nodeName:}" failed. No retries permitted until 2026-07-21 21:00:45.110307823 +0000 UTC m=+3.045057468 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e3dcf4f7-a30b-46f5-9c5d-a77f61d88c51-original-pull-secret") pod "global-pull-secret-syncer-5694f" (UID: "e3dcf4f7-a30b-46f5-9c5d-a77f61d88c51") : object "kube-system"/"original-pull-secret" not registered Jul 21 21:00:44.611177 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:44.610349 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/edd93c23-9f22-4d2c-9342-7f6c398fd6bb-var-lib-openvswitch\") pod \"ovnkube-node-m794j\" (UID: \"edd93c23-9f22-4d2c-9342-7f6c398fd6bb\") " pod="openshift-ovn-kubernetes/ovnkube-node-m794j" Jul 21 21:00:44.611177 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:44.610377 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/d648b77f-43a8-4540-b59c-63768e78c245-os-release\") pod \"multus-additional-cni-plugins-952f6\" (UID: \"d648b77f-43a8-4540-b59c-63768e78c245\") " pod="openshift-multus/multus-additional-cni-plugins-952f6" Jul 21 21:00:44.611177 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:44.610518 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/edd93c23-9f22-4d2c-9342-7f6c398fd6bb-host-run-netns\") pod \"ovnkube-node-m794j\" (UID: \"edd93c23-9f22-4d2c-9342-7f6c398fd6bb\") " pod="openshift-ovn-kubernetes/ovnkube-node-m794j" Jul 21 21:00:44.611177 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:44.610357 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/edd93c23-9f22-4d2c-9342-7f6c398fd6bb-host-run-netns\") pod \"ovnkube-node-m794j\" (UID: \"edd93c23-9f22-4d2c-9342-7f6c398fd6bb\") " pod="openshift-ovn-kubernetes/ovnkube-node-m794j" Jul 21 21:00:44.611177 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:44.610859 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/edd93c23-9f22-4d2c-9342-7f6c398fd6bb-node-log\") pod \"ovnkube-node-m794j\" (UID: \"edd93c23-9f22-4d2c-9342-7f6c398fd6bb\") " pod="openshift-ovn-kubernetes/ovnkube-node-m794j" Jul 21 21:00:44.611177 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:44.610886 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/edd93c23-9f22-4d2c-9342-7f6c398fd6bb-ovn-node-metrics-cert\") pod \"ovnkube-node-m794j\" (UID: \"edd93c23-9f22-4d2c-9342-7f6c398fd6bb\") " pod="openshift-ovn-kubernetes/ovnkube-node-m794j" Jul 21 21:00:44.611177 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:44.610915 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ce7f9a12-9e5b-4634-83d3-babc1e854a66-host\") pod \"node-ca-jhz69\" (UID: \"ce7f9a12-9e5b-4634-83d3-babc1e854a66\") " pod="openshift-image-registry/node-ca-jhz69" Jul 21 21:00:44.611177 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:44.610945 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/edd93c23-9f22-4d2c-9342-7f6c398fd6bb-host-cni-netd\") pod \"ovnkube-node-m794j\" (UID: \"edd93c23-9f22-4d2c-9342-7f6c398fd6bb\") " pod="openshift-ovn-kubernetes/ovnkube-node-m794j" Jul 21 21:00:44.611177 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:44.611047 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/edd93c23-9f22-4d2c-9342-7f6c398fd6bb-host-cni-netd\") pod \"ovnkube-node-m794j\" (UID: \"edd93c23-9f22-4d2c-9342-7f6c398fd6bb\") " pod="openshift-ovn-kubernetes/ovnkube-node-m794j" Jul 21 21:00:44.611177 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:44.611095 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ce7f9a12-9e5b-4634-83d3-babc1e854a66-host\") pod \"node-ca-jhz69\" (UID: \"ce7f9a12-9e5b-4634-83d3-babc1e854a66\") " pod="openshift-image-registry/node-ca-jhz69" Jul 21 21:00:44.611177 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:44.611139 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/edd93c23-9f22-4d2c-9342-7f6c398fd6bb-node-log\") pod \"ovnkube-node-m794j\" (UID: \"edd93c23-9f22-4d2c-9342-7f6c398fd6bb\") " pod="openshift-ovn-kubernetes/ovnkube-node-m794j" Jul 21 21:00:44.611756 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:44.611654 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/d648b77f-43a8-4540-b59c-63768e78c245-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-952f6\" (UID: \"d648b77f-43a8-4540-b59c-63768e78c245\") " pod="openshift-multus/multus-additional-cni-plugins-952f6" Jul 21 21:00:44.611756 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:44.611679 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/aaa2e752-150d-4b89-852d-64473f98f593-agent-certs\") pod \"konnectivity-agent-cq6dd\" (UID: \"aaa2e752-150d-4b89-852d-64473f98f593\") " pod="kube-system/konnectivity-agent-cq6dd" Jul 21 21:00:44.613767 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:44.613742 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/edd93c23-9f22-4d2c-9342-7f6c398fd6bb-ovn-node-metrics-cert\") pod \"ovnkube-node-m794j\" (UID: \"edd93c23-9f22-4d2c-9342-7f6c398fd6bb\") " pod="openshift-ovn-kubernetes/ovnkube-node-m794j" Jul 21 21:00:44.620749 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:44.620724 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7q2jm\" (UniqueName: \"kubernetes.io/projected/ce7f9a12-9e5b-4634-83d3-babc1e854a66-kube-api-access-7q2jm\") pod \"node-ca-jhz69\" (UID: \"ce7f9a12-9e5b-4634-83d3-babc1e854a66\") " pod="openshift-image-registry/node-ca-jhz69" Jul 21 21:00:44.621158 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:44.621140 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g69kf\" (UniqueName: \"kubernetes.io/projected/d648b77f-43a8-4540-b59c-63768e78c245-kube-api-access-g69kf\") pod \"multus-additional-cni-plugins-952f6\" (UID: \"d648b77f-43a8-4540-b59c-63768e78c245\") " pod="openshift-multus/multus-additional-cni-plugins-952f6" Jul 21 21:00:44.622395 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:44.622374 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s6lwj\" (UniqueName: \"kubernetes.io/projected/edd93c23-9f22-4d2c-9342-7f6c398fd6bb-kube-api-access-s6lwj\") pod \"ovnkube-node-m794j\" (UID: \"edd93c23-9f22-4d2c-9342-7f6c398fd6bb\") " pod="openshift-ovn-kubernetes/ovnkube-node-m794j" Jul 21 21:00:44.622504 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:44.622490 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-98tgq\" (UniqueName: \"kubernetes.io/projected/6a0eccd2-79bf-4f17-8217-6533fff9fbbb-kube-api-access-98tgq\") pod \"aws-ebs-csi-driver-node-x9lqv\" (UID: \"6a0eccd2-79bf-4f17-8217-6533fff9fbbb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x9lqv" Jul 21 21:00:44.686303 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:44.686244 2575 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 21 21:00:44.697738 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:44.697715 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-5pbk4" Jul 21 21:00:44.704899 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:44.704880 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-lphp9" Jul 21 21:00:44.706733 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:00:44.706671 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb69ef6b6_fca5_47e8_abc5_566b34e77058.slice/crio-b30c50e350d347d0911ecd66cb4d7a3b69e68384f24782cf900aead740cf8a36 WatchSource:0}: Error finding container b30c50e350d347d0911ecd66cb4d7a3b69e68384f24782cf900aead740cf8a36: Status 404 returned error can't find the container with id b30c50e350d347d0911ecd66cb4d7a3b69e68384f24782cf900aead740cf8a36 Jul 21 21:00:44.710926 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:44.710851 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-vdkfn" Jul 21 21:00:44.713836 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:00:44.713804 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6bb08697_52d4_4ebf_b058_5771faf60563.slice/crio-a2bb4bbe8906f62322a8998bc4b32383d0974289e645c55b06fbabc7c2e15841 WatchSource:0}: Error finding container a2bb4bbe8906f62322a8998bc4b32383d0974289e645c55b06fbabc7c2e15841: Status 404 returned error can't find the container with id a2bb4bbe8906f62322a8998bc4b32383d0974289e645c55b06fbabc7c2e15841 Jul 21 21:00:44.715220 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:44.715201 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-2czn5" Jul 21 21:00:44.720623 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:00:44.720602 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3877fdb0_b49b_444a_8f8c_597d3f935383.slice/crio-960bcdca44ed8aba69af6209ca483921110c3cf69bd35658d376e795d3122303 WatchSource:0}: Error finding container 960bcdca44ed8aba69af6209ca483921110c3cf69bd35658d376e795d3122303: Status 404 returned error can't find the container with id 960bcdca44ed8aba69af6209ca483921110c3cf69bd35658d376e795d3122303 Jul 21 21:00:44.722175 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:44.722144 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-m794j" Jul 21 21:00:44.725655 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:00:44.725633 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1fd6b467_f1bf_45b5_8d30_96201ffccd36.slice/crio-42164588963fa1352b857d9b26fc84bd2c39ae9c917d696e35947efc54e1f338 WatchSource:0}: Error finding container 42164588963fa1352b857d9b26fc84bd2c39ae9c917d696e35947efc54e1f338: Status 404 returned error can't find the container with id 42164588963fa1352b857d9b26fc84bd2c39ae9c917d696e35947efc54e1f338 Jul 21 21:00:44.727563 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:44.727543 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x9lqv" Jul 21 21:00:44.731174 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:00:44.731147 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podedd93c23_9f22_4d2c_9342_7f6c398fd6bb.slice/crio-b14419b29481fd5fb7703147bfbe1a5ff58bd38efa7dd4e2922e0f233368fd66 WatchSource:0}: Error finding container b14419b29481fd5fb7703147bfbe1a5ff58bd38efa7dd4e2922e0f233368fd66: Status 404 returned error can't find the container with id b14419b29481fd5fb7703147bfbe1a5ff58bd38efa7dd4e2922e0f233368fd66 Jul 21 21:00:44.732908 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:44.732886 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-cq6dd" Jul 21 21:00:44.737114 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:00:44.737088 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6a0eccd2_79bf_4f17_8217_6533fff9fbbb.slice/crio-102e7a1caca8eb4b3918cd28223a4709442a9250a1c5deb2131393e3540a60f9 WatchSource:0}: Error finding container 102e7a1caca8eb4b3918cd28223a4709442a9250a1c5deb2131393e3540a60f9: Status 404 returned error can't find the container with id 102e7a1caca8eb4b3918cd28223a4709442a9250a1c5deb2131393e3540a60f9 Jul 21 21:00:44.738607 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:44.738588 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-jhz69" Jul 21 21:00:44.739927 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:44.739879 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-97.ec2.internal" event={"ID":"df77ae6a34653e84fea42c7c1fb02209","Type":"ContainerStarted","Data":"6c615e8d1f562a4cedeca4bf8630c76392a82b916a2c35e0154a38e909bd486c"} Jul 21 21:00:44.741123 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:44.741089 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x9lqv" event={"ID":"6a0eccd2-79bf-4f17-8217-6533fff9fbbb","Type":"ContainerStarted","Data":"102e7a1caca8eb4b3918cd28223a4709442a9250a1c5deb2131393e3540a60f9"} Jul 21 21:00:44.741666 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:00:44.741643 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaaa2e752_150d_4b89_852d_64473f98f593.slice/crio-e6a1efa045b2e8d8a6596ce0d86d0b16594567e344fd798a8ff96bce74363914 WatchSource:0}: Error finding container e6a1efa045b2e8d8a6596ce0d86d0b16594567e344fd798a8ff96bce74363914: Status 404 returned error can't find the container with id e6a1efa045b2e8d8a6596ce0d86d0b16594567e344fd798a8ff96bce74363914 Jul 21 21:00:44.742794 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:44.742676 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-m794j" event={"ID":"edd93c23-9f22-4d2c-9342-7f6c398fd6bb","Type":"ContainerStarted","Data":"b14419b29481fd5fb7703147bfbe1a5ff58bd38efa7dd4e2922e0f233368fd66"} Jul 21 21:00:44.742794 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:44.742787 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-952f6" Jul 21 21:00:44.744084 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:44.744065 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-2czn5" event={"ID":"1fd6b467-f1bf-45b5-8d30-96201ffccd36","Type":"ContainerStarted","Data":"42164588963fa1352b857d9b26fc84bd2c39ae9c917d696e35947efc54e1f338"} Jul 21 21:00:44.745229 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:44.745211 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-vdkfn" event={"ID":"3877fdb0-b49b-444a-8f8c-597d3f935383","Type":"ContainerStarted","Data":"960bcdca44ed8aba69af6209ca483921110c3cf69bd35658d376e795d3122303"} Jul 21 21:00:44.746522 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:44.746501 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-lphp9" event={"ID":"6bb08697-52d4-4ebf-b058-5771faf60563","Type":"ContainerStarted","Data":"a2bb4bbe8906f62322a8998bc4b32383d0974289e645c55b06fbabc7c2e15841"} Jul 21 21:00:44.747896 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:44.747872 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-5pbk4" event={"ID":"b69ef6b6-fca5-47e8-abc5-566b34e77058","Type":"ContainerStarted","Data":"b30c50e350d347d0911ecd66cb4d7a3b69e68384f24782cf900aead740cf8a36"} Jul 21 21:00:44.749268 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:00:44.749247 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podce7f9a12_9e5b_4634_83d3_babc1e854a66.slice/crio-8e4bd4f664be4c97fee06df5ca771747eee02f1c506d1fb37c375b96b1352127 WatchSource:0}: Error finding container 8e4bd4f664be4c97fee06df5ca771747eee02f1c506d1fb37c375b96b1352127: Status 404 returned error can't find the container with id 8e4bd4f664be4c97fee06df5ca771747eee02f1c506d1fb37c375b96b1352127 Jul 21 21:00:44.749372 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:44.749343 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-97.ec2.internal" event={"ID":"810679f65980473b59e3cca08ad4f0fc","Type":"ContainerStarted","Data":"bc5b3d9e4ebaf24ab318f972e45c39d7ddcd9f8538c7e8d7c1beeffc1d53a0b6"} Jul 21 21:00:44.752963 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:00:44.752928 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd648b77f_43a8_4540_b59c_63768e78c245.slice/crio-5dfa7013fb3408d42a6ca05bbdff947c2313ba18ee95238304276f1f2f00e429 WatchSource:0}: Error finding container 5dfa7013fb3408d42a6ca05bbdff947c2313ba18ee95238304276f1f2f00e429: Status 404 returned error can't find the container with id 5dfa7013fb3408d42a6ca05bbdff947c2313ba18ee95238304276f1f2f00e429 Jul 21 21:00:45.015048 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:45.014960 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/67b28cbf-d3f3-4d32-b87f-f1e586e2faca-metrics-certs\") pod \"network-metrics-daemon-6fx5v\" (UID: \"67b28cbf-d3f3-4d32-b87f-f1e586e2faca\") " pod="openshift-multus/network-metrics-daemon-6fx5v" Jul 21 21:00:45.015048 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:45.015034 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-djhxf\" (UniqueName: \"kubernetes.io/projected/7a92ce6a-922d-4196-819d-c03a22e230d1-kube-api-access-djhxf\") pod \"network-check-target-4nvs6\" (UID: \"7a92ce6a-922d-4196-819d-c03a22e230d1\") " pod="openshift-network-diagnostics/network-check-target-4nvs6" Jul 21 21:00:45.015261 ip-10-0-140-97 kubenswrapper[2575]: E0721 21:00:45.015130 2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 21 21:00:45.015261 ip-10-0-140-97 kubenswrapper[2575]: E0721 21:00:45.015166 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 21 21:00:45.015261 ip-10-0-140-97 kubenswrapper[2575]: E0721 21:00:45.015183 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 21 21:00:45.015261 ip-10-0-140-97 kubenswrapper[2575]: E0721 21:00:45.015195 2575 projected.go:196] Error preparing data for projected volume kube-api-access-djhxf for pod openshift-network-diagnostics/network-check-target-4nvs6: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 21 21:00:45.015261 ip-10-0-140-97 kubenswrapper[2575]: E0721 21:00:45.015195 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/67b28cbf-d3f3-4d32-b87f-f1e586e2faca-metrics-certs podName:67b28cbf-d3f3-4d32-b87f-f1e586e2faca nodeName:}" failed. No retries permitted until 2026-07-21 21:00:46.015176144 +0000 UTC m=+3.949925800 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/67b28cbf-d3f3-4d32-b87f-f1e586e2faca-metrics-certs") pod "network-metrics-daemon-6fx5v" (UID: "67b28cbf-d3f3-4d32-b87f-f1e586e2faca") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 21 21:00:45.015261 ip-10-0-140-97 kubenswrapper[2575]: E0721 21:00:45.015237 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7a92ce6a-922d-4196-819d-c03a22e230d1-kube-api-access-djhxf podName:7a92ce6a-922d-4196-819d-c03a22e230d1 nodeName:}" failed. No retries permitted until 2026-07-21 21:00:46.01522634 +0000 UTC m=+3.949975989 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-djhxf" (UniqueName: "kubernetes.io/projected/7a92ce6a-922d-4196-819d-c03a22e230d1-kube-api-access-djhxf") pod "network-check-target-4nvs6" (UID: "7a92ce6a-922d-4196-819d-c03a22e230d1") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 21 21:00:45.074650 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:45.074621 2575 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 21 21:00:45.116373 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:45.116338 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e3dcf4f7-a30b-46f5-9c5d-a77f61d88c51-original-pull-secret\") pod \"global-pull-secret-syncer-5694f\" (UID: \"e3dcf4f7-a30b-46f5-9c5d-a77f61d88c51\") " pod="kube-system/global-pull-secret-syncer-5694f" Jul 21 21:00:45.116557 ip-10-0-140-97 kubenswrapper[2575]: E0721 21:00:45.116467 2575 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 21 21:00:45.116557 ip-10-0-140-97 kubenswrapper[2575]: E0721 21:00:45.116555 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e3dcf4f7-a30b-46f5-9c5d-a77f61d88c51-original-pull-secret podName:e3dcf4f7-a30b-46f5-9c5d-a77f61d88c51 nodeName:}" failed. No retries permitted until 2026-07-21 21:00:46.116515599 +0000 UTC m=+4.051265246 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e3dcf4f7-a30b-46f5-9c5d-a77f61d88c51-original-pull-secret") pod "global-pull-secret-syncer-5694f" (UID: "e3dcf4f7-a30b-46f5-9c5d-a77f61d88c51") : object "kube-system"/"original-pull-secret" not registered Jul 21 21:00:45.569250 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:45.569203 2575 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-20 20:55:44 +0000 UTC" deadline="2028-01-26 09:59:07.294469565 +0000 UTC" Jul 21 21:00:45.569250 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:45.569246 2575 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13284h58m21.72522905s" Jul 21 21:00:45.734906 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:45.734882 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4nvs6" Jul 21 21:00:45.735044 ip-10-0-140-97 kubenswrapper[2575]: E0721 21:00:45.735006 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4nvs6" podUID="7a92ce6a-922d-4196-819d-c03a22e230d1" Jul 21 21:00:45.735347 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:45.735327 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5694f" Jul 21 21:00:45.735454 ip-10-0-140-97 kubenswrapper[2575]: E0721 21:00:45.735434 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5694f" podUID="e3dcf4f7-a30b-46f5-9c5d-a77f61d88c51" Jul 21 21:00:45.735510 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:45.735494 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6fx5v" Jul 21 21:00:45.735606 ip-10-0-140-97 kubenswrapper[2575]: E0721 21:00:45.735589 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6fx5v" podUID="67b28cbf-d3f3-4d32-b87f-f1e586e2faca" Jul 21 21:00:45.758143 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:45.758115 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-jhz69" event={"ID":"ce7f9a12-9e5b-4634-83d3-babc1e854a66","Type":"ContainerStarted","Data":"8e4bd4f664be4c97fee06df5ca771747eee02f1c506d1fb37c375b96b1352127"} Jul 21 21:00:45.763330 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:45.763233 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-cq6dd" event={"ID":"aaa2e752-150d-4b89-852d-64473f98f593","Type":"ContainerStarted","Data":"e6a1efa045b2e8d8a6596ce0d86d0b16594567e344fd798a8ff96bce74363914"} Jul 21 21:00:45.776010 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:45.775986 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-952f6" event={"ID":"d648b77f-43a8-4540-b59c-63768e78c245","Type":"ContainerStarted","Data":"5dfa7013fb3408d42a6ca05bbdff947c2313ba18ee95238304276f1f2f00e429"} Jul 21 21:00:46.028520 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:46.027686 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-djhxf\" (UniqueName: \"kubernetes.io/projected/7a92ce6a-922d-4196-819d-c03a22e230d1-kube-api-access-djhxf\") pod \"network-check-target-4nvs6\" (UID: \"7a92ce6a-922d-4196-819d-c03a22e230d1\") " pod="openshift-network-diagnostics/network-check-target-4nvs6" Jul 21 21:00:46.028520 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:46.027749 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/67b28cbf-d3f3-4d32-b87f-f1e586e2faca-metrics-certs\") pod \"network-metrics-daemon-6fx5v\" (UID: \"67b28cbf-d3f3-4d32-b87f-f1e586e2faca\") " pod="openshift-multus/network-metrics-daemon-6fx5v" Jul 21 21:00:46.028520 ip-10-0-140-97 kubenswrapper[2575]: E0721 21:00:46.027894 2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 21 21:00:46.028520 ip-10-0-140-97 kubenswrapper[2575]: E0721 21:00:46.027949 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/67b28cbf-d3f3-4d32-b87f-f1e586e2faca-metrics-certs podName:67b28cbf-d3f3-4d32-b87f-f1e586e2faca nodeName:}" failed. No retries permitted until 2026-07-21 21:00:48.027931589 +0000 UTC m=+5.962681235 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/67b28cbf-d3f3-4d32-b87f-f1e586e2faca-metrics-certs") pod "network-metrics-daemon-6fx5v" (UID: "67b28cbf-d3f3-4d32-b87f-f1e586e2faca") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 21 21:00:46.028520 ip-10-0-140-97 kubenswrapper[2575]: E0721 21:00:46.028341 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 21 21:00:46.028520 ip-10-0-140-97 kubenswrapper[2575]: E0721 21:00:46.028361 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 21 21:00:46.028520 ip-10-0-140-97 kubenswrapper[2575]: E0721 21:00:46.028374 2575 projected.go:196] Error preparing data for projected volume kube-api-access-djhxf for pod openshift-network-diagnostics/network-check-target-4nvs6: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 21 21:00:46.028520 ip-10-0-140-97 kubenswrapper[2575]: E0721 21:00:46.028418 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7a92ce6a-922d-4196-819d-c03a22e230d1-kube-api-access-djhxf podName:7a92ce6a-922d-4196-819d-c03a22e230d1 nodeName:}" failed. No retries permitted until 2026-07-21 21:00:48.028403908 +0000 UTC m=+5.963153555 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-djhxf" (UniqueName: "kubernetes.io/projected/7a92ce6a-922d-4196-819d-c03a22e230d1-kube-api-access-djhxf") pod "network-check-target-4nvs6" (UID: "7a92ce6a-922d-4196-819d-c03a22e230d1") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 21 21:00:46.128788 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:46.128263 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e3dcf4f7-a30b-46f5-9c5d-a77f61d88c51-original-pull-secret\") pod \"global-pull-secret-syncer-5694f\" (UID: \"e3dcf4f7-a30b-46f5-9c5d-a77f61d88c51\") " pod="kube-system/global-pull-secret-syncer-5694f" Jul 21 21:00:46.128788 ip-10-0-140-97 kubenswrapper[2575]: E0721 21:00:46.128411 2575 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 21 21:00:46.128788 ip-10-0-140-97 kubenswrapper[2575]: E0721 21:00:46.128468 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e3dcf4f7-a30b-46f5-9c5d-a77f61d88c51-original-pull-secret podName:e3dcf4f7-a30b-46f5-9c5d-a77f61d88c51 nodeName:}" failed. No retries permitted until 2026-07-21 21:00:48.128450125 +0000 UTC m=+6.063199773 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e3dcf4f7-a30b-46f5-9c5d-a77f61d88c51-original-pull-secret") pod "global-pull-secret-syncer-5694f" (UID: "e3dcf4f7-a30b-46f5-9c5d-a77f61d88c51") : object "kube-system"/"original-pull-secret" not registered Jul 21 21:00:46.184294 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:46.184105 2575 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Jul 21 21:00:46.569406 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:46.569333 2575 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-20 20:55:44 +0000 UTC" deadline="2028-03-22 19:14:46.198342143 +0000 UTC" Jul 21 21:00:46.569406 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:46.569404 2575 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14638h13m59.628942728s" Jul 21 21:00:46.795604 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:46.795231 2575 generic.go:356] "Generic (PLEG): container finished" podID="df77ae6a34653e84fea42c7c1fb02209" containerID="b7b9bbb2475b2dbd558a22913cc6a6482a488c405f548d5426849d9a30e5c48d" exitCode=0 Jul 21 21:00:46.795604 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:46.795277 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-97.ec2.internal" event={"ID":"df77ae6a34653e84fea42c7c1fb02209","Type":"ContainerDied","Data":"b7b9bbb2475b2dbd558a22913cc6a6482a488c405f548d5426849d9a30e5c48d"} Jul 21 21:00:47.734615 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:47.734575 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5694f" Jul 21 21:00:47.735067 ip-10-0-140-97 kubenswrapper[2575]: E0721 21:00:47.734711 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5694f" podUID="e3dcf4f7-a30b-46f5-9c5d-a77f61d88c51" Jul 21 21:00:47.735153 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:47.735120 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6fx5v" Jul 21 21:00:47.735256 ip-10-0-140-97 kubenswrapper[2575]: E0721 21:00:47.735234 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6fx5v" podUID="67b28cbf-d3f3-4d32-b87f-f1e586e2faca" Jul 21 21:00:47.735315 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:47.735299 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4nvs6" Jul 21 21:00:47.735391 ip-10-0-140-97 kubenswrapper[2575]: E0721 21:00:47.735368 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4nvs6" podUID="7a92ce6a-922d-4196-819d-c03a22e230d1" Jul 21 21:00:47.804751 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:47.804115 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-97.ec2.internal" event={"ID":"df77ae6a34653e84fea42c7c1fb02209","Type":"ContainerStarted","Data":"ff361bab499f6e673b6d32754d3a5ce4b226757c8c8e897a77cfc81d937bbc64"} Jul 21 21:00:48.045705 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:48.045588 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/67b28cbf-d3f3-4d32-b87f-f1e586e2faca-metrics-certs\") pod \"network-metrics-daemon-6fx5v\" (UID: \"67b28cbf-d3f3-4d32-b87f-f1e586e2faca\") " pod="openshift-multus/network-metrics-daemon-6fx5v" Jul 21 21:00:48.045884 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:48.045735 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-djhxf\" (UniqueName: \"kubernetes.io/projected/7a92ce6a-922d-4196-819d-c03a22e230d1-kube-api-access-djhxf\") pod \"network-check-target-4nvs6\" (UID: \"7a92ce6a-922d-4196-819d-c03a22e230d1\") " pod="openshift-network-diagnostics/network-check-target-4nvs6" Jul 21 21:00:48.045944 ip-10-0-140-97 kubenswrapper[2575]: E0721 21:00:48.045905 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 21 21:00:48.045944 ip-10-0-140-97 kubenswrapper[2575]: E0721 21:00:48.045927 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 21 21:00:48.045944 ip-10-0-140-97 kubenswrapper[2575]: E0721 21:00:48.045940 2575 projected.go:196] Error preparing data for projected volume kube-api-access-djhxf for pod openshift-network-diagnostics/network-check-target-4nvs6: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 21 21:00:48.046137 ip-10-0-140-97 kubenswrapper[2575]: E0721 21:00:48.045999 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7a92ce6a-922d-4196-819d-c03a22e230d1-kube-api-access-djhxf podName:7a92ce6a-922d-4196-819d-c03a22e230d1 nodeName:}" failed. No retries permitted until 2026-07-21 21:00:52.045980974 +0000 UTC m=+9.980730620 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-djhxf" (UniqueName: "kubernetes.io/projected/7a92ce6a-922d-4196-819d-c03a22e230d1-kube-api-access-djhxf") pod "network-check-target-4nvs6" (UID: "7a92ce6a-922d-4196-819d-c03a22e230d1") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 21 21:00:48.046439 ip-10-0-140-97 kubenswrapper[2575]: E0721 21:00:48.046419 2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 21 21:00:48.046522 ip-10-0-140-97 kubenswrapper[2575]: E0721 21:00:48.046476 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/67b28cbf-d3f3-4d32-b87f-f1e586e2faca-metrics-certs podName:67b28cbf-d3f3-4d32-b87f-f1e586e2faca nodeName:}" failed. No retries permitted until 2026-07-21 21:00:52.046461489 +0000 UTC m=+9.981211135 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/67b28cbf-d3f3-4d32-b87f-f1e586e2faca-metrics-certs") pod "network-metrics-daemon-6fx5v" (UID: "67b28cbf-d3f3-4d32-b87f-f1e586e2faca") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 21 21:00:48.146704 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:48.146641 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e3dcf4f7-a30b-46f5-9c5d-a77f61d88c51-original-pull-secret\") pod \"global-pull-secret-syncer-5694f\" (UID: \"e3dcf4f7-a30b-46f5-9c5d-a77f61d88c51\") " pod="kube-system/global-pull-secret-syncer-5694f" Jul 21 21:00:48.146880 ip-10-0-140-97 kubenswrapper[2575]: E0721 21:00:48.146799 2575 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 21 21:00:48.146880 ip-10-0-140-97 kubenswrapper[2575]: E0721 21:00:48.146860 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e3dcf4f7-a30b-46f5-9c5d-a77f61d88c51-original-pull-secret podName:e3dcf4f7-a30b-46f5-9c5d-a77f61d88c51 nodeName:}" failed. No retries permitted until 2026-07-21 21:00:52.146842641 +0000 UTC m=+10.081592291 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e3dcf4f7-a30b-46f5-9c5d-a77f61d88c51-original-pull-secret") pod "global-pull-secret-syncer-5694f" (UID: "e3dcf4f7-a30b-46f5-9c5d-a77f61d88c51") : object "kube-system"/"original-pull-secret" not registered Jul 21 21:00:49.734971 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:49.734935 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6fx5v" Jul 21 21:00:49.735372 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:49.734936 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4nvs6" Jul 21 21:00:49.735372 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:49.735071 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5694f" Jul 21 21:00:49.735372 ip-10-0-140-97 kubenswrapper[2575]: E0721 21:00:49.735079 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6fx5v" podUID="67b28cbf-d3f3-4d32-b87f-f1e586e2faca" Jul 21 21:00:49.735372 ip-10-0-140-97 kubenswrapper[2575]: E0721 21:00:49.735142 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4nvs6" podUID="7a92ce6a-922d-4196-819d-c03a22e230d1" Jul 21 21:00:49.735372 ip-10-0-140-97 kubenswrapper[2575]: E0721 21:00:49.735212 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5694f" podUID="e3dcf4f7-a30b-46f5-9c5d-a77f61d88c51" Jul 21 21:00:51.734570 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:51.734522 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4nvs6" Jul 21 21:00:51.734570 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:51.734568 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5694f" Jul 21 21:00:51.735095 ip-10-0-140-97 kubenswrapper[2575]: E0721 21:00:51.734669 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4nvs6" podUID="7a92ce6a-922d-4196-819d-c03a22e230d1" Jul 21 21:00:51.735095 ip-10-0-140-97 kubenswrapper[2575]: E0721 21:00:51.734728 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5694f" podUID="e3dcf4f7-a30b-46f5-9c5d-a77f61d88c51" Jul 21 21:00:51.735095 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:51.734784 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6fx5v" Jul 21 21:00:51.735095 ip-10-0-140-97 kubenswrapper[2575]: E0721 21:00:51.734913 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6fx5v" podUID="67b28cbf-d3f3-4d32-b87f-f1e586e2faca" Jul 21 21:00:52.078870 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:52.078787 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/67b28cbf-d3f3-4d32-b87f-f1e586e2faca-metrics-certs\") pod \"network-metrics-daemon-6fx5v\" (UID: \"67b28cbf-d3f3-4d32-b87f-f1e586e2faca\") " pod="openshift-multus/network-metrics-daemon-6fx5v" Jul 21 21:00:52.079041 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:52.078889 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-djhxf\" (UniqueName: \"kubernetes.io/projected/7a92ce6a-922d-4196-819d-c03a22e230d1-kube-api-access-djhxf\") pod \"network-check-target-4nvs6\" (UID: \"7a92ce6a-922d-4196-819d-c03a22e230d1\") " pod="openshift-network-diagnostics/network-check-target-4nvs6" Jul 21 21:00:52.079041 ip-10-0-140-97 kubenswrapper[2575]: E0721 21:00:52.078955 2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 21 21:00:52.079041 ip-10-0-140-97 kubenswrapper[2575]: E0721 21:00:52.079019 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 21 21:00:52.079041 ip-10-0-140-97 kubenswrapper[2575]: E0721 21:00:52.079040 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 21 21:00:52.079239 ip-10-0-140-97 kubenswrapper[2575]: E0721 21:00:52.079056 2575 projected.go:196] Error preparing data for projected volume kube-api-access-djhxf for pod openshift-network-diagnostics/network-check-target-4nvs6: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 21 21:00:52.079239 ip-10-0-140-97 kubenswrapper[2575]: E0721 21:00:52.079022 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/67b28cbf-d3f3-4d32-b87f-f1e586e2faca-metrics-certs podName:67b28cbf-d3f3-4d32-b87f-f1e586e2faca nodeName:}" failed. No retries permitted until 2026-07-21 21:01:00.079006917 +0000 UTC m=+18.013756561 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/67b28cbf-d3f3-4d32-b87f-f1e586e2faca-metrics-certs") pod "network-metrics-daemon-6fx5v" (UID: "67b28cbf-d3f3-4d32-b87f-f1e586e2faca") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 21 21:00:52.079239 ip-10-0-140-97 kubenswrapper[2575]: E0721 21:00:52.079111 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7a92ce6a-922d-4196-819d-c03a22e230d1-kube-api-access-djhxf podName:7a92ce6a-922d-4196-819d-c03a22e230d1 nodeName:}" failed. No retries permitted until 2026-07-21 21:01:00.079095891 +0000 UTC m=+18.013845538 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-djhxf" (UniqueName: "kubernetes.io/projected/7a92ce6a-922d-4196-819d-c03a22e230d1-kube-api-access-djhxf") pod "network-check-target-4nvs6" (UID: "7a92ce6a-922d-4196-819d-c03a22e230d1") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 21 21:00:52.179413 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:52.179364 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e3dcf4f7-a30b-46f5-9c5d-a77f61d88c51-original-pull-secret\") pod \"global-pull-secret-syncer-5694f\" (UID: \"e3dcf4f7-a30b-46f5-9c5d-a77f61d88c51\") " pod="kube-system/global-pull-secret-syncer-5694f" Jul 21 21:00:52.179626 ip-10-0-140-97 kubenswrapper[2575]: E0721 21:00:52.179599 2575 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 21 21:00:52.179698 ip-10-0-140-97 kubenswrapper[2575]: E0721 21:00:52.179667 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e3dcf4f7-a30b-46f5-9c5d-a77f61d88c51-original-pull-secret podName:e3dcf4f7-a30b-46f5-9c5d-a77f61d88c51 nodeName:}" failed. No retries permitted until 2026-07-21 21:01:00.179648567 +0000 UTC m=+18.114398213 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e3dcf4f7-a30b-46f5-9c5d-a77f61d88c51-original-pull-secret") pod "global-pull-secret-syncer-5694f" (UID: "e3dcf4f7-a30b-46f5-9c5d-a77f61d88c51") : object "kube-system"/"original-pull-secret" not registered Jul 21 21:00:53.734800 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:53.734762 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4nvs6" Jul 21 21:00:53.735235 ip-10-0-140-97 kubenswrapper[2575]: E0721 21:00:53.734887 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4nvs6" podUID="7a92ce6a-922d-4196-819d-c03a22e230d1" Jul 21 21:00:53.735235 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:53.735149 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6fx5v" Jul 21 21:00:53.735347 ip-10-0-140-97 kubenswrapper[2575]: E0721 21:00:53.735286 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6fx5v" podUID="67b28cbf-d3f3-4d32-b87f-f1e586e2faca" Jul 21 21:00:53.735347 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:53.735333 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5694f" Jul 21 21:00:53.735463 ip-10-0-140-97 kubenswrapper[2575]: E0721 21:00:53.735410 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5694f" podUID="e3dcf4f7-a30b-46f5-9c5d-a77f61d88c51" Jul 21 21:00:55.735286 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:55.735246 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4nvs6" Jul 21 21:00:55.735286 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:55.735286 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5694f" Jul 21 21:00:55.735838 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:55.735250 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6fx5v" Jul 21 21:00:55.735838 ip-10-0-140-97 kubenswrapper[2575]: E0721 21:00:55.735369 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4nvs6" podUID="7a92ce6a-922d-4196-819d-c03a22e230d1" Jul 21 21:00:55.735838 ip-10-0-140-97 kubenswrapper[2575]: E0721 21:00:55.735446 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5694f" podUID="e3dcf4f7-a30b-46f5-9c5d-a77f61d88c51" Jul 21 21:00:55.735838 ip-10-0-140-97 kubenswrapper[2575]: E0721 21:00:55.735565 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6fx5v" podUID="67b28cbf-d3f3-4d32-b87f-f1e586e2faca" Jul 21 21:00:57.735053 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:57.735019 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4nvs6" Jul 21 21:00:57.735053 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:57.735046 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5694f" Jul 21 21:00:57.735581 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:57.735022 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6fx5v" Jul 21 21:00:57.735581 ip-10-0-140-97 kubenswrapper[2575]: E0721 21:00:57.735124 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4nvs6" podUID="7a92ce6a-922d-4196-819d-c03a22e230d1" Jul 21 21:00:57.735581 ip-10-0-140-97 kubenswrapper[2575]: E0721 21:00:57.735231 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6fx5v" podUID="67b28cbf-d3f3-4d32-b87f-f1e586e2faca" Jul 21 21:00:57.735581 ip-10-0-140-97 kubenswrapper[2575]: E0721 21:00:57.735323 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5694f" podUID="e3dcf4f7-a30b-46f5-9c5d-a77f61d88c51" Jul 21 21:00:59.734988 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:59.734953 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4nvs6" Jul 21 21:00:59.734988 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:59.734991 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6fx5v" Jul 21 21:00:59.735477 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:00:59.735072 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5694f" Jul 21 21:00:59.735477 ip-10-0-140-97 kubenswrapper[2575]: E0721 21:00:59.735080 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4nvs6" podUID="7a92ce6a-922d-4196-819d-c03a22e230d1" Jul 21 21:00:59.735477 ip-10-0-140-97 kubenswrapper[2575]: E0721 21:00:59.735197 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5694f" podUID="e3dcf4f7-a30b-46f5-9c5d-a77f61d88c51" Jul 21 21:00:59.735477 ip-10-0-140-97 kubenswrapper[2575]: E0721 21:00:59.735292 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6fx5v" podUID="67b28cbf-d3f3-4d32-b87f-f1e586e2faca" Jul 21 21:01:00.141674 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:01:00.141581 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-djhxf\" (UniqueName: \"kubernetes.io/projected/7a92ce6a-922d-4196-819d-c03a22e230d1-kube-api-access-djhxf\") pod \"network-check-target-4nvs6\" (UID: \"7a92ce6a-922d-4196-819d-c03a22e230d1\") " pod="openshift-network-diagnostics/network-check-target-4nvs6" Jul 21 21:01:00.141674 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:01:00.141633 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/67b28cbf-d3f3-4d32-b87f-f1e586e2faca-metrics-certs\") pod \"network-metrics-daemon-6fx5v\" (UID: \"67b28cbf-d3f3-4d32-b87f-f1e586e2faca\") " pod="openshift-multus/network-metrics-daemon-6fx5v" Jul 21 21:01:00.141896 ip-10-0-140-97 kubenswrapper[2575]: E0721 21:01:00.141741 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 21 21:01:00.141896 ip-10-0-140-97 kubenswrapper[2575]: E0721 21:01:00.141761 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 21 21:01:00.141896 ip-10-0-140-97 kubenswrapper[2575]: E0721 21:01:00.141764 2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 21 21:01:00.141896 ip-10-0-140-97 kubenswrapper[2575]: E0721 21:01:00.141840 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/67b28cbf-d3f3-4d32-b87f-f1e586e2faca-metrics-certs podName:67b28cbf-d3f3-4d32-b87f-f1e586e2faca nodeName:}" failed. No retries permitted until 2026-07-21 21:01:16.14181993 +0000 UTC m=+34.076569586 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/67b28cbf-d3f3-4d32-b87f-f1e586e2faca-metrics-certs") pod "network-metrics-daemon-6fx5v" (UID: "67b28cbf-d3f3-4d32-b87f-f1e586e2faca") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 21 21:01:00.141896 ip-10-0-140-97 kubenswrapper[2575]: E0721 21:01:00.141773 2575 projected.go:196] Error preparing data for projected volume kube-api-access-djhxf for pod openshift-network-diagnostics/network-check-target-4nvs6: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 21 21:01:00.142146 ip-10-0-140-97 kubenswrapper[2575]: E0721 21:01:00.141906 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7a92ce6a-922d-4196-819d-c03a22e230d1-kube-api-access-djhxf podName:7a92ce6a-922d-4196-819d-c03a22e230d1 nodeName:}" failed. No retries permitted until 2026-07-21 21:01:16.141892973 +0000 UTC m=+34.076642616 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-djhxf" (UniqueName: "kubernetes.io/projected/7a92ce6a-922d-4196-819d-c03a22e230d1-kube-api-access-djhxf") pod "network-check-target-4nvs6" (UID: "7a92ce6a-922d-4196-819d-c03a22e230d1") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 21 21:01:00.242343 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:01:00.242311 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e3dcf4f7-a30b-46f5-9c5d-a77f61d88c51-original-pull-secret\") pod \"global-pull-secret-syncer-5694f\" (UID: \"e3dcf4f7-a30b-46f5-9c5d-a77f61d88c51\") " pod="kube-system/global-pull-secret-syncer-5694f" Jul 21 21:01:00.242501 ip-10-0-140-97 kubenswrapper[2575]: E0721 21:01:00.242475 2575 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 21 21:01:00.242589 ip-10-0-140-97 kubenswrapper[2575]: E0721 21:01:00.242577 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e3dcf4f7-a30b-46f5-9c5d-a77f61d88c51-original-pull-secret podName:e3dcf4f7-a30b-46f5-9c5d-a77f61d88c51 nodeName:}" failed. No retries permitted until 2026-07-21 21:01:16.242556611 +0000 UTC m=+34.177306260 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e3dcf4f7-a30b-46f5-9c5d-a77f61d88c51-original-pull-secret") pod "global-pull-secret-syncer-5694f" (UID: "e3dcf4f7-a30b-46f5-9c5d-a77f61d88c51") : object "kube-system"/"original-pull-secret" not registered Jul 21 21:01:01.734804 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:01:01.734771 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5694f" Jul 21 21:01:01.735241 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:01:01.734887 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6fx5v" Jul 21 21:01:01.735241 ip-10-0-140-97 kubenswrapper[2575]: E0721 21:01:01.734898 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5694f" podUID="e3dcf4f7-a30b-46f5-9c5d-a77f61d88c51" Jul 21 21:01:01.735241 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:01:01.734920 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4nvs6" Jul 21 21:01:01.735241 ip-10-0-140-97 kubenswrapper[2575]: E0721 21:01:01.735008 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6fx5v" podUID="67b28cbf-d3f3-4d32-b87f-f1e586e2faca" Jul 21 21:01:01.735241 ip-10-0-140-97 kubenswrapper[2575]: E0721 21:01:01.735092 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4nvs6" podUID="7a92ce6a-922d-4196-819d-c03a22e230d1" Jul 21 21:01:03.734596 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:01:03.734560 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6fx5v" Jul 21 21:01:03.735025 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:01:03.734560 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4nvs6" Jul 21 21:01:03.735025 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:01:03.734560 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5694f" Jul 21 21:01:03.735025 ip-10-0-140-97 kubenswrapper[2575]: E0721 21:01:03.734704 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6fx5v" podUID="67b28cbf-d3f3-4d32-b87f-f1e586e2faca" Jul 21 21:01:03.735025 ip-10-0-140-97 kubenswrapper[2575]: E0721 21:01:03.734812 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4nvs6" podUID="7a92ce6a-922d-4196-819d-c03a22e230d1" Jul 21 21:01:03.735025 ip-10-0-140-97 kubenswrapper[2575]: E0721 21:01:03.734881 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5694f" podUID="e3dcf4f7-a30b-46f5-9c5d-a77f61d88c51" Jul 21 21:01:04.854289 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:01:04.853153 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-5pbk4" event={"ID":"b69ef6b6-fca5-47e8-abc5-566b34e77058","Type":"ContainerStarted","Data":"5e446081f1e95d97bf80f4e38bf285f84fd7c3d707c6233c4ba7ac3ba0a20065"} Jul 21 21:01:04.856270 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:01:04.856014 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-97.ec2.internal" event={"ID":"810679f65980473b59e3cca08ad4f0fc","Type":"ContainerStarted","Data":"9714988e0e0e9513c80ed439cb484695df5ce2bfbb0ad79fd287c9170df0de03"} Jul 21 21:01:04.858916 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:01:04.858855 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-lphp9" event={"ID":"6bb08697-52d4-4ebf-b058-5771faf60563","Type":"ContainerStarted","Data":"a1fa199aff679053bc49d522d4bf4a156195ce3778d5b06d7cc2d113a9595d0e"} Jul 21 21:01:04.873006 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:01:04.872941 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-97.ec2.internal" podStartSLOduration=21.872262966 podStartE2EDuration="21.872262966s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-21 21:00:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 21:00:47.821285454 +0000 UTC m=+5.756035121" watchObservedRunningTime="2026-07-21 21:01:04.872262966 +0000 UTC m=+22.807012633" Jul 21 21:01:04.892509 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:01:04.892454 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-5pbk4" podStartSLOduration=2.893811896 podStartE2EDuration="22.892433243s" totalImagesPullingTime="19.998621347s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-21 21:00:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 21:01:04.871660958 +0000 UTC m=+22.806410625" watchObservedRunningTime="2026-07-21 21:01:04.892433243 +0000 UTC m=+22.827182911" Jul 21 21:01:04.892845 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:01:04.892802 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-lphp9" podStartSLOduration=2.851315079 podStartE2EDuration="22.892784086s" totalImagesPullingTime="20.041469007s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-21 21:00:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 21:01:04.891979912 +0000 UTC m=+22.826729602" watchObservedRunningTime="2026-07-21 21:01:04.892784086 +0000 UTC m=+22.827533754" Jul 21 21:01:04.908123 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:01:04.907670 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-97.ec2.internal" podStartSLOduration=21.907649092 podStartE2EDuration="21.907649092s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-21 21:00:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 21:01:04.907476563 +0000 UTC m=+22.842226247" watchObservedRunningTime="2026-07-21 21:01:04.907649092 +0000 UTC m=+22.842398760" Jul 21 21:01:05.735238 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:01:05.735079 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6fx5v" Jul 21 21:01:05.735367 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:01:05.735132 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4nvs6" Jul 21 21:01:05.735367 ip-10-0-140-97 kubenswrapper[2575]: E0721 21:01:05.735317 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6fx5v" podUID="67b28cbf-d3f3-4d32-b87f-f1e586e2faca" Jul 21 21:01:05.735440 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:01:05.735133 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5694f" Jul 21 21:01:05.735440 ip-10-0-140-97 kubenswrapper[2575]: E0721 21:01:05.735391 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4nvs6" podUID="7a92ce6a-922d-4196-819d-c03a22e230d1" Jul 21 21:01:05.735509 ip-10-0-140-97 kubenswrapper[2575]: E0721 21:01:05.735451 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5694f" podUID="e3dcf4f7-a30b-46f5-9c5d-a77f61d88c51" Jul 21 21:01:05.862448 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:01:05.862420 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x9lqv" event={"ID":"6a0eccd2-79bf-4f17-8217-6533fff9fbbb","Type":"ContainerStarted","Data":"08f989f0b9ac84977dcaa609bb61e292a351bd7cea705355ca43ef9873bd6bd8"} Jul 21 21:01:05.863806 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:01:05.863786 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-2czn5" event={"ID":"1fd6b467-f1bf-45b5-8d30-96201ffccd36","Type":"ContainerStarted","Data":"3d2da08a47fa986d6fa404ea8df870019bb469cd5a471a07a0a41134b74cc17f"} Jul 21 21:01:05.865162 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:01:05.865143 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-vdkfn" event={"ID":"3877fdb0-b49b-444a-8f8c-597d3f935383","Type":"ContainerStarted","Data":"109be6b06e067c2359442dac744b154a1d5ca5ff5cabd73b6cb1a32ce9c88cc7"} Jul 21 21:01:05.866555 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:01:05.866524 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-jhz69" event={"ID":"ce7f9a12-9e5b-4634-83d3-babc1e854a66","Type":"ContainerStarted","Data":"ae107c5d90908b7a14d238cf0814be7ea2539c6eb7378fc70e552f6e851480a1"} Jul 21 21:01:05.867716 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:01:05.867700 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-cq6dd" event={"ID":"aaa2e752-150d-4b89-852d-64473f98f593","Type":"ContainerStarted","Data":"cf68686867d61bcab0edd1a6c9ff4ccbe321e6f1149cf7b4534179e34dad819e"} Jul 21 21:01:05.870517 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:01:05.870500 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-m794j" event={"ID":"edd93c23-9f22-4d2c-9342-7f6c398fd6bb","Type":"ContainerStarted","Data":"62c5442ae0d139bdc0598a15f26423ccffadf2d00d3750798de5f250ba04243f"} Jul 21 21:01:05.870607 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:01:05.870569 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-m794j" event={"ID":"edd93c23-9f22-4d2c-9342-7f6c398fd6bb","Type":"ContainerStarted","Data":"e12499c6fd8baf42e83bf361c6a467d6eabfcd64d40d7d28d485b338800b71df"} Jul 21 21:01:05.870655 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:01:05.870606 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-m794j" event={"ID":"edd93c23-9f22-4d2c-9342-7f6c398fd6bb","Type":"ContainerStarted","Data":"d5b6e1ecc71dc9d3714dde654bf2a8d190f631b5141ef9c2b60bd24f6a1e39e2"} Jul 21 21:01:05.870655 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:01:05.870639 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-m794j" event={"ID":"edd93c23-9f22-4d2c-9342-7f6c398fd6bb","Type":"ContainerStarted","Data":"416f0596917dfcd5e64c0bba570e448364972df3d129dff35eac60d55f0ca22d"} Jul 21 21:01:05.870725 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:01:05.870666 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-m794j" event={"ID":"edd93c23-9f22-4d2c-9342-7f6c398fd6bb","Type":"ContainerStarted","Data":"c77f10119ed6713fad64dd9dcaa609e524f210bef2c98963b1286a615e13ed77"} Jul 21 21:01:05.870725 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:01:05.870698 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-m794j" event={"ID":"edd93c23-9f22-4d2c-9342-7f6c398fd6bb","Type":"ContainerStarted","Data":"10962b93b6a831be47ed3b17e4d4d84051d928888303c8f514e42e7192a8e547"} Jul 21 21:01:05.871868 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:01:05.871850 2575 generic.go:356] "Generic (PLEG): container finished" podID="d648b77f-43a8-4540-b59c-63768e78c245" containerID="cc8f4b7d76c36259c339d524cea1c2d5db160ed2a175bcbeeb4d0d3797290b12" exitCode=0 Jul 21 21:01:05.871954 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:01:05.871916 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-952f6" event={"ID":"d648b77f-43a8-4540-b59c-63768e78c245","Type":"ContainerDied","Data":"cc8f4b7d76c36259c339d524cea1c2d5db160ed2a175bcbeeb4d0d3797290b12"} Jul 21 21:01:05.881481 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:01:05.881445 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-2czn5" podStartSLOduration=3.894482093 podStartE2EDuration="23.881434968s" totalImagesPullingTime="19.986952875s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-21 21:00:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 21:01:05.880799337 +0000 UTC m=+23.815549002" watchObservedRunningTime="2026-07-21 21:01:05.881434968 +0000 UTC m=+23.816184634" Jul 21 21:01:05.899101 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:01:05.899065 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-vdkfn" podStartSLOduration=3.9212084860000003 podStartE2EDuration="23.89905306s" totalImagesPullingTime="19.977844574s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-21 21:00:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 21:01:05.8990226 +0000 UTC m=+23.833772277" watchObservedRunningTime="2026-07-21 21:01:05.89905306 +0000 UTC m=+23.833802727" Jul 21 21:01:05.940304 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:01:05.940272 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-cq6dd" podStartSLOduration=3.9832603090000003 podStartE2EDuration="23.940261643s" totalImagesPullingTime="19.957001334s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 21:00:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 21:01:05.939955437 +0000 UTC m=+23.874705126" watchObservedRunningTime="2026-07-21 21:01:05.940261643 +0000 UTC m=+23.875011309" Jul 21 21:01:05.955550 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:01:05.955503 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-jhz69" podStartSLOduration=4.004812967 podStartE2EDuration="23.955493445s" totalImagesPullingTime="19.950680478s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-21 21:00:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 21:01:05.955204824 +0000 UTC m=+23.889954489" watchObservedRunningTime="2026-07-21 21:01:05.955493445 +0000 UTC m=+23.890243111" Jul 21 21:01:06.593168 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:01:06.593135 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-cq6dd" Jul 21 21:01:06.593978 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:01:06.593952 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-cq6dd" Jul 21 21:01:06.705343 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:01:06.705319 2575 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Jul 21 21:01:06.876017 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:01:06.875926 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x9lqv" event={"ID":"6a0eccd2-79bf-4f17-8217-6533fff9fbbb","Type":"ContainerStarted","Data":"8c74146aca5a52adac3aa94668855baf9805e312a07fce46a0b852398bdd9697"} Jul 21 21:01:06.876515 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:01:06.876339 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-cq6dd" Jul 21 21:01:06.876605 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:01:06.876590 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-cq6dd" Jul 21 21:01:07.643576 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:01:07.643460 2575 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-21T21:01:06.705339271Z","UUID":"29bd4d67-aa84-4afd-8cb3-632e28d989b0","Handler":null,"Name":"","Endpoint":""} Jul 21 21:01:07.645252 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:01:07.645219 2575 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Jul 21 21:01:07.645252 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:01:07.645257 2575 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Jul 21 21:01:07.735146 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:01:07.734987 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5694f" Jul 21 21:01:07.735146 ip-10-0-140-97 kubenswrapper[2575]: E0721 21:01:07.735107 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5694f" podUID="e3dcf4f7-a30b-46f5-9c5d-a77f61d88c51" Jul 21 21:01:07.735146 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:01:07.735141 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4nvs6" Jul 21 21:01:07.735408 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:01:07.735158 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6fx5v" Jul 21 21:01:07.735408 ip-10-0-140-97 kubenswrapper[2575]: E0721 21:01:07.735226 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4nvs6" podUID="7a92ce6a-922d-4196-819d-c03a22e230d1" Jul 21 21:01:07.735408 ip-10-0-140-97 kubenswrapper[2575]: E0721 21:01:07.735308 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6fx5v" podUID="67b28cbf-d3f3-4d32-b87f-f1e586e2faca" Jul 21 21:01:07.882468 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:01:07.882442 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-m794j" event={"ID":"edd93c23-9f22-4d2c-9342-7f6c398fd6bb","Type":"ContainerStarted","Data":"184df024d095b98f87b474104bad135a42f272cfaa8c77681ae16846db695a16"} Jul 21 21:01:08.887930 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:01:08.887672 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x9lqv" event={"ID":"6a0eccd2-79bf-4f17-8217-6533fff9fbbb","Type":"ContainerStarted","Data":"9f8ad3e8fa9670438efc572917c8ac003de7e3b40bd32d1ceb9d632f5bffb117"} Jul 21 21:01:08.907172 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:01:08.907124 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x9lqv" podStartSLOduration=4.063020191 podStartE2EDuration="26.907110139s" totalImagesPullingTime="22.844089948s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-21 21:00:42 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 21:01:08.906521992 +0000 UTC m=+26.841271886" watchObservedRunningTime="2026-07-21 21:01:08.907110139 +0000 UTC m=+26.841859805" Jul 21 21:01:09.734903 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:01:09.734853 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4nvs6" Jul 21 21:01:09.734903 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:01:09.734892 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5694f" Jul 21 21:01:09.735158 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:01:09.734908 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6fx5v" Jul 21 21:01:09.735158 ip-10-0-140-97 kubenswrapper[2575]: E0721 21:01:09.735003 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5694f" podUID="e3dcf4f7-a30b-46f5-9c5d-a77f61d88c51" Jul 21 21:01:09.735158 ip-10-0-140-97 kubenswrapper[2575]: E0721 21:01:09.735060 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6fx5v" podUID="67b28cbf-d3f3-4d32-b87f-f1e586e2faca" Jul 21 21:01:09.735275 ip-10-0-140-97 kubenswrapper[2575]: E0721 21:01:09.735151 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4nvs6" podUID="7a92ce6a-922d-4196-819d-c03a22e230d1" Jul 21 21:01:10.894976 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:01:10.894747 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-m794j" event={"ID":"edd93c23-9f22-4d2c-9342-7f6c398fd6bb","Type":"ContainerStarted","Data":"ca0bcdc5fe718dca704b4cdbd7ce338e52ce9c64b38d43ec700eda0ccc816ae7"} Jul 21 21:01:10.895735 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:01:10.895025 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-m794j" Jul 21 21:01:10.896731 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:01:10.896708 2575 generic.go:356] "Generic (PLEG): container finished" podID="d648b77f-43a8-4540-b59c-63768e78c245" containerID="fe901679923eef055e2d5cd67b8ba471fe1daf62b4a74e94c150bc29808c2e46" exitCode=0 Jul 21 21:01:10.896840 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:01:10.896745 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-952f6" event={"ID":"d648b77f-43a8-4540-b59c-63768e78c245","Type":"ContainerDied","Data":"fe901679923eef055e2d5cd67b8ba471fe1daf62b4a74e94c150bc29808c2e46"} Jul 21 21:01:10.911645 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:01:10.911626 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-m794j" Jul 21 21:01:10.921979 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:01:10.921943 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-m794j" podStartSLOduration=8.873781119 podStartE2EDuration="28.921931063s" totalImagesPullingTime="20.048149944s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-21 21:00:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 21:01:10.921633211 +0000 UTC m=+28.856382878" watchObservedRunningTime="2026-07-21 21:01:10.921931063 +0000 UTC m=+28.856680729" Jul 21 21:01:11.734442 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:01:11.734406 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5694f" Jul 21 21:01:11.734835 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:01:11.734453 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6fx5v" Jul 21 21:01:11.734835 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:01:11.734454 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4nvs6" Jul 21 21:01:11.734835 ip-10-0-140-97 kubenswrapper[2575]: E0721 21:01:11.734579 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5694f" podUID="e3dcf4f7-a30b-46f5-9c5d-a77f61d88c51" Jul 21 21:01:11.734835 ip-10-0-140-97 kubenswrapper[2575]: E0721 21:01:11.734579 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4nvs6" podUID="7a92ce6a-922d-4196-819d-c03a22e230d1" Jul 21 21:01:11.734835 ip-10-0-140-97 kubenswrapper[2575]: E0721 21:01:11.734650 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6fx5v" podUID="67b28cbf-d3f3-4d32-b87f-f1e586e2faca" Jul 21 21:01:11.899474 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:01:11.899444 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-m794j" Jul 21 21:01:11.899890 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:01:11.899546 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-m794j" Jul 21 21:01:11.913655 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:01:11.913635 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-m794j" Jul 21 21:01:12.017070 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:01:12.016851 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-5694f"] Jul 21 21:01:12.017070 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:01:12.016982 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5694f" Jul 21 21:01:12.017265 ip-10-0-140-97 kubenswrapper[2575]: E0721 21:01:12.017099 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5694f" podUID="e3dcf4f7-a30b-46f5-9c5d-a77f61d88c51" Jul 21 21:01:12.017726 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:01:12.017705 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-4nvs6"] Jul 21 21:01:12.017865 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:01:12.017810 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4nvs6" Jul 21 21:01:12.017981 ip-10-0-140-97 kubenswrapper[2575]: E0721 21:01:12.017957 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4nvs6" podUID="7a92ce6a-922d-4196-819d-c03a22e230d1" Jul 21 21:01:12.018433 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:01:12.018411 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-6fx5v"] Jul 21 21:01:12.018543 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:01:12.018500 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6fx5v" Jul 21 21:01:12.018629 ip-10-0-140-97 kubenswrapper[2575]: E0721 21:01:12.018608 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6fx5v" podUID="67b28cbf-d3f3-4d32-b87f-f1e586e2faca" Jul 21 21:01:12.902559 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:01:12.902289 2575 generic.go:356] "Generic (PLEG): container finished" podID="d648b77f-43a8-4540-b59c-63768e78c245" containerID="9073074cd6976e1d6bf52c256698b03c8dc9d77306418162bdfa549623dedbfe" exitCode=0 Jul 21 21:01:12.903065 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:01:12.902371 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-952f6" event={"ID":"d648b77f-43a8-4540-b59c-63768e78c245","Type":"ContainerDied","Data":"9073074cd6976e1d6bf52c256698b03c8dc9d77306418162bdfa549623dedbfe"} Jul 21 21:01:13.734921 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:01:13.734890 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5694f" Jul 21 21:01:13.735133 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:01:13.734890 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4nvs6" Jul 21 21:01:13.735133 ip-10-0-140-97 kubenswrapper[2575]: E0721 21:01:13.735080 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4nvs6" podUID="7a92ce6a-922d-4196-819d-c03a22e230d1" Jul 21 21:01:13.735133 ip-10-0-140-97 kubenswrapper[2575]: E0721 21:01:13.734982 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5694f" podUID="e3dcf4f7-a30b-46f5-9c5d-a77f61d88c51" Jul 21 21:01:13.735133 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:01:13.734890 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6fx5v" Jul 21 21:01:13.735304 ip-10-0-140-97 kubenswrapper[2575]: E0721 21:01:13.735193 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6fx5v" podUID="67b28cbf-d3f3-4d32-b87f-f1e586e2faca" Jul 21 21:01:14.909691 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:01:14.909654 2575 generic.go:356] "Generic (PLEG): container finished" podID="d648b77f-43a8-4540-b59c-63768e78c245" containerID="2b81f55822c65f3dcf0fc29194ae59f45f9b5a538417b150c16ce70095c7b0ed" exitCode=0 Jul 21 21:01:14.910338 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:01:14.909696 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-952f6" event={"ID":"d648b77f-43a8-4540-b59c-63768e78c245","Type":"ContainerDied","Data":"2b81f55822c65f3dcf0fc29194ae59f45f9b5a538417b150c16ce70095c7b0ed"} Jul 21 21:01:15.735127 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:01:15.735094 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4nvs6" Jul 21 21:01:15.735317 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:01:15.735205 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5694f" Jul 21 21:01:15.735317 ip-10-0-140-97 kubenswrapper[2575]: E0721 21:01:15.735212 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4nvs6" podUID="7a92ce6a-922d-4196-819d-c03a22e230d1" Jul 21 21:01:15.735317 ip-10-0-140-97 kubenswrapper[2575]: E0721 21:01:15.735299 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5694f" podUID="e3dcf4f7-a30b-46f5-9c5d-a77f61d88c51" Jul 21 21:01:15.735483 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:01:15.735214 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6fx5v" Jul 21 21:01:15.735483 ip-10-0-140-97 kubenswrapper[2575]: E0721 21:01:15.735404 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6fx5v" podUID="67b28cbf-d3f3-4d32-b87f-f1e586e2faca" Jul 21 21:01:16.169999 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:01:16.169964 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-djhxf\" (UniqueName: \"kubernetes.io/projected/7a92ce6a-922d-4196-819d-c03a22e230d1-kube-api-access-djhxf\") pod \"network-check-target-4nvs6\" (UID: \"7a92ce6a-922d-4196-819d-c03a22e230d1\") " pod="openshift-network-diagnostics/network-check-target-4nvs6" Jul 21 21:01:16.170365 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:01:16.170010 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/67b28cbf-d3f3-4d32-b87f-f1e586e2faca-metrics-certs\") pod \"network-metrics-daemon-6fx5v\" (UID: \"67b28cbf-d3f3-4d32-b87f-f1e586e2faca\") " pod="openshift-multus/network-metrics-daemon-6fx5v" Jul 21 21:01:16.170365 ip-10-0-140-97 kubenswrapper[2575]: E0721 21:01:16.170097 2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 21 21:01:16.170365 ip-10-0-140-97 kubenswrapper[2575]: E0721 21:01:16.170118 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 21 21:01:16.170365 ip-10-0-140-97 kubenswrapper[2575]: E0721 21:01:16.170141 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/67b28cbf-d3f3-4d32-b87f-f1e586e2faca-metrics-certs podName:67b28cbf-d3f3-4d32-b87f-f1e586e2faca nodeName:}" failed. No retries permitted until 2026-07-21 21:01:48.170128348 +0000 UTC m=+66.104877993 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/67b28cbf-d3f3-4d32-b87f-f1e586e2faca-metrics-certs") pod "network-metrics-daemon-6fx5v" (UID: "67b28cbf-d3f3-4d32-b87f-f1e586e2faca") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 21 21:01:16.170365 ip-10-0-140-97 kubenswrapper[2575]: E0721 21:01:16.170145 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 21 21:01:16.170365 ip-10-0-140-97 kubenswrapper[2575]: E0721 21:01:16.170159 2575 projected.go:196] Error preparing data for projected volume kube-api-access-djhxf for pod openshift-network-diagnostics/network-check-target-4nvs6: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 21 21:01:16.170365 ip-10-0-140-97 kubenswrapper[2575]: E0721 21:01:16.170207 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7a92ce6a-922d-4196-819d-c03a22e230d1-kube-api-access-djhxf podName:7a92ce6a-922d-4196-819d-c03a22e230d1 nodeName:}" failed. No retries permitted until 2026-07-21 21:01:48.170194795 +0000 UTC m=+66.104944439 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-djhxf" (UniqueName: "kubernetes.io/projected/7a92ce6a-922d-4196-819d-c03a22e230d1-kube-api-access-djhxf") pod "network-check-target-4nvs6" (UID: "7a92ce6a-922d-4196-819d-c03a22e230d1") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 21 21:01:16.271198 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:01:16.271160 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e3dcf4f7-a30b-46f5-9c5d-a77f61d88c51-original-pull-secret\") pod \"global-pull-secret-syncer-5694f\" (UID: \"e3dcf4f7-a30b-46f5-9c5d-a77f61d88c51\") " pod="kube-system/global-pull-secret-syncer-5694f" Jul 21 21:01:16.271357 ip-10-0-140-97 kubenswrapper[2575]: E0721 21:01:16.271311 2575 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 21 21:01:16.271403 ip-10-0-140-97 kubenswrapper[2575]: E0721 21:01:16.271373 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e3dcf4f7-a30b-46f5-9c5d-a77f61d88c51-original-pull-secret podName:e3dcf4f7-a30b-46f5-9c5d-a77f61d88c51 nodeName:}" failed. No retries permitted until 2026-07-21 21:01:48.271357882 +0000 UTC m=+66.206107526 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/e3dcf4f7-a30b-46f5-9c5d-a77f61d88c51-original-pull-secret") pod "global-pull-secret-syncer-5694f" (UID: "e3dcf4f7-a30b-46f5-9c5d-a77f61d88c51") : object "kube-system"/"original-pull-secret" not registered Jul 21 21:01:17.407417 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:01:17.407384 2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-97.ec2.internal" event="NodeReady" Jul 21 21:01:17.407936 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:01:17.407540 2575 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Jul 21 21:01:17.460969 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:01:17.460894 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-hfmtx"] Jul 21 21:01:17.484235 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:01:17.484203 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-bfws8"] Jul 21 21:01:17.484382 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:01:17.484332 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-hfmtx" Jul 21 21:01:17.487633 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:01:17.487612 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Jul 21 21:01:17.487762 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:01:17.487610 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Jul 21 21:01:17.488022 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:01:17.487874 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Jul 21 21:01:17.488022 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:01:17.487967 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-bv9cb\"" Jul 21 21:01:17.500439 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:01:17.500415 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-hfmtx"] Jul 21 21:01:17.500560 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:01:17.500450 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-bfws8"] Jul 21 21:01:17.500560 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:01:17.500475 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-bfws8" Jul 21 21:01:17.503352 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:01:17.503332 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-wp8nm\"" Jul 21 21:01:17.503472 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:01:17.503360 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Jul 21 21:01:17.503838 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:01:17.503822 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Jul 21 21:01:17.682602 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:01:17.682563 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/4f86eef6-207f-4ad3-baa2-98cef60dbe52-cert\") pod \"ingress-canary-hfmtx\" (UID: \"4f86eef6-207f-4ad3-baa2-98cef60dbe52\") " pod="openshift-ingress-canary/ingress-canary-hfmtx" Jul 21 21:01:17.682602 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:01:17.682607 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e057811f-88be-47b7-9d41-4815de59f0df-metrics-tls\") pod \"dns-default-bfws8\" (UID: \"e057811f-88be-47b7-9d41-4815de59f0df\") " pod="openshift-dns/dns-default-bfws8" Jul 21 21:01:17.682839 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:01:17.682634 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e057811f-88be-47b7-9d41-4815de59f0df-tmp-dir\") pod \"dns-default-bfws8\" (UID: \"e057811f-88be-47b7-9d41-4815de59f0df\") " pod="openshift-dns/dns-default-bfws8" Jul 21 21:01:17.682839 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:01:17.682685 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vrhp2\" (UniqueName: \"kubernetes.io/projected/4f86eef6-207f-4ad3-baa2-98cef60dbe52-kube-api-access-vrhp2\") pod \"ingress-canary-hfmtx\" (UID: \"4f86eef6-207f-4ad3-baa2-98cef60dbe52\") " pod="openshift-ingress-canary/ingress-canary-hfmtx" Jul 21 21:01:17.682839 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:01:17.682784 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/e057811f-88be-47b7-9d41-4815de59f0df-config-volume\") pod \"dns-default-bfws8\" (UID: \"e057811f-88be-47b7-9d41-4815de59f0df\") " pod="openshift-dns/dns-default-bfws8" Jul 21 21:01:17.682839 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:01:17.682815 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z49jx\" (UniqueName: \"kubernetes.io/projected/e057811f-88be-47b7-9d41-4815de59f0df-kube-api-access-z49jx\") pod \"dns-default-bfws8\" (UID: \"e057811f-88be-47b7-9d41-4815de59f0df\") " pod="openshift-dns/dns-default-bfws8" Jul 21 21:01:17.734870 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:01:17.734789 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4nvs6" Jul 21 21:01:17.734870 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:01:17.734837 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5694f" Jul 21 21:01:17.735070 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:01:17.734805 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6fx5v" Jul 21 21:01:17.737685 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:01:17.737660 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 21 21:01:17.737811 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:01:17.737784 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 21 21:01:17.737811 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:01:17.737792 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 21 21:01:17.737967 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:01:17.737945 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-slzpm\"" Jul 21 21:01:17.738031 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:01:17.737981 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 21 21:01:17.738819 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:01:17.738797 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-cvwws\"" Jul 21 21:01:17.783303 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:01:17.783272 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/4f86eef6-207f-4ad3-baa2-98cef60dbe52-cert\") pod \"ingress-canary-hfmtx\" (UID: \"4f86eef6-207f-4ad3-baa2-98cef60dbe52\") " pod="openshift-ingress-canary/ingress-canary-hfmtx" Jul 21 21:01:17.783409 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:01:17.783305 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e057811f-88be-47b7-9d41-4815de59f0df-metrics-tls\") pod \"dns-default-bfws8\" (UID: \"e057811f-88be-47b7-9d41-4815de59f0df\") " pod="openshift-dns/dns-default-bfws8" Jul 21 21:01:17.783409 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:01:17.783322 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e057811f-88be-47b7-9d41-4815de59f0df-tmp-dir\") pod \"dns-default-bfws8\" (UID: \"e057811f-88be-47b7-9d41-4815de59f0df\") " pod="openshift-dns/dns-default-bfws8" Jul 21 21:01:17.783517 ip-10-0-140-97 kubenswrapper[2575]: E0721 21:01:17.783446 2575 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 21 21:01:17.783596 ip-10-0-140-97 kubenswrapper[2575]: E0721 21:01:17.783519 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e057811f-88be-47b7-9d41-4815de59f0df-metrics-tls podName:e057811f-88be-47b7-9d41-4815de59f0df nodeName:}" failed. No retries permitted until 2026-07-21 21:01:18.283498406 +0000 UTC m=+36.218248065 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/e057811f-88be-47b7-9d41-4815de59f0df-metrics-tls") pod "dns-default-bfws8" (UID: "e057811f-88be-47b7-9d41-4815de59f0df") : secret "dns-default-metrics-tls" not found Jul 21 21:01:17.783655 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:01:17.783591 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vrhp2\" (UniqueName: \"kubernetes.io/projected/4f86eef6-207f-4ad3-baa2-98cef60dbe52-kube-api-access-vrhp2\") pod \"ingress-canary-hfmtx\" (UID: \"4f86eef6-207f-4ad3-baa2-98cef60dbe52\") " pod="openshift-ingress-canary/ingress-canary-hfmtx" Jul 21 21:01:17.783655 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:01:17.783644 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e057811f-88be-47b7-9d41-4815de59f0df-tmp-dir\") pod \"dns-default-bfws8\" (UID: \"e057811f-88be-47b7-9d41-4815de59f0df\") " pod="openshift-dns/dns-default-bfws8" Jul 21 21:01:17.787217 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:01:17.783648 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/e057811f-88be-47b7-9d41-4815de59f0df-config-volume\") pod \"dns-default-bfws8\" (UID: \"e057811f-88be-47b7-9d41-4815de59f0df\") " pod="openshift-dns/dns-default-bfws8" Jul 21 21:01:17.787217 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:01:17.783994 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z49jx\" (UniqueName: \"kubernetes.io/projected/e057811f-88be-47b7-9d41-4815de59f0df-kube-api-access-z49jx\") pod \"dns-default-bfws8\" (UID: \"e057811f-88be-47b7-9d41-4815de59f0df\") " pod="openshift-dns/dns-default-bfws8" Jul 21 21:01:17.787217 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:01:17.784704 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/e057811f-88be-47b7-9d41-4815de59f0df-config-volume\") pod \"dns-default-bfws8\" (UID: \"e057811f-88be-47b7-9d41-4815de59f0df\") " pod="openshift-dns/dns-default-bfws8" Jul 21 21:01:17.789201 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:01:17.789181 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/4f86eef6-207f-4ad3-baa2-98cef60dbe52-cert\") pod \"ingress-canary-hfmtx\" (UID: \"4f86eef6-207f-4ad3-baa2-98cef60dbe52\") " pod="openshift-ingress-canary/ingress-canary-hfmtx" Jul 21 21:01:17.793296 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:01:17.793272 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vrhp2\" (UniqueName: \"kubernetes.io/projected/4f86eef6-207f-4ad3-baa2-98cef60dbe52-kube-api-access-vrhp2\") pod \"ingress-canary-hfmtx\" (UID: \"4f86eef6-207f-4ad3-baa2-98cef60dbe52\") " pod="openshift-ingress-canary/ingress-canary-hfmtx" Jul 21 21:01:17.793552 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:01:17.793499 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z49jx\" (UniqueName: \"kubernetes.io/projected/e057811f-88be-47b7-9d41-4815de59f0df-kube-api-access-z49jx\") pod \"dns-default-bfws8\" (UID: \"e057811f-88be-47b7-9d41-4815de59f0df\") " pod="openshift-dns/dns-default-bfws8" Jul 21 21:01:17.795346 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:01:17.795327 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-hfmtx" Jul 21 21:01:18.288015 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:01:18.287981 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e057811f-88be-47b7-9d41-4815de59f0df-metrics-tls\") pod \"dns-default-bfws8\" (UID: \"e057811f-88be-47b7-9d41-4815de59f0df\") " pod="openshift-dns/dns-default-bfws8" Jul 21 21:01:18.288174 ip-10-0-140-97 kubenswrapper[2575]: E0721 21:01:18.288121 2575 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 21 21:01:18.288222 ip-10-0-140-97 kubenswrapper[2575]: E0721 21:01:18.288178 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e057811f-88be-47b7-9d41-4815de59f0df-metrics-tls podName:e057811f-88be-47b7-9d41-4815de59f0df nodeName:}" failed. No retries permitted until 2026-07-21 21:01:19.288165575 +0000 UTC m=+37.222915220 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/e057811f-88be-47b7-9d41-4815de59f0df-metrics-tls") pod "dns-default-bfws8" (UID: "e057811f-88be-47b7-9d41-4815de59f0df") : secret "dns-default-metrics-tls" not found Jul 21 21:01:19.188410 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:01:19.188271 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-hfmtx"] Jul 21 21:01:19.191991 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:01:19.191958 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4f86eef6_207f_4ad3_baa2_98cef60dbe52.slice/crio-974994254ca3b7d066309951ca8ee7a04ceac01a91287674f3e25d754064d15c WatchSource:0}: Error finding container 974994254ca3b7d066309951ca8ee7a04ceac01a91287674f3e25d754064d15c: Status 404 returned error can't find the container with id 974994254ca3b7d066309951ca8ee7a04ceac01a91287674f3e25d754064d15c Jul 21 21:01:19.293982 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:01:19.293957 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e057811f-88be-47b7-9d41-4815de59f0df-metrics-tls\") pod \"dns-default-bfws8\" (UID: \"e057811f-88be-47b7-9d41-4815de59f0df\") " pod="openshift-dns/dns-default-bfws8" Jul 21 21:01:19.294224 ip-10-0-140-97 kubenswrapper[2575]: E0721 21:01:19.294099 2575 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 21 21:01:19.294224 ip-10-0-140-97 kubenswrapper[2575]: E0721 21:01:19.294162 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e057811f-88be-47b7-9d41-4815de59f0df-metrics-tls podName:e057811f-88be-47b7-9d41-4815de59f0df nodeName:}" failed. No retries permitted until 2026-07-21 21:01:21.294148065 +0000 UTC m=+39.228897712 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/e057811f-88be-47b7-9d41-4815de59f0df-metrics-tls") pod "dns-default-bfws8" (UID: "e057811f-88be-47b7-9d41-4815de59f0df") : secret "dns-default-metrics-tls" not found Jul 21 21:01:19.923365 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:01:19.923329 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-hfmtx" event={"ID":"4f86eef6-207f-4ad3-baa2-98cef60dbe52","Type":"ContainerStarted","Data":"974994254ca3b7d066309951ca8ee7a04ceac01a91287674f3e25d754064d15c"} Jul 21 21:01:19.926597 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:01:19.926569 2575 generic.go:356] "Generic (PLEG): container finished" podID="d648b77f-43a8-4540-b59c-63768e78c245" containerID="d1e93582ff0d9e876ac1de6a6011b0d555f25d6673f8ae49e14f62fb69e27e2f" exitCode=0 Jul 21 21:01:19.926728 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:01:19.926602 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-952f6" event={"ID":"d648b77f-43a8-4540-b59c-63768e78c245","Type":"ContainerDied","Data":"d1e93582ff0d9e876ac1de6a6011b0d555f25d6673f8ae49e14f62fb69e27e2f"} Jul 21 21:01:20.932824 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:01:20.932785 2575 generic.go:356] "Generic (PLEG): container finished" podID="d648b77f-43a8-4540-b59c-63768e78c245" containerID="6b5faa3e31945feaefcf7dcdd2dbbf6155acc1f39fc758e2a46e7a4d601ef0f1" exitCode=0 Jul 21 21:01:20.933314 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:01:20.932835 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-952f6" event={"ID":"d648b77f-43a8-4540-b59c-63768e78c245","Type":"ContainerDied","Data":"6b5faa3e31945feaefcf7dcdd2dbbf6155acc1f39fc758e2a46e7a4d601ef0f1"} Jul 21 21:01:21.309577 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:01:21.309411 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e057811f-88be-47b7-9d41-4815de59f0df-metrics-tls\") pod \"dns-default-bfws8\" (UID: \"e057811f-88be-47b7-9d41-4815de59f0df\") " pod="openshift-dns/dns-default-bfws8" Jul 21 21:01:21.309669 ip-10-0-140-97 kubenswrapper[2575]: E0721 21:01:21.309602 2575 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 21 21:01:21.309704 ip-10-0-140-97 kubenswrapper[2575]: E0721 21:01:21.309683 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e057811f-88be-47b7-9d41-4815de59f0df-metrics-tls podName:e057811f-88be-47b7-9d41-4815de59f0df nodeName:}" failed. No retries permitted until 2026-07-21 21:01:25.309660093 +0000 UTC m=+43.244409741 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/e057811f-88be-47b7-9d41-4815de59f0df-metrics-tls") pod "dns-default-bfws8" (UID: "e057811f-88be-47b7-9d41-4815de59f0df") : secret "dns-default-metrics-tls" not found Jul 21 21:01:21.936206 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:01:21.936165 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-hfmtx" event={"ID":"4f86eef6-207f-4ad3-baa2-98cef60dbe52","Type":"ContainerStarted","Data":"dd3f8fb58a63431838e2d5dbaaf4d27494b881fc09aabd73542698513e40cbc5"} Jul 21 21:01:21.939224 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:01:21.939198 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-952f6" event={"ID":"d648b77f-43a8-4540-b59c-63768e78c245","Type":"ContainerStarted","Data":"cf5b9be4b414638d836805603065cc84ea77e7a7760a8cfc334d4b14eae9bfb6"} Jul 21 21:01:21.952243 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:01:21.952203 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-hfmtx" podStartSLOduration=2.675329137 podStartE2EDuration="4.952192154s" totalImagesPullingTime="2.276863017s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 21:01:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 21:01:21.951622034 +0000 UTC m=+39.886371700" watchObservedRunningTime="2026-07-21 21:01:21.952192154 +0000 UTC m=+39.886941820" Jul 21 21:01:21.978053 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:01:21.978003 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-952f6" podStartSLOduration=9.458636732 podStartE2EDuration="39.97798555s" totalImagesPullingTime="30.519348818s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-21 21:00:42 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 21:01:21.977068012 +0000 UTC m=+39.911817678" watchObservedRunningTime="2026-07-21 21:01:21.97798555 +0000 UTC m=+39.912735217" Jul 21 21:01:25.338129 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:01:25.338093 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e057811f-88be-47b7-9d41-4815de59f0df-metrics-tls\") pod \"dns-default-bfws8\" (UID: \"e057811f-88be-47b7-9d41-4815de59f0df\") " pod="openshift-dns/dns-default-bfws8" Jul 21 21:01:25.338513 ip-10-0-140-97 kubenswrapper[2575]: E0721 21:01:25.338239 2575 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 21 21:01:25.338513 ip-10-0-140-97 kubenswrapper[2575]: E0721 21:01:25.338300 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e057811f-88be-47b7-9d41-4815de59f0df-metrics-tls podName:e057811f-88be-47b7-9d41-4815de59f0df nodeName:}" failed. No retries permitted until 2026-07-21 21:01:33.338284991 +0000 UTC m=+51.273034635 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/e057811f-88be-47b7-9d41-4815de59f0df-metrics-tls") pod "dns-default-bfws8" (UID: "e057811f-88be-47b7-9d41-4815de59f0df") : secret "dns-default-metrics-tls" not found Jul 21 21:01:33.389370 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:01:33.389331 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e057811f-88be-47b7-9d41-4815de59f0df-metrics-tls\") pod \"dns-default-bfws8\" (UID: \"e057811f-88be-47b7-9d41-4815de59f0df\") " pod="openshift-dns/dns-default-bfws8" Jul 21 21:01:33.389847 ip-10-0-140-97 kubenswrapper[2575]: E0721 21:01:33.389441 2575 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 21 21:01:33.389847 ip-10-0-140-97 kubenswrapper[2575]: E0721 21:01:33.389493 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e057811f-88be-47b7-9d41-4815de59f0df-metrics-tls podName:e057811f-88be-47b7-9d41-4815de59f0df nodeName:}" failed. No retries permitted until 2026-07-21 21:01:49.389480092 +0000 UTC m=+67.324229736 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/e057811f-88be-47b7-9d41-4815de59f0df-metrics-tls") pod "dns-default-bfws8" (UID: "e057811f-88be-47b7-9d41-4815de59f0df") : secret "dns-default-metrics-tls" not found Jul 21 21:01:33.727994 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:01:33.727908 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-76cf8966f8-7px4w"] Jul 21 21:01:33.740129 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:01:33.740103 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-7px4w" Jul 21 21:01:33.741395 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:01:33.741366 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-76cf8966f8-7px4w"] Jul 21 21:01:33.744881 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:01:33.744848 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Jul 21 21:01:33.744881 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:01:33.744842 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-tgt79\"" Jul 21 21:01:33.744881 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:01:33.744875 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Jul 21 21:01:33.745098 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:01:33.744855 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\"" Jul 21 21:01:33.745098 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:01:33.744855 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\"" Jul 21 21:01:33.792657 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:01:33.792636 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pmb25\" (UniqueName: \"kubernetes.io/projected/76e94cf3-ab49-4698-8ac9-bf3ad1dcdf3a-kube-api-access-pmb25\") pod \"cluster-monitoring-operator-76cf8966f8-7px4w\" (UID: \"76e94cf3-ab49-4698-8ac9-bf3ad1dcdf3a\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-7px4w" Jul 21 21:01:33.792779 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:01:33.792675 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/76e94cf3-ab49-4698-8ac9-bf3ad1dcdf3a-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-76cf8966f8-7px4w\" (UID: \"76e94cf3-ab49-4698-8ac9-bf3ad1dcdf3a\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-7px4w" Jul 21 21:01:33.792779 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:01:33.792695 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/76e94cf3-ab49-4698-8ac9-bf3ad1dcdf3a-telemetry-config\") pod \"cluster-monitoring-operator-76cf8966f8-7px4w\" (UID: \"76e94cf3-ab49-4698-8ac9-bf3ad1dcdf3a\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-7px4w" Jul 21 21:01:33.813177 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:01:33.813154 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-7zc94"] Jul 21 21:01:33.828555 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:01:33.828520 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-5bcbc94c78-b2hpw"] Jul 21 21:01:33.828649 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:01:33.828633 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-7zc94" Jul 21 21:01:33.830874 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:01:33.830858 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\"" Jul 21 21:01:33.830991 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:01:33.830971 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-hgkdh\"" Jul 21 21:01:33.831149 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:01:33.831119 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\"" Jul 21 21:01:33.831268 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:01:33.831179 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\"" Jul 21 21:01:33.831268 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:01:33.831237 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\"" Jul 21 21:01:33.842468 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:01:33.842448 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-69864f84f6-qzxnd"] Jul 21 21:01:33.842614 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:01:33.842597 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-b2hpw" Jul 21 21:01:33.844888 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:01:33.844869 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\"" Jul 21 21:01:33.844991 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:01:33.844902 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\"" Jul 21 21:01:33.844991 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:01:33.844954 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\"" Jul 21 21:01:33.844991 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:01:33.844960 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\"" Jul 21 21:01:33.845139 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:01:33.844954 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-kpcg6\"" Jul 21 21:01:33.858310 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:01:33.858293 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-7zc94"] Jul 21 21:01:33.858388 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:01:33.858319 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-5bcbc94c78-b2hpw"] Jul 21 21:01:33.858388 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:01:33.858334 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-69864f84f6-qzxnd"] Jul 21 21:01:33.858456 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:01:33.858393 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-69864f84f6-qzxnd" Jul 21 21:01:33.860738 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:01:33.860721 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\"" Jul 21 21:01:33.860934 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:01:33.860911 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\"" Jul 21 21:01:33.861030 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:01:33.860938 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\"" Jul 21 21:01:33.861030 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:01:33.860993 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\"" Jul 21 21:01:33.861175 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:01:33.861074 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Jul 21 21:01:33.861231 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:01:33.861177 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-795s6\"" Jul 21 21:01:33.861231 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:01:33.861208 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Jul 21 21:01:33.893111 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:01:33.893082 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2adb651d-e8aa-4e76-878c-e5cdafafbe5f-service-ca-bundle\") pod \"router-default-69864f84f6-qzxnd\" (UID: \"2adb651d-e8aa-4e76-878c-e5cdafafbe5f\") " pod="openshift-ingress/router-default-69864f84f6-qzxnd" Jul 21 21:01:33.893280 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:01:33.893119 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pmb25\" (UniqueName: \"kubernetes.io/projected/76e94cf3-ab49-4698-8ac9-bf3ad1dcdf3a-kube-api-access-pmb25\") pod \"cluster-monitoring-operator-76cf8966f8-7px4w\" (UID: \"76e94cf3-ab49-4698-8ac9-bf3ad1dcdf3a\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-7px4w" Jul 21 21:01:33.893280 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:01:33.893139 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cwfrc\" (UniqueName: \"kubernetes.io/projected/ca2c8246-50a5-48fb-be89-a95b401523a7-kube-api-access-cwfrc\") pod \"kube-storage-version-migrator-operator-7658d9b749-7zc94\" (UID: \"ca2c8246-50a5-48fb-be89-a95b401523a7\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-7zc94" Jul 21 21:01:33.893280 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:01:33.893159 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/8a30add9-0efd-4cf9-99df-17229907f235-config\") pod \"service-ca-operator-5bcbc94c78-b2hpw\" (UID: \"8a30add9-0efd-4cf9-99df-17229907f235\") " pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-b2hpw" Jul 21 21:01:33.893280 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:01:33.893225 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2adb651d-e8aa-4e76-878c-e5cdafafbe5f-metrics-certs\") pod \"router-default-69864f84f6-qzxnd\" (UID: \"2adb651d-e8aa-4e76-878c-e5cdafafbe5f\") " pod="openshift-ingress/router-default-69864f84f6-qzxnd" Jul 21 21:01:33.893476 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:01:33.893307 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/76e94cf3-ab49-4698-8ac9-bf3ad1dcdf3a-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-76cf8966f8-7px4w\" (UID: \"76e94cf3-ab49-4698-8ac9-bf3ad1dcdf3a\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-7px4w" Jul 21 21:01:33.893476 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:01:33.893325 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/76e94cf3-ab49-4698-8ac9-bf3ad1dcdf3a-telemetry-config\") pod \"cluster-monitoring-operator-76cf8966f8-7px4w\" (UID: \"76e94cf3-ab49-4698-8ac9-bf3ad1dcdf3a\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-7px4w" Jul 21 21:01:33.893476 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:01:33.893341 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/8a30add9-0efd-4cf9-99df-17229907f235-serving-cert\") pod \"service-ca-operator-5bcbc94c78-b2hpw\" (UID: \"8a30add9-0efd-4cf9-99df-17229907f235\") " pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-b2hpw" Jul 21 21:01:33.893476 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:01:33.893363 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/2adb651d-e8aa-4e76-878c-e5cdafafbe5f-stats-auth\") pod \"router-default-69864f84f6-qzxnd\" (UID: \"2adb651d-e8aa-4e76-878c-e5cdafafbe5f\") " pod="openshift-ingress/router-default-69864f84f6-qzxnd" Jul 21 21:01:33.893476 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:01:33.893388 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/2adb651d-e8aa-4e76-878c-e5cdafafbe5f-default-certificate\") pod \"router-default-69864f84f6-qzxnd\" (UID: \"2adb651d-e8aa-4e76-878c-e5cdafafbe5f\") " pod="openshift-ingress/router-default-69864f84f6-qzxnd" Jul 21 21:01:33.893476 ip-10-0-140-97 kubenswrapper[2575]: E0721 21:01:33.893402 2575 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 21 21:01:33.893476 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:01:33.893404 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8pqfj\" (UniqueName: \"kubernetes.io/projected/2adb651d-e8aa-4e76-878c-e5cdafafbe5f-kube-api-access-8pqfj\") pod \"router-default-69864f84f6-qzxnd\" (UID: \"2adb651d-e8aa-4e76-878c-e5cdafafbe5f\") " pod="openshift-ingress/router-default-69864f84f6-qzxnd" Jul 21 21:01:33.893476 ip-10-0-140-97 kubenswrapper[2575]: E0721 21:01:33.893470 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/76e94cf3-ab49-4698-8ac9-bf3ad1dcdf3a-cluster-monitoring-operator-tls podName:76e94cf3-ab49-4698-8ac9-bf3ad1dcdf3a nodeName:}" failed. No retries permitted until 2026-07-21 21:01:34.393449475 +0000 UTC m=+52.328199134 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/76e94cf3-ab49-4698-8ac9-bf3ad1dcdf3a-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-76cf8966f8-7px4w" (UID: "76e94cf3-ab49-4698-8ac9-bf3ad1dcdf3a") : secret "cluster-monitoring-operator-tls" not found Jul 21 21:01:33.893822 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:01:33.893503 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/ca2c8246-50a5-48fb-be89-a95b401523a7-serving-cert\") pod \"kube-storage-version-migrator-operator-7658d9b749-7zc94\" (UID: \"ca2c8246-50a5-48fb-be89-a95b401523a7\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-7zc94" Jul 21 21:01:33.893822 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:01:33.893538 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lbl6s\" (UniqueName: \"kubernetes.io/projected/8a30add9-0efd-4cf9-99df-17229907f235-kube-api-access-lbl6s\") pod \"service-ca-operator-5bcbc94c78-b2hpw\" (UID: \"8a30add9-0efd-4cf9-99df-17229907f235\") " pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-b2hpw" Jul 21 21:01:33.893822 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:01:33.893594 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/ca2c8246-50a5-48fb-be89-a95b401523a7-config\") pod \"kube-storage-version-migrator-operator-7658d9b749-7zc94\" (UID: \"ca2c8246-50a5-48fb-be89-a95b401523a7\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-7zc94" Jul 21 21:01:33.893998 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:01:33.893982 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/76e94cf3-ab49-4698-8ac9-bf3ad1dcdf3a-telemetry-config\") pod \"cluster-monitoring-operator-76cf8966f8-7px4w\" (UID: \"76e94cf3-ab49-4698-8ac9-bf3ad1dcdf3a\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-7px4w" Jul 21 21:01:33.905078 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:01:33.905062 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pmb25\" (UniqueName: \"kubernetes.io/projected/76e94cf3-ab49-4698-8ac9-bf3ad1dcdf3a-kube-api-access-pmb25\") pod \"cluster-monitoring-operator-76cf8966f8-7px4w\" (UID: \"76e94cf3-ab49-4698-8ac9-bf3ad1dcdf3a\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-7px4w" Jul 21 21:01:33.993959 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:01:33.993886 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2adb651d-e8aa-4e76-878c-e5cdafafbe5f-service-ca-bundle\") pod \"router-default-69864f84f6-qzxnd\" (UID: \"2adb651d-e8aa-4e76-878c-e5cdafafbe5f\") " pod="openshift-ingress/router-default-69864f84f6-qzxnd" Jul 21 21:01:33.993959 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:01:33.993929 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cwfrc\" (UniqueName: \"kubernetes.io/projected/ca2c8246-50a5-48fb-be89-a95b401523a7-kube-api-access-cwfrc\") pod \"kube-storage-version-migrator-operator-7658d9b749-7zc94\" (UID: \"ca2c8246-50a5-48fb-be89-a95b401523a7\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-7zc94" Jul 21 21:01:33.993959 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:01:33.993955 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/8a30add9-0efd-4cf9-99df-17229907f235-config\") pod \"service-ca-operator-5bcbc94c78-b2hpw\" (UID: \"8a30add9-0efd-4cf9-99df-17229907f235\") " pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-b2hpw" Jul 21 21:01:33.994197 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:01:33.993980 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2adb651d-e8aa-4e76-878c-e5cdafafbe5f-metrics-certs\") pod \"router-default-69864f84f6-qzxnd\" (UID: \"2adb651d-e8aa-4e76-878c-e5cdafafbe5f\") " pod="openshift-ingress/router-default-69864f84f6-qzxnd" Jul 21 21:01:33.994197 ip-10-0-140-97 kubenswrapper[2575]: E0721 21:01:33.994038 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/2adb651d-e8aa-4e76-878c-e5cdafafbe5f-service-ca-bundle podName:2adb651d-e8aa-4e76-878c-e5cdafafbe5f nodeName:}" failed. No retries permitted until 2026-07-21 21:01:34.494020701 +0000 UTC m=+52.428770345 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/2adb651d-e8aa-4e76-878c-e5cdafafbe5f-service-ca-bundle") pod "router-default-69864f84f6-qzxnd" (UID: "2adb651d-e8aa-4e76-878c-e5cdafafbe5f") : configmap references non-existent config key: service-ca.crt Jul 21 21:01:33.994197 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:01:33.994063 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/8a30add9-0efd-4cf9-99df-17229907f235-serving-cert\") pod \"service-ca-operator-5bcbc94c78-b2hpw\" (UID: \"8a30add9-0efd-4cf9-99df-17229907f235\") " pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-b2hpw" Jul 21 21:01:33.994197 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:01:33.994093 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/2adb651d-e8aa-4e76-878c-e5cdafafbe5f-stats-auth\") pod \"router-default-69864f84f6-qzxnd\" (UID: \"2adb651d-e8aa-4e76-878c-e5cdafafbe5f\") " pod="openshift-ingress/router-default-69864f84f6-qzxnd" Jul 21 21:01:33.994197 ip-10-0-140-97 kubenswrapper[2575]: E0721 21:01:33.994099 2575 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 21 21:01:33.994197 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:01:33.994123 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/2adb651d-e8aa-4e76-878c-e5cdafafbe5f-default-certificate\") pod \"router-default-69864f84f6-qzxnd\" (UID: \"2adb651d-e8aa-4e76-878c-e5cdafafbe5f\") " pod="openshift-ingress/router-default-69864f84f6-qzxnd" Jul 21 21:01:33.994197 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:01:33.994145 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8pqfj\" (UniqueName: \"kubernetes.io/projected/2adb651d-e8aa-4e76-878c-e5cdafafbe5f-kube-api-access-8pqfj\") pod \"router-default-69864f84f6-qzxnd\" (UID: \"2adb651d-e8aa-4e76-878c-e5cdafafbe5f\") " pod="openshift-ingress/router-default-69864f84f6-qzxnd" Jul 21 21:01:33.994197 ip-10-0-140-97 kubenswrapper[2575]: E0721 21:01:33.994175 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2adb651d-e8aa-4e76-878c-e5cdafafbe5f-metrics-certs podName:2adb651d-e8aa-4e76-878c-e5cdafafbe5f nodeName:}" failed. No retries permitted until 2026-07-21 21:01:34.49415824 +0000 UTC m=+52.428907888 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2adb651d-e8aa-4e76-878c-e5cdafafbe5f-metrics-certs") pod "router-default-69864f84f6-qzxnd" (UID: "2adb651d-e8aa-4e76-878c-e5cdafafbe5f") : secret "router-metrics-certs-default" not found Jul 21 21:01:33.994197 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:01:33.994194 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/ca2c8246-50a5-48fb-be89-a95b401523a7-serving-cert\") pod \"kube-storage-version-migrator-operator-7658d9b749-7zc94\" (UID: \"ca2c8246-50a5-48fb-be89-a95b401523a7\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-7zc94" Jul 21 21:01:33.994671 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:01:33.994222 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lbl6s\" (UniqueName: \"kubernetes.io/projected/8a30add9-0efd-4cf9-99df-17229907f235-kube-api-access-lbl6s\") pod \"service-ca-operator-5bcbc94c78-b2hpw\" (UID: \"8a30add9-0efd-4cf9-99df-17229907f235\") " pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-b2hpw" Jul 21 21:01:33.994671 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:01:33.994256 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/ca2c8246-50a5-48fb-be89-a95b401523a7-config\") pod \"kube-storage-version-migrator-operator-7658d9b749-7zc94\" (UID: \"ca2c8246-50a5-48fb-be89-a95b401523a7\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-7zc94" Jul 21 21:01:33.994671 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:01:33.994584 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/8a30add9-0efd-4cf9-99df-17229907f235-config\") pod \"service-ca-operator-5bcbc94c78-b2hpw\" (UID: \"8a30add9-0efd-4cf9-99df-17229907f235\") " pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-b2hpw" Jul 21 21:01:33.994858 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:01:33.994839 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/ca2c8246-50a5-48fb-be89-a95b401523a7-config\") pod \"kube-storage-version-migrator-operator-7658d9b749-7zc94\" (UID: \"ca2c8246-50a5-48fb-be89-a95b401523a7\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-7zc94" Jul 21 21:01:33.996407 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:01:33.996382 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/ca2c8246-50a5-48fb-be89-a95b401523a7-serving-cert\") pod \"kube-storage-version-migrator-operator-7658d9b749-7zc94\" (UID: \"ca2c8246-50a5-48fb-be89-a95b401523a7\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-7zc94" Jul 21 21:01:33.996498 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:01:33.996476 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/2adb651d-e8aa-4e76-878c-e5cdafafbe5f-default-certificate\") pod \"router-default-69864f84f6-qzxnd\" (UID: \"2adb651d-e8aa-4e76-878c-e5cdafafbe5f\") " pod="openshift-ingress/router-default-69864f84f6-qzxnd" Jul 21 21:01:33.996721 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:01:33.996696 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/8a30add9-0efd-4cf9-99df-17229907f235-serving-cert\") pod \"service-ca-operator-5bcbc94c78-b2hpw\" (UID: \"8a30add9-0efd-4cf9-99df-17229907f235\") " pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-b2hpw" Jul 21 21:01:33.996863 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:01:33.996844 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/2adb651d-e8aa-4e76-878c-e5cdafafbe5f-stats-auth\") pod \"router-default-69864f84f6-qzxnd\" (UID: \"2adb651d-e8aa-4e76-878c-e5cdafafbe5f\") " pod="openshift-ingress/router-default-69864f84f6-qzxnd" Jul 21 21:01:34.007277 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:01:34.007249 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cwfrc\" (UniqueName: \"kubernetes.io/projected/ca2c8246-50a5-48fb-be89-a95b401523a7-kube-api-access-cwfrc\") pod \"kube-storage-version-migrator-operator-7658d9b749-7zc94\" (UID: \"ca2c8246-50a5-48fb-be89-a95b401523a7\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-7zc94" Jul 21 21:01:34.007399 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:01:34.007380 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lbl6s\" (UniqueName: \"kubernetes.io/projected/8a30add9-0efd-4cf9-99df-17229907f235-kube-api-access-lbl6s\") pod \"service-ca-operator-5bcbc94c78-b2hpw\" (UID: \"8a30add9-0efd-4cf9-99df-17229907f235\") " pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-b2hpw" Jul 21 21:01:34.007513 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:01:34.007498 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8pqfj\" (UniqueName: \"kubernetes.io/projected/2adb651d-e8aa-4e76-878c-e5cdafafbe5f-kube-api-access-8pqfj\") pod \"router-default-69864f84f6-qzxnd\" (UID: \"2adb651d-e8aa-4e76-878c-e5cdafafbe5f\") " pod="openshift-ingress/router-default-69864f84f6-qzxnd" Jul 21 21:01:34.137396 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:01:34.137368 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-7zc94" Jul 21 21:01:34.151495 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:01:34.151468 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-b2hpw" Jul 21 21:01:34.292574 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:01:34.292542 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-7zc94"] Jul 21 21:01:34.295041 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:01:34.295011 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podca2c8246_50a5_48fb_be89_a95b401523a7.slice/crio-92c9918dae86c1cf826d954411df1ac96ccea833a1ba9ace406b9210c64367c6 WatchSource:0}: Error finding container 92c9918dae86c1cf826d954411df1ac96ccea833a1ba9ace406b9210c64367c6: Status 404 returned error can't find the container with id 92c9918dae86c1cf826d954411df1ac96ccea833a1ba9ace406b9210c64367c6 Jul 21 21:01:34.306462 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:01:34.306438 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-5bcbc94c78-b2hpw"] Jul 21 21:01:34.310510 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:01:34.310487 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8a30add9_0efd_4cf9_99df_17229907f235.slice/crio-f329a74715d0689bec4103f3e3f952e2023582f4a261459294b63dadf26f8018 WatchSource:0}: Error finding container f329a74715d0689bec4103f3e3f952e2023582f4a261459294b63dadf26f8018: Status 404 returned error can't find the container with id f329a74715d0689bec4103f3e3f952e2023582f4a261459294b63dadf26f8018 Jul 21 21:01:34.397214 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:01:34.397183 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/76e94cf3-ab49-4698-8ac9-bf3ad1dcdf3a-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-76cf8966f8-7px4w\" (UID: \"76e94cf3-ab49-4698-8ac9-bf3ad1dcdf3a\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-7px4w" Jul 21 21:01:34.397557 ip-10-0-140-97 kubenswrapper[2575]: E0721 21:01:34.397306 2575 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 21 21:01:34.397557 ip-10-0-140-97 kubenswrapper[2575]: E0721 21:01:34.397354 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/76e94cf3-ab49-4698-8ac9-bf3ad1dcdf3a-cluster-monitoring-operator-tls podName:76e94cf3-ab49-4698-8ac9-bf3ad1dcdf3a nodeName:}" failed. No retries permitted until 2026-07-21 21:01:35.397340948 +0000 UTC m=+53.332090592 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/76e94cf3-ab49-4698-8ac9-bf3ad1dcdf3a-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-76cf8966f8-7px4w" (UID: "76e94cf3-ab49-4698-8ac9-bf3ad1dcdf3a") : secret "cluster-monitoring-operator-tls" not found Jul 21 21:01:34.497943 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:01:34.497874 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2adb651d-e8aa-4e76-878c-e5cdafafbe5f-service-ca-bundle\") pod \"router-default-69864f84f6-qzxnd\" (UID: \"2adb651d-e8aa-4e76-878c-e5cdafafbe5f\") " pod="openshift-ingress/router-default-69864f84f6-qzxnd" Jul 21 21:01:34.497943 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:01:34.497914 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2adb651d-e8aa-4e76-878c-e5cdafafbe5f-metrics-certs\") pod \"router-default-69864f84f6-qzxnd\" (UID: \"2adb651d-e8aa-4e76-878c-e5cdafafbe5f\") " pod="openshift-ingress/router-default-69864f84f6-qzxnd" Jul 21 21:01:34.498095 ip-10-0-140-97 kubenswrapper[2575]: E0721 21:01:34.498036 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/2adb651d-e8aa-4e76-878c-e5cdafafbe5f-service-ca-bundle podName:2adb651d-e8aa-4e76-878c-e5cdafafbe5f nodeName:}" failed. No retries permitted until 2026-07-21 21:01:35.498018861 +0000 UTC m=+53.432768505 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/2adb651d-e8aa-4e76-878c-e5cdafafbe5f-service-ca-bundle") pod "router-default-69864f84f6-qzxnd" (UID: "2adb651d-e8aa-4e76-878c-e5cdafafbe5f") : configmap references non-existent config key: service-ca.crt Jul 21 21:01:34.498095 ip-10-0-140-97 kubenswrapper[2575]: E0721 21:01:34.498057 2575 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 21 21:01:34.498164 ip-10-0-140-97 kubenswrapper[2575]: E0721 21:01:34.498101 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2adb651d-e8aa-4e76-878c-e5cdafafbe5f-metrics-certs podName:2adb651d-e8aa-4e76-878c-e5cdafafbe5f nodeName:}" failed. No retries permitted until 2026-07-21 21:01:35.498086054 +0000 UTC m=+53.432835697 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2adb651d-e8aa-4e76-878c-e5cdafafbe5f-metrics-certs") pod "router-default-69864f84f6-qzxnd" (UID: "2adb651d-e8aa-4e76-878c-e5cdafafbe5f") : secret "router-metrics-certs-default" not found Jul 21 21:01:34.968208 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:01:34.968172 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-b2hpw" event={"ID":"8a30add9-0efd-4cf9-99df-17229907f235","Type":"ContainerStarted","Data":"f329a74715d0689bec4103f3e3f952e2023582f4a261459294b63dadf26f8018"} Jul 21 21:01:34.969608 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:01:34.969574 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-7zc94" event={"ID":"ca2c8246-50a5-48fb-be89-a95b401523a7","Type":"ContainerStarted","Data":"92c9918dae86c1cf826d954411df1ac96ccea833a1ba9ace406b9210c64367c6"} Jul 21 21:01:35.405041 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:01:35.405005 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/76e94cf3-ab49-4698-8ac9-bf3ad1dcdf3a-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-76cf8966f8-7px4w\" (UID: \"76e94cf3-ab49-4698-8ac9-bf3ad1dcdf3a\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-7px4w" Jul 21 21:01:35.405494 ip-10-0-140-97 kubenswrapper[2575]: E0721 21:01:35.405179 2575 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 21 21:01:35.405494 ip-10-0-140-97 kubenswrapper[2575]: E0721 21:01:35.405265 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/76e94cf3-ab49-4698-8ac9-bf3ad1dcdf3a-cluster-monitoring-operator-tls podName:76e94cf3-ab49-4698-8ac9-bf3ad1dcdf3a nodeName:}" failed. No retries permitted until 2026-07-21 21:01:37.405244441 +0000 UTC m=+55.339994108 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/76e94cf3-ab49-4698-8ac9-bf3ad1dcdf3a-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-76cf8966f8-7px4w" (UID: "76e94cf3-ab49-4698-8ac9-bf3ad1dcdf3a") : secret "cluster-monitoring-operator-tls" not found Jul 21 21:01:35.505935 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:01:35.505903 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2adb651d-e8aa-4e76-878c-e5cdafafbe5f-metrics-certs\") pod \"router-default-69864f84f6-qzxnd\" (UID: \"2adb651d-e8aa-4e76-878c-e5cdafafbe5f\") " pod="openshift-ingress/router-default-69864f84f6-qzxnd" Jul 21 21:01:35.506077 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:01:35.506012 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2adb651d-e8aa-4e76-878c-e5cdafafbe5f-service-ca-bundle\") pod \"router-default-69864f84f6-qzxnd\" (UID: \"2adb651d-e8aa-4e76-878c-e5cdafafbe5f\") " pod="openshift-ingress/router-default-69864f84f6-qzxnd" Jul 21 21:01:35.506077 ip-10-0-140-97 kubenswrapper[2575]: E0721 21:01:35.506047 2575 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 21 21:01:35.506150 ip-10-0-140-97 kubenswrapper[2575]: E0721 21:01:35.506113 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2adb651d-e8aa-4e76-878c-e5cdafafbe5f-metrics-certs podName:2adb651d-e8aa-4e76-878c-e5cdafafbe5f nodeName:}" failed. No retries permitted until 2026-07-21 21:01:37.506098387 +0000 UTC m=+55.440848031 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2adb651d-e8aa-4e76-878c-e5cdafafbe5f-metrics-certs") pod "router-default-69864f84f6-qzxnd" (UID: "2adb651d-e8aa-4e76-878c-e5cdafafbe5f") : secret "router-metrics-certs-default" not found Jul 21 21:01:35.506150 ip-10-0-140-97 kubenswrapper[2575]: E0721 21:01:35.506126 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/2adb651d-e8aa-4e76-878c-e5cdafafbe5f-service-ca-bundle podName:2adb651d-e8aa-4e76-878c-e5cdafafbe5f nodeName:}" failed. No retries permitted until 2026-07-21 21:01:37.506120539 +0000 UTC m=+55.440870183 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/2adb651d-e8aa-4e76-878c-e5cdafafbe5f-service-ca-bundle") pod "router-default-69864f84f6-qzxnd" (UID: "2adb651d-e8aa-4e76-878c-e5cdafafbe5f") : configmap references non-existent config key: service-ca.crt Jul 21 21:01:37.420164 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:01:37.420135 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/76e94cf3-ab49-4698-8ac9-bf3ad1dcdf3a-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-76cf8966f8-7px4w\" (UID: \"76e94cf3-ab49-4698-8ac9-bf3ad1dcdf3a\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-7px4w" Jul 21 21:01:37.420610 ip-10-0-140-97 kubenswrapper[2575]: E0721 21:01:37.420271 2575 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 21 21:01:37.420610 ip-10-0-140-97 kubenswrapper[2575]: E0721 21:01:37.420333 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/76e94cf3-ab49-4698-8ac9-bf3ad1dcdf3a-cluster-monitoring-operator-tls podName:76e94cf3-ab49-4698-8ac9-bf3ad1dcdf3a nodeName:}" failed. No retries permitted until 2026-07-21 21:01:41.420318721 +0000 UTC m=+59.355068364 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/76e94cf3-ab49-4698-8ac9-bf3ad1dcdf3a-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-76cf8966f8-7px4w" (UID: "76e94cf3-ab49-4698-8ac9-bf3ad1dcdf3a") : secret "cluster-monitoring-operator-tls" not found Jul 21 21:01:37.520626 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:01:37.520596 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2adb651d-e8aa-4e76-878c-e5cdafafbe5f-metrics-certs\") pod \"router-default-69864f84f6-qzxnd\" (UID: \"2adb651d-e8aa-4e76-878c-e5cdafafbe5f\") " pod="openshift-ingress/router-default-69864f84f6-qzxnd" Jul 21 21:01:37.520796 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:01:37.520750 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2adb651d-e8aa-4e76-878c-e5cdafafbe5f-service-ca-bundle\") pod \"router-default-69864f84f6-qzxnd\" (UID: \"2adb651d-e8aa-4e76-878c-e5cdafafbe5f\") " pod="openshift-ingress/router-default-69864f84f6-qzxnd" Jul 21 21:01:37.520796 ip-10-0-140-97 kubenswrapper[2575]: E0721 21:01:37.520757 2575 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 21 21:01:37.520898 ip-10-0-140-97 kubenswrapper[2575]: E0721 21:01:37.520818 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2adb651d-e8aa-4e76-878c-e5cdafafbe5f-metrics-certs podName:2adb651d-e8aa-4e76-878c-e5cdafafbe5f nodeName:}" failed. No retries permitted until 2026-07-21 21:01:41.520798461 +0000 UTC m=+59.455548105 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2adb651d-e8aa-4e76-878c-e5cdafafbe5f-metrics-certs") pod "router-default-69864f84f6-qzxnd" (UID: "2adb651d-e8aa-4e76-878c-e5cdafafbe5f") : secret "router-metrics-certs-default" not found Jul 21 21:01:37.520898 ip-10-0-140-97 kubenswrapper[2575]: E0721 21:01:37.520835 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/2adb651d-e8aa-4e76-878c-e5cdafafbe5f-service-ca-bundle podName:2adb651d-e8aa-4e76-878c-e5cdafafbe5f nodeName:}" failed. No retries permitted until 2026-07-21 21:01:41.520825089 +0000 UTC m=+59.455574733 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/2adb651d-e8aa-4e76-878c-e5cdafafbe5f-service-ca-bundle") pod "router-default-69864f84f6-qzxnd" (UID: "2adb651d-e8aa-4e76-878c-e5cdafafbe5f") : configmap references non-existent config key: service-ca.crt Jul 21 21:01:37.979063 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:01:37.979027 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-b2hpw" event={"ID":"8a30add9-0efd-4cf9-99df-17229907f235","Type":"ContainerStarted","Data":"5f9ba926f889efc7afb3b0d2f432352d9d650da9b36592accb3322eb3c813b8d"} Jul 21 21:01:37.980490 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:01:37.980464 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-7zc94" event={"ID":"ca2c8246-50a5-48fb-be89-a95b401523a7","Type":"ContainerStarted","Data":"934d73c3726f025ff75e9210cd3e3f1086c309270229e07e5930fec16b6acd0e"} Jul 21 21:01:37.997475 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:01:37.997436 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-b2hpw" podStartSLOduration=2.20169489 podStartE2EDuration="4.997421206s" totalImagesPullingTime="2.795726316s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 21:01:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 21:01:37.996732758 +0000 UTC m=+55.931482421" watchObservedRunningTime="2026-07-21 21:01:37.997421206 +0000 UTC m=+55.932170873" Jul 21 21:01:38.018111 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:01:38.018067 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-7zc94" podStartSLOduration=2.207964695 podStartE2EDuration="5.018053344s" totalImagesPullingTime="2.810088649s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 21:01:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 21:01:38.017008261 +0000 UTC m=+55.951757926" watchObservedRunningTime="2026-07-21 21:01:38.018053344 +0000 UTC m=+55.952803010" Jul 21 21:01:38.541742 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:01:38.541708 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-wbwbl"] Jul 21 21:01:38.543752 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:01:38.543736 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-wbwbl" Jul 21 21:01:38.546887 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:01:38.546870 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Jul 21 21:01:38.546981 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:01:38.546931 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Jul 21 21:01:38.547188 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:01:38.547171 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-vj8lm\"" Jul 21 21:01:38.547582 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:01:38.547560 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Jul 21 21:01:38.547671 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:01:38.547584 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Jul 21 21:01:38.557635 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:01:38.557616 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-wbwbl"] Jul 21 21:01:38.629819 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:01:38.629794 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/4ebec69c-3c62-4d4b-bb73-3a170576ec58-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-wbwbl\" (UID: \"4ebec69c-3c62-4d4b-bb73-3a170576ec58\") " pod="openshift-insights/insights-runtime-extractor-wbwbl" Jul 21 21:01:38.629930 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:01:38.629822 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/4ebec69c-3c62-4d4b-bb73-3a170576ec58-crio-socket\") pod \"insights-runtime-extractor-wbwbl\" (UID: \"4ebec69c-3c62-4d4b-bb73-3a170576ec58\") " pod="openshift-insights/insights-runtime-extractor-wbwbl" Jul 21 21:01:38.629930 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:01:38.629844 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/4ebec69c-3c62-4d4b-bb73-3a170576ec58-data-volume\") pod \"insights-runtime-extractor-wbwbl\" (UID: \"4ebec69c-3c62-4d4b-bb73-3a170576ec58\") " pod="openshift-insights/insights-runtime-extractor-wbwbl" Jul 21 21:01:38.630009 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:01:38.629941 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jjspk\" (UniqueName: \"kubernetes.io/projected/4ebec69c-3c62-4d4b-bb73-3a170576ec58-kube-api-access-jjspk\") pod \"insights-runtime-extractor-wbwbl\" (UID: \"4ebec69c-3c62-4d4b-bb73-3a170576ec58\") " pod="openshift-insights/insights-runtime-extractor-wbwbl" Jul 21 21:01:38.630009 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:01:38.629967 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/4ebec69c-3c62-4d4b-bb73-3a170576ec58-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-wbwbl\" (UID: \"4ebec69c-3c62-4d4b-bb73-3a170576ec58\") " pod="openshift-insights/insights-runtime-extractor-wbwbl" Jul 21 21:01:38.730428 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:01:38.730393 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/4ebec69c-3c62-4d4b-bb73-3a170576ec58-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-wbwbl\" (UID: \"4ebec69c-3c62-4d4b-bb73-3a170576ec58\") " pod="openshift-insights/insights-runtime-extractor-wbwbl" Jul 21 21:01:38.730428 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:01:38.730432 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/4ebec69c-3c62-4d4b-bb73-3a170576ec58-crio-socket\") pod \"insights-runtime-extractor-wbwbl\" (UID: \"4ebec69c-3c62-4d4b-bb73-3a170576ec58\") " pod="openshift-insights/insights-runtime-extractor-wbwbl" Jul 21 21:01:38.730786 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:01:38.730456 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/4ebec69c-3c62-4d4b-bb73-3a170576ec58-data-volume\") pod \"insights-runtime-extractor-wbwbl\" (UID: \"4ebec69c-3c62-4d4b-bb73-3a170576ec58\") " pod="openshift-insights/insights-runtime-extractor-wbwbl" Jul 21 21:01:38.730786 ip-10-0-140-97 kubenswrapper[2575]: E0721 21:01:38.730602 2575 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 21 21:01:38.730786 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:01:38.730644 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jjspk\" (UniqueName: \"kubernetes.io/projected/4ebec69c-3c62-4d4b-bb73-3a170576ec58-kube-api-access-jjspk\") pod \"insights-runtime-extractor-wbwbl\" (UID: \"4ebec69c-3c62-4d4b-bb73-3a170576ec58\") " pod="openshift-insights/insights-runtime-extractor-wbwbl" Jul 21 21:01:38.730786 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:01:38.730664 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/4ebec69c-3c62-4d4b-bb73-3a170576ec58-crio-socket\") pod \"insights-runtime-extractor-wbwbl\" (UID: \"4ebec69c-3c62-4d4b-bb73-3a170576ec58\") " pod="openshift-insights/insights-runtime-extractor-wbwbl" Jul 21 21:01:38.730786 ip-10-0-140-97 kubenswrapper[2575]: E0721 21:01:38.730680 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/4ebec69c-3c62-4d4b-bb73-3a170576ec58-kube-rbac-proxy-cm podName:4ebec69c-3c62-4d4b-bb73-3a170576ec58 nodeName:}" failed. No retries permitted until 2026-07-21 21:01:39.230659555 +0000 UTC m=+57.165409217 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/4ebec69c-3c62-4d4b-bb73-3a170576ec58-kube-rbac-proxy-cm") pod "insights-runtime-extractor-wbwbl" (UID: "4ebec69c-3c62-4d4b-bb73-3a170576ec58") : configmap "kube-rbac-proxy" not found Jul 21 21:01:38.730786 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:01:38.730721 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/4ebec69c-3c62-4d4b-bb73-3a170576ec58-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-wbwbl\" (UID: \"4ebec69c-3c62-4d4b-bb73-3a170576ec58\") " pod="openshift-insights/insights-runtime-extractor-wbwbl" Jul 21 21:01:38.731244 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:01:38.730835 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/4ebec69c-3c62-4d4b-bb73-3a170576ec58-data-volume\") pod \"insights-runtime-extractor-wbwbl\" (UID: \"4ebec69c-3c62-4d4b-bb73-3a170576ec58\") " pod="openshift-insights/insights-runtime-extractor-wbwbl" Jul 21 21:01:38.731244 ip-10-0-140-97 kubenswrapper[2575]: E0721 21:01:38.730839 2575 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 21 21:01:38.731244 ip-10-0-140-97 kubenswrapper[2575]: E0721 21:01:38.730903 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4ebec69c-3c62-4d4b-bb73-3a170576ec58-insights-runtime-extractor-tls podName:4ebec69c-3c62-4d4b-bb73-3a170576ec58 nodeName:}" failed. No retries permitted until 2026-07-21 21:01:39.230888989 +0000 UTC m=+57.165638647 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/4ebec69c-3c62-4d4b-bb73-3a170576ec58-insights-runtime-extractor-tls") pod "insights-runtime-extractor-wbwbl" (UID: "4ebec69c-3c62-4d4b-bb73-3a170576ec58") : secret "insights-runtime-extractor-tls" not found Jul 21 21:01:38.742543 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:01:38.742505 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jjspk\" (UniqueName: \"kubernetes.io/projected/4ebec69c-3c62-4d4b-bb73-3a170576ec58-kube-api-access-jjspk\") pod \"insights-runtime-extractor-wbwbl\" (UID: \"4ebec69c-3c62-4d4b-bb73-3a170576ec58\") " pod="openshift-insights/insights-runtime-extractor-wbwbl" Jul 21 21:01:39.235082 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:01:39.235047 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/4ebec69c-3c62-4d4b-bb73-3a170576ec58-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-wbwbl\" (UID: \"4ebec69c-3c62-4d4b-bb73-3a170576ec58\") " pod="openshift-insights/insights-runtime-extractor-wbwbl" Jul 21 21:01:39.235272 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:01:39.235147 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/4ebec69c-3c62-4d4b-bb73-3a170576ec58-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-wbwbl\" (UID: \"4ebec69c-3c62-4d4b-bb73-3a170576ec58\") " pod="openshift-insights/insights-runtime-extractor-wbwbl" Jul 21 21:01:39.235272 ip-10-0-140-97 kubenswrapper[2575]: E0721 21:01:39.235190 2575 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 21 21:01:39.235272 ip-10-0-140-97 kubenswrapper[2575]: E0721 21:01:39.235242 2575 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 21 21:01:39.235272 ip-10-0-140-97 kubenswrapper[2575]: E0721 21:01:39.235254 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4ebec69c-3c62-4d4b-bb73-3a170576ec58-insights-runtime-extractor-tls podName:4ebec69c-3c62-4d4b-bb73-3a170576ec58 nodeName:}" failed. No retries permitted until 2026-07-21 21:01:40.235240753 +0000 UTC m=+58.169990397 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/4ebec69c-3c62-4d4b-bb73-3a170576ec58-insights-runtime-extractor-tls") pod "insights-runtime-extractor-wbwbl" (UID: "4ebec69c-3c62-4d4b-bb73-3a170576ec58") : secret "insights-runtime-extractor-tls" not found Jul 21 21:01:39.235438 ip-10-0-140-97 kubenswrapper[2575]: E0721 21:01:39.235291 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/4ebec69c-3c62-4d4b-bb73-3a170576ec58-kube-rbac-proxy-cm podName:4ebec69c-3c62-4d4b-bb73-3a170576ec58 nodeName:}" failed. No retries permitted until 2026-07-21 21:01:40.235276562 +0000 UTC m=+58.170026222 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/4ebec69c-3c62-4d4b-bb73-3a170576ec58-kube-rbac-proxy-cm") pod "insights-runtime-extractor-wbwbl" (UID: "4ebec69c-3c62-4d4b-bb73-3a170576ec58") : configmap "kube-rbac-proxy" not found Jul 21 21:01:40.243888 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:01:40.243854 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/4ebec69c-3c62-4d4b-bb73-3a170576ec58-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-wbwbl\" (UID: \"4ebec69c-3c62-4d4b-bb73-3a170576ec58\") " pod="openshift-insights/insights-runtime-extractor-wbwbl" Jul 21 21:01:40.244225 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:01:40.243936 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/4ebec69c-3c62-4d4b-bb73-3a170576ec58-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-wbwbl\" (UID: \"4ebec69c-3c62-4d4b-bb73-3a170576ec58\") " pod="openshift-insights/insights-runtime-extractor-wbwbl" Jul 21 21:01:40.244225 ip-10-0-140-97 kubenswrapper[2575]: E0721 21:01:40.243971 2575 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 21 21:01:40.244225 ip-10-0-140-97 kubenswrapper[2575]: E0721 21:01:40.244031 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/4ebec69c-3c62-4d4b-bb73-3a170576ec58-kube-rbac-proxy-cm podName:4ebec69c-3c62-4d4b-bb73-3a170576ec58 nodeName:}" failed. No retries permitted until 2026-07-21 21:01:42.244016874 +0000 UTC m=+60.178766518 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/4ebec69c-3c62-4d4b-bb73-3a170576ec58-kube-rbac-proxy-cm") pod "insights-runtime-extractor-wbwbl" (UID: "4ebec69c-3c62-4d4b-bb73-3a170576ec58") : configmap "kube-rbac-proxy" not found Jul 21 21:01:40.244225 ip-10-0-140-97 kubenswrapper[2575]: E0721 21:01:40.244038 2575 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 21 21:01:40.244225 ip-10-0-140-97 kubenswrapper[2575]: E0721 21:01:40.244080 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4ebec69c-3c62-4d4b-bb73-3a170576ec58-insights-runtime-extractor-tls podName:4ebec69c-3c62-4d4b-bb73-3a170576ec58 nodeName:}" failed. No retries permitted until 2026-07-21 21:01:42.244066642 +0000 UTC m=+60.178816286 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/4ebec69c-3c62-4d4b-bb73-3a170576ec58-insights-runtime-extractor-tls") pod "insights-runtime-extractor-wbwbl" (UID: "4ebec69c-3c62-4d4b-bb73-3a170576ec58") : secret "insights-runtime-extractor-tls" not found Jul 21 21:01:41.321898 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:01:41.321872 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-vdkfn_3877fdb0-b49b-444a-8f8c-597d3f935383/dns-node-resolver/0.log" Jul 21 21:01:41.453147 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:01:41.453112 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/76e94cf3-ab49-4698-8ac9-bf3ad1dcdf3a-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-76cf8966f8-7px4w\" (UID: \"76e94cf3-ab49-4698-8ac9-bf3ad1dcdf3a\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-7px4w" Jul 21 21:01:41.453289 ip-10-0-140-97 kubenswrapper[2575]: E0721 21:01:41.453245 2575 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 21 21:01:41.453328 ip-10-0-140-97 kubenswrapper[2575]: E0721 21:01:41.453304 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/76e94cf3-ab49-4698-8ac9-bf3ad1dcdf3a-cluster-monitoring-operator-tls podName:76e94cf3-ab49-4698-8ac9-bf3ad1dcdf3a nodeName:}" failed. No retries permitted until 2026-07-21 21:01:49.453290268 +0000 UTC m=+67.388039914 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/76e94cf3-ab49-4698-8ac9-bf3ad1dcdf3a-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-76cf8966f8-7px4w" (UID: "76e94cf3-ab49-4698-8ac9-bf3ad1dcdf3a") : secret "cluster-monitoring-operator-tls" not found Jul 21 21:01:41.553579 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:01:41.553512 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2adb651d-e8aa-4e76-878c-e5cdafafbe5f-metrics-certs\") pod \"router-default-69864f84f6-qzxnd\" (UID: \"2adb651d-e8aa-4e76-878c-e5cdafafbe5f\") " pod="openshift-ingress/router-default-69864f84f6-qzxnd" Jul 21 21:01:41.553721 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:01:41.553653 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2adb651d-e8aa-4e76-878c-e5cdafafbe5f-service-ca-bundle\") pod \"router-default-69864f84f6-qzxnd\" (UID: \"2adb651d-e8aa-4e76-878c-e5cdafafbe5f\") " pod="openshift-ingress/router-default-69864f84f6-qzxnd" Jul 21 21:01:41.553721 ip-10-0-140-97 kubenswrapper[2575]: E0721 21:01:41.553661 2575 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 21 21:01:41.553851 ip-10-0-140-97 kubenswrapper[2575]: E0721 21:01:41.553723 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2adb651d-e8aa-4e76-878c-e5cdafafbe5f-metrics-certs podName:2adb651d-e8aa-4e76-878c-e5cdafafbe5f nodeName:}" failed. No retries permitted until 2026-07-21 21:01:49.553705741 +0000 UTC m=+67.488455403 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2adb651d-e8aa-4e76-878c-e5cdafafbe5f-metrics-certs") pod "router-default-69864f84f6-qzxnd" (UID: "2adb651d-e8aa-4e76-878c-e5cdafafbe5f") : secret "router-metrics-certs-default" not found Jul 21 21:01:41.553851 ip-10-0-140-97 kubenswrapper[2575]: E0721 21:01:41.553743 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/2adb651d-e8aa-4e76-878c-e5cdafafbe5f-service-ca-bundle podName:2adb651d-e8aa-4e76-878c-e5cdafafbe5f nodeName:}" failed. No retries permitted until 2026-07-21 21:01:49.553732545 +0000 UTC m=+67.488482189 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/2adb651d-e8aa-4e76-878c-e5cdafafbe5f-service-ca-bundle") pod "router-default-69864f84f6-qzxnd" (UID: "2adb651d-e8aa-4e76-878c-e5cdafafbe5f") : configmap references non-existent config key: service-ca.crt Jul 21 21:01:42.260068 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:01:42.260032 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/4ebec69c-3c62-4d4b-bb73-3a170576ec58-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-wbwbl\" (UID: \"4ebec69c-3c62-4d4b-bb73-3a170576ec58\") " pod="openshift-insights/insights-runtime-extractor-wbwbl" Jul 21 21:01:42.260264 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:01:42.260111 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/4ebec69c-3c62-4d4b-bb73-3a170576ec58-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-wbwbl\" (UID: \"4ebec69c-3c62-4d4b-bb73-3a170576ec58\") " pod="openshift-insights/insights-runtime-extractor-wbwbl" Jul 21 21:01:42.260264 ip-10-0-140-97 kubenswrapper[2575]: E0721 21:01:42.260196 2575 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 21 21:01:42.260264 ip-10-0-140-97 kubenswrapper[2575]: E0721 21:01:42.260210 2575 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 21 21:01:42.260264 ip-10-0-140-97 kubenswrapper[2575]: E0721 21:01:42.260251 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/4ebec69c-3c62-4d4b-bb73-3a170576ec58-kube-rbac-proxy-cm podName:4ebec69c-3c62-4d4b-bb73-3a170576ec58 nodeName:}" failed. No retries permitted until 2026-07-21 21:01:46.260236459 +0000 UTC m=+64.194986128 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/4ebec69c-3c62-4d4b-bb73-3a170576ec58-kube-rbac-proxy-cm") pod "insights-runtime-extractor-wbwbl" (UID: "4ebec69c-3c62-4d4b-bb73-3a170576ec58") : configmap "kube-rbac-proxy" not found Jul 21 21:01:42.260427 ip-10-0-140-97 kubenswrapper[2575]: E0721 21:01:42.260275 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4ebec69c-3c62-4d4b-bb73-3a170576ec58-insights-runtime-extractor-tls podName:4ebec69c-3c62-4d4b-bb73-3a170576ec58 nodeName:}" failed. No retries permitted until 2026-07-21 21:01:46.260258522 +0000 UTC m=+64.195008165 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/4ebec69c-3c62-4d4b-bb73-3a170576ec58-insights-runtime-extractor-tls") pod "insights-runtime-extractor-wbwbl" (UID: "4ebec69c-3c62-4d4b-bb73-3a170576ec58") : secret "insights-runtime-extractor-tls" not found Jul 21 21:01:42.513634 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:01:42.513560 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-jhz69_ce7f9a12-9e5b-4634-83d3-babc1e854a66/node-ca/0.log" Jul 21 21:01:43.313610 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:01:43.313576 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-hfmtx_4f86eef6-207f-4ad3-baa2-98cef60dbe52/serve-healthcheck-canary/0.log" Jul 21 21:01:43.915111 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:01:43.915082 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-7658d9b749-7zc94_ca2c8246-50a5-48fb-be89-a95b401523a7/kube-storage-version-migrator-operator/0.log" Jul 21 21:01:43.920702 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:01:43.920680 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-m794j" Jul 21 21:01:46.294054 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:01:46.294018 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/4ebec69c-3c62-4d4b-bb73-3a170576ec58-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-wbwbl\" (UID: \"4ebec69c-3c62-4d4b-bb73-3a170576ec58\") " pod="openshift-insights/insights-runtime-extractor-wbwbl" Jul 21 21:01:46.294462 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:01:46.294103 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/4ebec69c-3c62-4d4b-bb73-3a170576ec58-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-wbwbl\" (UID: \"4ebec69c-3c62-4d4b-bb73-3a170576ec58\") " pod="openshift-insights/insights-runtime-extractor-wbwbl" Jul 21 21:01:46.294462 ip-10-0-140-97 kubenswrapper[2575]: E0721 21:01:46.294146 2575 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 21 21:01:46.294462 ip-10-0-140-97 kubenswrapper[2575]: E0721 21:01:46.294199 2575 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 21 21:01:46.294462 ip-10-0-140-97 kubenswrapper[2575]: E0721 21:01:46.294212 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/4ebec69c-3c62-4d4b-bb73-3a170576ec58-kube-rbac-proxy-cm podName:4ebec69c-3c62-4d4b-bb73-3a170576ec58 nodeName:}" failed. No retries permitted until 2026-07-21 21:01:54.29419633 +0000 UTC m=+72.228945974 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/4ebec69c-3c62-4d4b-bb73-3a170576ec58-kube-rbac-proxy-cm") pod "insights-runtime-extractor-wbwbl" (UID: "4ebec69c-3c62-4d4b-bb73-3a170576ec58") : configmap "kube-rbac-proxy" not found Jul 21 21:01:46.294462 ip-10-0-140-97 kubenswrapper[2575]: E0721 21:01:46.294240 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4ebec69c-3c62-4d4b-bb73-3a170576ec58-insights-runtime-extractor-tls podName:4ebec69c-3c62-4d4b-bb73-3a170576ec58 nodeName:}" failed. No retries permitted until 2026-07-21 21:01:54.294228151 +0000 UTC m=+72.228977820 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/4ebec69c-3c62-4d4b-bb73-3a170576ec58-insights-runtime-extractor-tls") pod "insights-runtime-extractor-wbwbl" (UID: "4ebec69c-3c62-4d4b-bb73-3a170576ec58") : secret "insights-runtime-extractor-tls" not found Jul 21 21:01:48.209339 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:01:48.209303 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-djhxf\" (UniqueName: \"kubernetes.io/projected/7a92ce6a-922d-4196-819d-c03a22e230d1-kube-api-access-djhxf\") pod \"network-check-target-4nvs6\" (UID: \"7a92ce6a-922d-4196-819d-c03a22e230d1\") " pod="openshift-network-diagnostics/network-check-target-4nvs6" Jul 21 21:01:48.209737 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:01:48.209353 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/67b28cbf-d3f3-4d32-b87f-f1e586e2faca-metrics-certs\") pod \"network-metrics-daemon-6fx5v\" (UID: \"67b28cbf-d3f3-4d32-b87f-f1e586e2faca\") " pod="openshift-multus/network-metrics-daemon-6fx5v" Jul 21 21:01:48.211707 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:01:48.211687 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 21 21:01:48.211782 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:01:48.211726 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 21 21:01:48.219570 ip-10-0-140-97 kubenswrapper[2575]: E0721 21:01:48.219548 2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Jul 21 21:01:48.219660 ip-10-0-140-97 kubenswrapper[2575]: E0721 21:01:48.219616 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/67b28cbf-d3f3-4d32-b87f-f1e586e2faca-metrics-certs podName:67b28cbf-d3f3-4d32-b87f-f1e586e2faca nodeName:}" failed. No retries permitted until 2026-07-21 21:02:52.219595991 +0000 UTC m=+130.154345640 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/67b28cbf-d3f3-4d32-b87f-f1e586e2faca-metrics-certs") pod "network-metrics-daemon-6fx5v" (UID: "67b28cbf-d3f3-4d32-b87f-f1e586e2faca") : secret "metrics-daemon-secret" not found Jul 21 21:01:48.222029 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:01:48.222014 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 21 21:01:48.232044 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:01:48.232026 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-djhxf\" (UniqueName: \"kubernetes.io/projected/7a92ce6a-922d-4196-819d-c03a22e230d1-kube-api-access-djhxf\") pod \"network-check-target-4nvs6\" (UID: \"7a92ce6a-922d-4196-819d-c03a22e230d1\") " pod="openshift-network-diagnostics/network-check-target-4nvs6" Jul 21 21:01:48.310255 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:01:48.310228 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e3dcf4f7-a30b-46f5-9c5d-a77f61d88c51-original-pull-secret\") pod \"global-pull-secret-syncer-5694f\" (UID: \"e3dcf4f7-a30b-46f5-9c5d-a77f61d88c51\") " pod="kube-system/global-pull-secret-syncer-5694f" Jul 21 21:01:48.312572 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:01:48.312557 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 21 21:01:48.322743 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:01:48.322722 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/e3dcf4f7-a30b-46f5-9c5d-a77f61d88c51-original-pull-secret\") pod \"global-pull-secret-syncer-5694f\" (UID: \"e3dcf4f7-a30b-46f5-9c5d-a77f61d88c51\") " pod="kube-system/global-pull-secret-syncer-5694f" Jul 21 21:01:48.350251 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:01:48.350228 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-slzpm\"" Jul 21 21:01:48.358569 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:01:48.358546 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4nvs6" Jul 21 21:01:48.360742 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:01:48.360725 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5694f" Jul 21 21:01:48.484414 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:01:48.484352 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-5694f"] Jul 21 21:01:48.487148 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:01:48.487126 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode3dcf4f7_a30b_46f5_9c5d_a77f61d88c51.slice/crio-5c531c814567821e78218fb4c6a7ec8d300a37b99a5111733153d95422f4035e WatchSource:0}: Error finding container 5c531c814567821e78218fb4c6a7ec8d300a37b99a5111733153d95422f4035e: Status 404 returned error can't find the container with id 5c531c814567821e78218fb4c6a7ec8d300a37b99a5111733153d95422f4035e Jul 21 21:01:48.496725 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:01:48.496700 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-4nvs6"] Jul 21 21:01:48.499203 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:01:48.499179 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7a92ce6a_922d_4196_819d_c03a22e230d1.slice/crio-8370819148df563edcc1a4d82f3e98403eebd8579de25dc0bb6a77d4ba71c31a WatchSource:0}: Error finding container 8370819148df563edcc1a4d82f3e98403eebd8579de25dc0bb6a77d4ba71c31a: Status 404 returned error can't find the container with id 8370819148df563edcc1a4d82f3e98403eebd8579de25dc0bb6a77d4ba71c31a Jul 21 21:01:49.007987 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:01:49.007950 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-4nvs6" event={"ID":"7a92ce6a-922d-4196-819d-c03a22e230d1","Type":"ContainerStarted","Data":"8370819148df563edcc1a4d82f3e98403eebd8579de25dc0bb6a77d4ba71c31a"} Jul 21 21:01:49.009348 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:01:49.009315 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-5694f" event={"ID":"e3dcf4f7-a30b-46f5-9c5d-a77f61d88c51","Type":"ContainerStarted","Data":"5c531c814567821e78218fb4c6a7ec8d300a37b99a5111733153d95422f4035e"} Jul 21 21:01:49.418325 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:01:49.418278 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e057811f-88be-47b7-9d41-4815de59f0df-metrics-tls\") pod \"dns-default-bfws8\" (UID: \"e057811f-88be-47b7-9d41-4815de59f0df\") " pod="openshift-dns/dns-default-bfws8" Jul 21 21:01:49.421129 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:01:49.421069 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e057811f-88be-47b7-9d41-4815de59f0df-metrics-tls\") pod \"dns-default-bfws8\" (UID: \"e057811f-88be-47b7-9d41-4815de59f0df\") " pod="openshift-dns/dns-default-bfws8" Jul 21 21:01:49.518794 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:01:49.518763 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/76e94cf3-ab49-4698-8ac9-bf3ad1dcdf3a-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-76cf8966f8-7px4w\" (UID: \"76e94cf3-ab49-4698-8ac9-bf3ad1dcdf3a\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-7px4w" Jul 21 21:01:49.518965 ip-10-0-140-97 kubenswrapper[2575]: E0721 21:01:49.518932 2575 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 21 21:01:49.519035 ip-10-0-140-97 kubenswrapper[2575]: E0721 21:01:49.519007 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/76e94cf3-ab49-4698-8ac9-bf3ad1dcdf3a-cluster-monitoring-operator-tls podName:76e94cf3-ab49-4698-8ac9-bf3ad1dcdf3a nodeName:}" failed. No retries permitted until 2026-07-21 21:02:05.518984249 +0000 UTC m=+83.453733902 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/76e94cf3-ab49-4698-8ac9-bf3ad1dcdf3a-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-76cf8966f8-7px4w" (UID: "76e94cf3-ab49-4698-8ac9-bf3ad1dcdf3a") : secret "cluster-monitoring-operator-tls" not found Jul 21 21:01:49.612854 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:01:49.612825 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-wp8nm\"" Jul 21 21:01:49.620232 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:01:49.620198 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2adb651d-e8aa-4e76-878c-e5cdafafbe5f-service-ca-bundle\") pod \"router-default-69864f84f6-qzxnd\" (UID: \"2adb651d-e8aa-4e76-878c-e5cdafafbe5f\") " pod="openshift-ingress/router-default-69864f84f6-qzxnd" Jul 21 21:01:49.620375 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:01:49.620261 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2adb651d-e8aa-4e76-878c-e5cdafafbe5f-metrics-certs\") pod \"router-default-69864f84f6-qzxnd\" (UID: \"2adb651d-e8aa-4e76-878c-e5cdafafbe5f\") " pod="openshift-ingress/router-default-69864f84f6-qzxnd" Jul 21 21:01:49.620887 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:01:49.620502 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-bfws8" Jul 21 21:01:49.620887 ip-10-0-140-97 kubenswrapper[2575]: E0721 21:01:49.620580 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/2adb651d-e8aa-4e76-878c-e5cdafafbe5f-service-ca-bundle podName:2adb651d-e8aa-4e76-878c-e5cdafafbe5f nodeName:}" failed. No retries permitted until 2026-07-21 21:02:05.620556288 +0000 UTC m=+83.555305947 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/2adb651d-e8aa-4e76-878c-e5cdafafbe5f-service-ca-bundle") pod "router-default-69864f84f6-qzxnd" (UID: "2adb651d-e8aa-4e76-878c-e5cdafafbe5f") : configmap references non-existent config key: service-ca.crt Jul 21 21:01:49.623044 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:01:49.623021 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2adb651d-e8aa-4e76-878c-e5cdafafbe5f-metrics-certs\") pod \"router-default-69864f84f6-qzxnd\" (UID: \"2adb651d-e8aa-4e76-878c-e5cdafafbe5f\") " pod="openshift-ingress/router-default-69864f84f6-qzxnd" Jul 21 21:01:49.779883 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:01:49.779852 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-bfws8"] Jul 21 21:01:49.783093 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:01:49.783050 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode057811f_88be_47b7_9d41_4815de59f0df.slice/crio-1d68afc4945c1dcb80aba5913fb9b11b2fcc72a99ca08e161953a1c4381ee0b2 WatchSource:0}: Error finding container 1d68afc4945c1dcb80aba5913fb9b11b2fcc72a99ca08e161953a1c4381ee0b2: Status 404 returned error can't find the container with id 1d68afc4945c1dcb80aba5913fb9b11b2fcc72a99ca08e161953a1c4381ee0b2 Jul 21 21:01:50.013369 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:01:50.013289 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-bfws8" event={"ID":"e057811f-88be-47b7-9d41-4815de59f0df","Type":"ContainerStarted","Data":"1d68afc4945c1dcb80aba5913fb9b11b2fcc72a99ca08e161953a1c4381ee0b2"} Jul 21 21:01:54.027688 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:01:54.027599 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-5694f" event={"ID":"e3dcf4f7-a30b-46f5-9c5d-a77f61d88c51","Type":"ContainerStarted","Data":"e0d8432ad0606c8a96753d6115c9986e4c7a046586ca559cd97a4578a6d41615"} Jul 21 21:01:54.029239 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:01:54.029212 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-4nvs6" event={"ID":"7a92ce6a-922d-4196-819d-c03a22e230d1","Type":"ContainerStarted","Data":"23a636b421ade4167c25cff578829238f3bf7d1031cce06bc351b5b28241a2e7"} Jul 21 21:01:54.029363 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:01:54.029282 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-4nvs6" Jul 21 21:01:54.030874 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:01:54.030850 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-bfws8" event={"ID":"e057811f-88be-47b7-9d41-4815de59f0df","Type":"ContainerStarted","Data":"6b3e87969941f3b711d087e073b22e3cd1488cefec3eb41cbc007e136c64c3ec"} Jul 21 21:01:54.031070 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:01:54.030889 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-bfws8" event={"ID":"e057811f-88be-47b7-9d41-4815de59f0df","Type":"ContainerStarted","Data":"c4ce4d7c6dda3e623c4897845e04e4f41d39583237944743e376260763bbde44"} Jul 21 21:01:54.031070 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:01:54.030997 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-bfws8" Jul 21 21:01:54.042618 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:01:54.042576 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-5694f" podStartSLOduration=66.739798632 podStartE2EDuration="1m12.042564253s" totalImagesPullingTime="5.302765621s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-21 21:00:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 21:01:54.04207539 +0000 UTC m=+71.976825058" watchObservedRunningTime="2026-07-21 21:01:54.042564253 +0000 UTC m=+71.977313919" Jul 21 21:01:54.058070 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:01:54.058031 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-4nvs6" podStartSLOduration=66.785117924 podStartE2EDuration="1m12.058018965s" totalImagesPullingTime="5.272901041s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 21:00:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 21:01:54.057352597 +0000 UTC m=+71.992102254" watchObservedRunningTime="2026-07-21 21:01:54.058018965 +0000 UTC m=+71.992768630" Jul 21 21:01:54.078202 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:01:54.078151 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-bfws8" podStartSLOduration=33.087343014 podStartE2EDuration="37.078139944s" totalImagesPullingTime="3.99079693s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 21:01:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 21:01:54.077318975 +0000 UTC m=+72.012068642" watchObservedRunningTime="2026-07-21 21:01:54.078139944 +0000 UTC m=+72.012889588" Jul 21 21:01:54.365408 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:01:54.365381 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/4ebec69c-3c62-4d4b-bb73-3a170576ec58-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-wbwbl\" (UID: \"4ebec69c-3c62-4d4b-bb73-3a170576ec58\") " pod="openshift-insights/insights-runtime-extractor-wbwbl" Jul 21 21:01:54.365588 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:01:54.365432 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/4ebec69c-3c62-4d4b-bb73-3a170576ec58-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-wbwbl\" (UID: \"4ebec69c-3c62-4d4b-bb73-3a170576ec58\") " pod="openshift-insights/insights-runtime-extractor-wbwbl" Jul 21 21:01:54.365919 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:01:54.365900 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/4ebec69c-3c62-4d4b-bb73-3a170576ec58-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-wbwbl\" (UID: \"4ebec69c-3c62-4d4b-bb73-3a170576ec58\") " pod="openshift-insights/insights-runtime-extractor-wbwbl" Jul 21 21:01:54.367633 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:01:54.367612 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/4ebec69c-3c62-4d4b-bb73-3a170576ec58-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-wbwbl\" (UID: \"4ebec69c-3c62-4d4b-bb73-3a170576ec58\") " pod="openshift-insights/insights-runtime-extractor-wbwbl" Jul 21 21:01:54.454372 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:01:54.454339 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-vj8lm\"" Jul 21 21:01:54.462494 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:01:54.462472 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-wbwbl" Jul 21 21:01:54.582775 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:01:54.582743 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-wbwbl"] Jul 21 21:01:54.585710 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:01:54.585685 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4ebec69c_3c62_4d4b_bb73_3a170576ec58.slice/crio-6fb9b0003a54096b5ccc4e6be74cd42e0b2cd2a781b3705b98dba9785086b21a WatchSource:0}: Error finding container 6fb9b0003a54096b5ccc4e6be74cd42e0b2cd2a781b3705b98dba9785086b21a: Status 404 returned error can't find the container with id 6fb9b0003a54096b5ccc4e6be74cd42e0b2cd2a781b3705b98dba9785086b21a Jul 21 21:01:55.038194 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:01:55.038156 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-wbwbl" event={"ID":"4ebec69c-3c62-4d4b-bb73-3a170576ec58","Type":"ContainerStarted","Data":"6fb9b0003a54096b5ccc4e6be74cd42e0b2cd2a781b3705b98dba9785086b21a"} Jul 21 21:02:00.058587 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:02:00.058548 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-wbwbl" event={"ID":"4ebec69c-3c62-4d4b-bb73-3a170576ec58","Type":"ContainerStarted","Data":"63b7fff88de1627e85d2a09cef0318091c9bee2c6ad5a09058459ffa6433b7f0"} Jul 21 21:02:03.077094 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:02:03.077011 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-wbwbl" event={"ID":"4ebec69c-3c62-4d4b-bb73-3a170576ec58","Type":"ContainerStarted","Data":"301118eb00cf889cec5a1c316db2fb3afa5a780c77247de20d8b3478b195100e"} Jul 21 21:02:03.145957 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:02:03.145927 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-7bc7489cb5-6jvh2"] Jul 21 21:02:03.147939 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:02:03.147923 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-7bc7489cb5-6jvh2" Jul 21 21:02:03.152139 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:02:03.152113 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\"" Jul 21 21:02:03.152139 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:02:03.152113 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\"" Jul 21 21:02:03.152392 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:02:03.152152 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-dbwgv\"" Jul 21 21:02:03.153034 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:02:03.153016 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-c696b7d6d-hl7hc"] Jul 21 21:02:03.154907 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:02:03.154891 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-c696b7d6d-hl7hc" Jul 21 21:02:03.157740 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:02:03.157550 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\"" Jul 21 21:02:03.157740 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:02:03.157575 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\"" Jul 21 21:02:03.157954 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:02:03.157940 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\"" Jul 21 21:02:03.157996 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:02:03.157949 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-hub-kubeconfig\"" Jul 21 21:02:03.161092 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:02:03.161055 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-dockercfg-nx8l4\"" Jul 21 21:02:03.162703 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:02:03.162678 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-7bc7489cb5-6jvh2"] Jul 21 21:02:03.170265 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:02:03.170244 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-c696b7d6d-hl7hc"] Jul 21 21:02:03.236341 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:02:03.236312 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/8a88c66c-8169-44d8-b719-5369203696ef-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-c696b7d6d-hl7hc\" (UID: \"8a88c66c-8169-44d8-b719-5369203696ef\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-c696b7d6d-hl7hc" Jul 21 21:02:03.236488 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:02:03.236355 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/c981da1f-67fe-49df-9230-91e4215eef08-networking-console-plugin-cert\") pod \"networking-console-plugin-7bc7489cb5-6jvh2\" (UID: \"c981da1f-67fe-49df-9230-91e4215eef08\") " pod="openshift-network-console/networking-console-plugin-7bc7489cb5-6jvh2" Jul 21 21:02:03.236488 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:02:03.236374 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zfdcp\" (UniqueName: \"kubernetes.io/projected/8a88c66c-8169-44d8-b719-5369203696ef-kube-api-access-zfdcp\") pod \"managed-serviceaccount-addon-agent-c696b7d6d-hl7hc\" (UID: \"8a88c66c-8169-44d8-b719-5369203696ef\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-c696b7d6d-hl7hc" Jul 21 21:02:03.236488 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:02:03.236455 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/c981da1f-67fe-49df-9230-91e4215eef08-nginx-conf\") pod \"networking-console-plugin-7bc7489cb5-6jvh2\" (UID: \"c981da1f-67fe-49df-9230-91e4215eef08\") " pod="openshift-network-console/networking-console-plugin-7bc7489cb5-6jvh2" Jul 21 21:02:03.253598 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:02:03.253567 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-5bc67dcd46-gqslk"] Jul 21 21:02:03.255807 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:02:03.255786 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-5bc67dcd46-gqslk" Jul 21 21:02:03.259468 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:02:03.259396 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Jul 21 21:02:03.259725 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:02:03.259708 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Jul 21 21:02:03.259956 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:02:03.259938 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"default-dockercfg-ll6qq\"" Jul 21 21:02:03.275383 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:02:03.275360 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-5bc67dcd46-gqslk"] Jul 21 21:02:03.337669 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:02:03.337589 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/c981da1f-67fe-49df-9230-91e4215eef08-networking-console-plugin-cert\") pod \"networking-console-plugin-7bc7489cb5-6jvh2\" (UID: \"c981da1f-67fe-49df-9230-91e4215eef08\") " pod="openshift-network-console/networking-console-plugin-7bc7489cb5-6jvh2" Jul 21 21:02:03.337669 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:02:03.337624 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zfdcp\" (UniqueName: \"kubernetes.io/projected/8a88c66c-8169-44d8-b719-5369203696ef-kube-api-access-zfdcp\") pod \"managed-serviceaccount-addon-agent-c696b7d6d-hl7hc\" (UID: \"8a88c66c-8169-44d8-b719-5369203696ef\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-c696b7d6d-hl7hc" Jul 21 21:02:03.337928 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:02:03.337680 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/c981da1f-67fe-49df-9230-91e4215eef08-nginx-conf\") pod \"networking-console-plugin-7bc7489cb5-6jvh2\" (UID: \"c981da1f-67fe-49df-9230-91e4215eef08\") " pod="openshift-network-console/networking-console-plugin-7bc7489cb5-6jvh2" Jul 21 21:02:03.337928 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:02:03.337734 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/8a88c66c-8169-44d8-b719-5369203696ef-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-c696b7d6d-hl7hc\" (UID: \"8a88c66c-8169-44d8-b719-5369203696ef\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-c696b7d6d-hl7hc" Jul 21 21:02:03.338046 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:02:03.337952 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-74tvm\" (UniqueName: \"kubernetes.io/projected/e5751a54-5d9c-4f26-9007-74c141b8ace9-kube-api-access-74tvm\") pod \"downloads-5bc67dcd46-gqslk\" (UID: \"e5751a54-5d9c-4f26-9007-74c141b8ace9\") " pod="openshift-console/downloads-5bc67dcd46-gqslk" Jul 21 21:02:03.338503 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:02:03.338477 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/c981da1f-67fe-49df-9230-91e4215eef08-nginx-conf\") pod \"networking-console-plugin-7bc7489cb5-6jvh2\" (UID: \"c981da1f-67fe-49df-9230-91e4215eef08\") " pod="openshift-network-console/networking-console-plugin-7bc7489cb5-6jvh2" Jul 21 21:02:03.340490 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:02:03.340461 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/8a88c66c-8169-44d8-b719-5369203696ef-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-c696b7d6d-hl7hc\" (UID: \"8a88c66c-8169-44d8-b719-5369203696ef\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-c696b7d6d-hl7hc" Jul 21 21:02:03.340611 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:02:03.340591 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/c981da1f-67fe-49df-9230-91e4215eef08-networking-console-plugin-cert\") pod \"networking-console-plugin-7bc7489cb5-6jvh2\" (UID: \"c981da1f-67fe-49df-9230-91e4215eef08\") " pod="openshift-network-console/networking-console-plugin-7bc7489cb5-6jvh2" Jul 21 21:02:03.346766 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:02:03.346733 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zfdcp\" (UniqueName: \"kubernetes.io/projected/8a88c66c-8169-44d8-b719-5369203696ef-kube-api-access-zfdcp\") pod \"managed-serviceaccount-addon-agent-c696b7d6d-hl7hc\" (UID: \"8a88c66c-8169-44d8-b719-5369203696ef\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-c696b7d6d-hl7hc" Jul 21 21:02:03.439401 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:02:03.439364 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-74tvm\" (UniqueName: \"kubernetes.io/projected/e5751a54-5d9c-4f26-9007-74c141b8ace9-kube-api-access-74tvm\") pod \"downloads-5bc67dcd46-gqslk\" (UID: \"e5751a54-5d9c-4f26-9007-74c141b8ace9\") " pod="openshift-console/downloads-5bc67dcd46-gqslk" Jul 21 21:02:03.450641 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:02:03.450603 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-74tvm\" (UniqueName: \"kubernetes.io/projected/e5751a54-5d9c-4f26-9007-74c141b8ace9-kube-api-access-74tvm\") pod \"downloads-5bc67dcd46-gqslk\" (UID: \"e5751a54-5d9c-4f26-9007-74c141b8ace9\") " pod="openshift-console/downloads-5bc67dcd46-gqslk" Jul 21 21:02:03.457272 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:02:03.457250 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-7bc7489cb5-6jvh2" Jul 21 21:02:03.471807 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:02:03.471788 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-c696b7d6d-hl7hc" Jul 21 21:02:03.567493 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:02:03.567219 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-5bc67dcd46-gqslk" Jul 21 21:02:03.617742 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:02:03.617677 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-7bc7489cb5-6jvh2"] Jul 21 21:02:03.625160 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:02:03.625125 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc981da1f_67fe_49df_9230_91e4215eef08.slice/crio-57cde14c61a2d4366ed847c0bd499bdaf028d8396603197a0c6ea990f5403f38 WatchSource:0}: Error finding container 57cde14c61a2d4366ed847c0bd499bdaf028d8396603197a0c6ea990f5403f38: Status 404 returned error can't find the container with id 57cde14c61a2d4366ed847c0bd499bdaf028d8396603197a0c6ea990f5403f38 Jul 21 21:02:03.631073 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:02:03.631045 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-c696b7d6d-hl7hc"] Jul 21 21:02:03.636023 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:02:03.635997 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8a88c66c_8169_44d8_b719_5369203696ef.slice/crio-013ecca16cd687dacab5955128dec54a224ee4cd2cbd46ec486d1432e17b70ac WatchSource:0}: Error finding container 013ecca16cd687dacab5955128dec54a224ee4cd2cbd46ec486d1432e17b70ac: Status 404 returned error can't find the container with id 013ecca16cd687dacab5955128dec54a224ee4cd2cbd46ec486d1432e17b70ac Jul 21 21:02:03.719565 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:02:03.719518 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-5bc67dcd46-gqslk"] Jul 21 21:02:03.723168 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:02:03.723136 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode5751a54_5d9c_4f26_9007_74c141b8ace9.slice/crio-7026964c9a1e89682952342a80aee11d2159f2d32835e2609bb93c0fd104d482 WatchSource:0}: Error finding container 7026964c9a1e89682952342a80aee11d2159f2d32835e2609bb93c0fd104d482: Status 404 returned error can't find the container with id 7026964c9a1e89682952342a80aee11d2159f2d32835e2609bb93c0fd104d482 Jul 21 21:02:04.040642 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:02:04.040567 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-bfws8" Jul 21 21:02:04.082670 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:02:04.082637 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-5bc67dcd46-gqslk" event={"ID":"e5751a54-5d9c-4f26-9007-74c141b8ace9","Type":"ContainerStarted","Data":"7026964c9a1e89682952342a80aee11d2159f2d32835e2609bb93c0fd104d482"} Jul 21 21:02:04.084086 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:02:04.084036 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-7bc7489cb5-6jvh2" event={"ID":"c981da1f-67fe-49df-9230-91e4215eef08","Type":"ContainerStarted","Data":"57cde14c61a2d4366ed847c0bd499bdaf028d8396603197a0c6ea990f5403f38"} Jul 21 21:02:04.085469 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:02:04.085421 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-c696b7d6d-hl7hc" event={"ID":"8a88c66c-8169-44d8-b719-5369203696ef","Type":"ContainerStarted","Data":"013ecca16cd687dacab5955128dec54a224ee4cd2cbd46ec486d1432e17b70ac"} Jul 21 21:02:05.558057 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:02:05.558008 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/76e94cf3-ab49-4698-8ac9-bf3ad1dcdf3a-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-76cf8966f8-7px4w\" (UID: \"76e94cf3-ab49-4698-8ac9-bf3ad1dcdf3a\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-7px4w" Jul 21 21:02:05.560748 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:02:05.560714 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/76e94cf3-ab49-4698-8ac9-bf3ad1dcdf3a-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-76cf8966f8-7px4w\" (UID: \"76e94cf3-ab49-4698-8ac9-bf3ad1dcdf3a\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-7px4w" Jul 21 21:02:05.658931 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:02:05.658888 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2adb651d-e8aa-4e76-878c-e5cdafafbe5f-service-ca-bundle\") pod \"router-default-69864f84f6-qzxnd\" (UID: \"2adb651d-e8aa-4e76-878c-e5cdafafbe5f\") " pod="openshift-ingress/router-default-69864f84f6-qzxnd" Jul 21 21:02:05.659607 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:02:05.659583 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2adb651d-e8aa-4e76-878c-e5cdafafbe5f-service-ca-bundle\") pod \"router-default-69864f84f6-qzxnd\" (UID: \"2adb651d-e8aa-4e76-878c-e5cdafafbe5f\") " pod="openshift-ingress/router-default-69864f84f6-qzxnd" Jul 21 21:02:05.669458 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:02:05.669433 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-795s6\"" Jul 21 21:02:05.677784 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:02:05.677760 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-69864f84f6-qzxnd" Jul 21 21:02:05.852855 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:02:05.852777 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-tgt79\"" Jul 21 21:02:05.861072 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:02:05.861045 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-7px4w" Jul 21 21:02:06.824029 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:02:06.824000 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-76cf8966f8-7px4w"] Jul 21 21:02:06.841360 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:02:06.841284 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-69864f84f6-qzxnd"] Jul 21 21:02:06.845585 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:02:06.845553 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2adb651d_e8aa_4e76_878c_e5cdafafbe5f.slice/crio-025894f6bb24968f723f5deae91b360912270b2d4486231dae3cc23141e0073b WatchSource:0}: Error finding container 025894f6bb24968f723f5deae91b360912270b2d4486231dae3cc23141e0073b: Status 404 returned error can't find the container with id 025894f6bb24968f723f5deae91b360912270b2d4486231dae3cc23141e0073b Jul 21 21:02:07.102235 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:02:07.101409 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-69864f84f6-qzxnd" event={"ID":"2adb651d-e8aa-4e76-878c-e5cdafafbe5f","Type":"ContainerStarted","Data":"92528a410cd7cad6888b04630cf5fcfecf80e63daae7791ae8779e46cb64aab7"} Jul 21 21:02:07.102235 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:02:07.101485 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-69864f84f6-qzxnd" event={"ID":"2adb651d-e8aa-4e76-878c-e5cdafafbe5f","Type":"ContainerStarted","Data":"025894f6bb24968f723f5deae91b360912270b2d4486231dae3cc23141e0073b"} Jul 21 21:02:07.105452 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:02:07.105421 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-wbwbl" event={"ID":"4ebec69c-3c62-4d4b-bb73-3a170576ec58","Type":"ContainerStarted","Data":"d2b7ff4a0e7e8df321b4da325138d2ac0dea725c1ffea9fbd57bae3cfb2e8e1c"} Jul 21 21:02:07.107061 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:02:07.107032 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-7px4w" event={"ID":"76e94cf3-ab49-4698-8ac9-bf3ad1dcdf3a","Type":"ContainerStarted","Data":"3f07bbdfd2d82abb523756c1f20ccf9ebc0da52c8a1fb08f177af6f5a7687789"} Jul 21 21:02:07.109770 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:02:07.109744 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-7bc7489cb5-6jvh2" event={"ID":"c981da1f-67fe-49df-9230-91e4215eef08","Type":"ContainerStarted","Data":"40df0033457287b2e8685ab735a0060802ce22b16dd7366bf1a48e9586328c1a"} Jul 21 21:02:07.121678 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:02:07.121628 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/router-default-69864f84f6-qzxnd" podStartSLOduration=34.121608974 podStartE2EDuration="34.121608974s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 21:01:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 21:02:07.120709084 +0000 UTC m=+85.055458749" watchObservedRunningTime="2026-07-21 21:02:07.121608974 +0000 UTC m=+85.056358644" Jul 21 21:02:07.142920 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:02:07.142841 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-wbwbl" podStartSLOduration=17.171723061 podStartE2EDuration="29.142823312s" totalImagesPullingTime="11.971100251s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-21 21:01:38 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 21:02:07.142248084 +0000 UTC m=+85.076997785" watchObservedRunningTime="2026-07-21 21:02:07.142823312 +0000 UTC m=+85.077572978" Jul 21 21:02:07.163559 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:02:07.163498 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-console/networking-console-plugin-7bc7489cb5-6jvh2" podStartSLOduration=3.038471912 podStartE2EDuration="4.163481127s" totalImagesPullingTime="1.125009215s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 21:02:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 21:02:07.163403722 +0000 UTC m=+85.098153423" watchObservedRunningTime="2026-07-21 21:02:07.163481127 +0000 UTC m=+85.098230794" Jul 21 21:02:07.678752 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:02:07.678728 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-69864f84f6-qzxnd" Jul 21 21:02:07.681614 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:02:07.681550 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-69864f84f6-qzxnd" Jul 21 21:02:07.771162 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:02:07.770053 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-6c4d67b688-6m6c6"] Jul 21 21:02:07.772581 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:02:07.772559 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6c4d67b688-6m6c6" Jul 21 21:02:07.775377 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:02:07.775345 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Jul 21 21:02:07.775890 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:02:07.775714 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Jul 21 21:02:07.775890 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:02:07.775781 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Jul 21 21:02:07.775890 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:02:07.775859 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Jul 21 21:02:07.776087 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:02:07.775858 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-fz5rw\"" Jul 21 21:02:07.776087 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:02:07.775952 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Jul 21 21:02:07.785372 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:02:07.785331 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6c4d67b688-6m6c6"] Jul 21 21:02:07.875629 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:02:07.875599 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/311eb160-46a8-4d6e-b448-5e6c0a6b776f-service-ca\") pod \"console-6c4d67b688-6m6c6\" (UID: \"311eb160-46a8-4d6e-b448-5e6c0a6b776f\") " pod="openshift-console/console-6c4d67b688-6m6c6" Jul 21 21:02:07.876060 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:02:07.875656 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/311eb160-46a8-4d6e-b448-5e6c0a6b776f-oauth-serving-cert\") pod \"console-6c4d67b688-6m6c6\" (UID: \"311eb160-46a8-4d6e-b448-5e6c0a6b776f\") " pod="openshift-console/console-6c4d67b688-6m6c6" Jul 21 21:02:07.876060 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:02:07.875675 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/311eb160-46a8-4d6e-b448-5e6c0a6b776f-console-config\") pod \"console-6c4d67b688-6m6c6\" (UID: \"311eb160-46a8-4d6e-b448-5e6c0a6b776f\") " pod="openshift-console/console-6c4d67b688-6m6c6" Jul 21 21:02:07.876060 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:02:07.875695 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/311eb160-46a8-4d6e-b448-5e6c0a6b776f-console-oauth-config\") pod \"console-6c4d67b688-6m6c6\" (UID: \"311eb160-46a8-4d6e-b448-5e6c0a6b776f\") " pod="openshift-console/console-6c4d67b688-6m6c6" Jul 21 21:02:07.876060 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:02:07.875718 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/311eb160-46a8-4d6e-b448-5e6c0a6b776f-console-serving-cert\") pod \"console-6c4d67b688-6m6c6\" (UID: \"311eb160-46a8-4d6e-b448-5e6c0a6b776f\") " pod="openshift-console/console-6c4d67b688-6m6c6" Jul 21 21:02:07.876060 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:02:07.875744 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fxvmk\" (UniqueName: \"kubernetes.io/projected/311eb160-46a8-4d6e-b448-5e6c0a6b776f-kube-api-access-fxvmk\") pod \"console-6c4d67b688-6m6c6\" (UID: \"311eb160-46a8-4d6e-b448-5e6c0a6b776f\") " pod="openshift-console/console-6c4d67b688-6m6c6" Jul 21 21:02:07.976700 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:02:07.976612 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/311eb160-46a8-4d6e-b448-5e6c0a6b776f-oauth-serving-cert\") pod \"console-6c4d67b688-6m6c6\" (UID: \"311eb160-46a8-4d6e-b448-5e6c0a6b776f\") " pod="openshift-console/console-6c4d67b688-6m6c6" Jul 21 21:02:07.976700 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:02:07.976657 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/311eb160-46a8-4d6e-b448-5e6c0a6b776f-console-config\") pod \"console-6c4d67b688-6m6c6\" (UID: \"311eb160-46a8-4d6e-b448-5e6c0a6b776f\") " pod="openshift-console/console-6c4d67b688-6m6c6" Jul 21 21:02:07.976700 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:02:07.976683 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/311eb160-46a8-4d6e-b448-5e6c0a6b776f-console-oauth-config\") pod \"console-6c4d67b688-6m6c6\" (UID: \"311eb160-46a8-4d6e-b448-5e6c0a6b776f\") " pod="openshift-console/console-6c4d67b688-6m6c6" Jul 21 21:02:07.976953 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:02:07.976715 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/311eb160-46a8-4d6e-b448-5e6c0a6b776f-console-serving-cert\") pod \"console-6c4d67b688-6m6c6\" (UID: \"311eb160-46a8-4d6e-b448-5e6c0a6b776f\") " pod="openshift-console/console-6c4d67b688-6m6c6" Jul 21 21:02:07.976953 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:02:07.976750 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fxvmk\" (UniqueName: \"kubernetes.io/projected/311eb160-46a8-4d6e-b448-5e6c0a6b776f-kube-api-access-fxvmk\") pod \"console-6c4d67b688-6m6c6\" (UID: \"311eb160-46a8-4d6e-b448-5e6c0a6b776f\") " pod="openshift-console/console-6c4d67b688-6m6c6" Jul 21 21:02:07.976953 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:02:07.976784 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/311eb160-46a8-4d6e-b448-5e6c0a6b776f-service-ca\") pod \"console-6c4d67b688-6m6c6\" (UID: \"311eb160-46a8-4d6e-b448-5e6c0a6b776f\") " pod="openshift-console/console-6c4d67b688-6m6c6" Jul 21 21:02:07.977319 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:02:07.977298 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/311eb160-46a8-4d6e-b448-5e6c0a6b776f-oauth-serving-cert\") pod \"console-6c4d67b688-6m6c6\" (UID: \"311eb160-46a8-4d6e-b448-5e6c0a6b776f\") " pod="openshift-console/console-6c4d67b688-6m6c6" Jul 21 21:02:07.977816 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:02:07.977770 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/311eb160-46a8-4d6e-b448-5e6c0a6b776f-service-ca\") pod \"console-6c4d67b688-6m6c6\" (UID: \"311eb160-46a8-4d6e-b448-5e6c0a6b776f\") " pod="openshift-console/console-6c4d67b688-6m6c6" Jul 21 21:02:07.978183 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:02:07.978157 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/311eb160-46a8-4d6e-b448-5e6c0a6b776f-console-config\") pod \"console-6c4d67b688-6m6c6\" (UID: \"311eb160-46a8-4d6e-b448-5e6c0a6b776f\") " pod="openshift-console/console-6c4d67b688-6m6c6" Jul 21 21:02:07.980870 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:02:07.980825 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/311eb160-46a8-4d6e-b448-5e6c0a6b776f-console-oauth-config\") pod \"console-6c4d67b688-6m6c6\" (UID: \"311eb160-46a8-4d6e-b448-5e6c0a6b776f\") " pod="openshift-console/console-6c4d67b688-6m6c6" Jul 21 21:02:07.983247 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:02:07.983226 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/311eb160-46a8-4d6e-b448-5e6c0a6b776f-console-serving-cert\") pod \"console-6c4d67b688-6m6c6\" (UID: \"311eb160-46a8-4d6e-b448-5e6c0a6b776f\") " pod="openshift-console/console-6c4d67b688-6m6c6" Jul 21 21:02:07.986080 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:02:07.986053 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fxvmk\" (UniqueName: \"kubernetes.io/projected/311eb160-46a8-4d6e-b448-5e6c0a6b776f-kube-api-access-fxvmk\") pod \"console-6c4d67b688-6m6c6\" (UID: \"311eb160-46a8-4d6e-b448-5e6c0a6b776f\") " pod="openshift-console/console-6c4d67b688-6m6c6" Jul 21 21:02:08.092492 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:02:08.092456 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6c4d67b688-6m6c6" Jul 21 21:02:08.114792 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:02:08.114762 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-69864f84f6-qzxnd" Jul 21 21:02:08.115987 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:02:08.115959 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-69864f84f6-qzxnd" Jul 21 21:02:10.123554 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:02:10.123503 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6c4d67b688-6m6c6"] Jul 21 21:02:10.125170 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:02:10.125135 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-7px4w" event={"ID":"76e94cf3-ab49-4698-8ac9-bf3ad1dcdf3a","Type":"ContainerStarted","Data":"ec542dae23fc95f1d0869a98dc9a1ffd715f4dd002eacaff262cdd30d44647cc"} Jul 21 21:02:10.127626 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:02:10.127470 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod311eb160_46a8_4d6e_b448_5e6c0a6b776f.slice/crio-5ece47c0432fcf5226ac4092a7ba4f8509030fbbf0e6e0e8508e6957033852b8 WatchSource:0}: Error finding container 5ece47c0432fcf5226ac4092a7ba4f8509030fbbf0e6e0e8508e6957033852b8: Status 404 returned error can't find the container with id 5ece47c0432fcf5226ac4092a7ba4f8509030fbbf0e6e0e8508e6957033852b8 Jul 21 21:02:10.127745 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:02:10.127623 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-c696b7d6d-hl7hc" event={"ID":"8a88c66c-8169-44d8-b719-5369203696ef","Type":"ContainerStarted","Data":"49d3c020fe11c201d6d04c25b0c2e7556b19c249b6cb57591ee6fb3894cad033"} Jul 21 21:02:10.157670 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:02:10.157616 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-c696b7d6d-hl7hc" podStartSLOduration=0.80727397 podStartE2EDuration="7.157592834s" totalImagesPullingTime="6.350318864s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 21:02:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 21:02:10.156496202 +0000 UTC m=+88.091245868" watchObservedRunningTime="2026-07-21 21:02:10.157592834 +0000 UTC m=+88.092342501" Jul 21 21:02:10.157985 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:02:10.157954 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-7px4w" podStartSLOduration=34.006693704 podStartE2EDuration="37.157943861s" totalImagesPullingTime="3.151250157s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 21:01:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 21:02:10.141154806 +0000 UTC m=+88.075904472" watchObservedRunningTime="2026-07-21 21:02:10.157943861 +0000 UTC m=+88.092693559" Jul 21 21:02:10.542401 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:02:10.542373 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-shswg"] Jul 21 21:02:10.545875 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:02:10.545853 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-shswg" Jul 21 21:02:10.548338 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:02:10.548316 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-6rdbh\"" Jul 21 21:02:10.548338 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:02:10.548334 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\"" Jul 21 21:02:10.554893 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:02:10.554868 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-shswg"] Jul 21 21:02:10.700626 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:02:10.700588 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/c922616d-f0b6-4513-a1d0-9cc5e2611708-tls-certificates\") pod \"prometheus-operator-admission-webhook-85574b5f49-shswg\" (UID: \"c922616d-f0b6-4513-a1d0-9cc5e2611708\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-shswg" Jul 21 21:02:10.801438 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:02:10.801359 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/c922616d-f0b6-4513-a1d0-9cc5e2611708-tls-certificates\") pod \"prometheus-operator-admission-webhook-85574b5f49-shswg\" (UID: \"c922616d-f0b6-4513-a1d0-9cc5e2611708\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-shswg" Jul 21 21:02:10.803785 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:02:10.803752 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/c922616d-f0b6-4513-a1d0-9cc5e2611708-tls-certificates\") pod \"prometheus-operator-admission-webhook-85574b5f49-shswg\" (UID: \"c922616d-f0b6-4513-a1d0-9cc5e2611708\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-shswg" Jul 21 21:02:10.857309 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:02:10.857279 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-shswg" Jul 21 21:02:11.016060 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:02:11.013951 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-shswg"] Jul 21 21:02:11.132624 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:02:11.132572 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-shswg" event={"ID":"c922616d-f0b6-4513-a1d0-9cc5e2611708","Type":"ContainerStarted","Data":"9799658783f3091c7dd6e5d932702137e33d4d7ea35223248b9096d2a4938e5e"} Jul 21 21:02:11.134452 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:02:11.134383 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6c4d67b688-6m6c6" event={"ID":"311eb160-46a8-4d6e-b448-5e6c0a6b776f","Type":"ContainerStarted","Data":"5ece47c0432fcf5226ac4092a7ba4f8509030fbbf0e6e0e8508e6957033852b8"} Jul 21 21:02:14.147794 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:02:14.147569 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6c4d67b688-6m6c6" event={"ID":"311eb160-46a8-4d6e-b448-5e6c0a6b776f","Type":"ContainerStarted","Data":"6ce1f4923c6552fcf392b759d2ee589db5cc9cde39f1203be115464d5dcc0e06"} Jul 21 21:02:14.149420 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:02:14.149395 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-shswg" event={"ID":"c922616d-f0b6-4513-a1d0-9cc5e2611708","Type":"ContainerStarted","Data":"7cd6cf429a9eea437a9a9ac17b301aa2620eb608c2b276b50e713f3c79efc067"} Jul 21 21:02:14.149639 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:02:14.149615 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-shswg" Jul 21 21:02:14.155052 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:02:14.155033 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-shswg" Jul 21 21:02:14.166830 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:02:14.166784 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-6c4d67b688-6m6c6" podStartSLOduration=3.811655188 podStartE2EDuration="7.166767852s" totalImagesPullingTime="3.355112664s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 21:02:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 21:02:14.166358467 +0000 UTC m=+92.101108194" watchObservedRunningTime="2026-07-21 21:02:14.166767852 +0000 UTC m=+92.101517519" Jul 21 21:02:14.183334 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:02:14.183299 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-shswg" podStartSLOduration=1.727024653 podStartE2EDuration="4.183287496s" totalImagesPullingTime="2.456262843s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 21:02:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 21:02:14.181985382 +0000 UTC m=+92.116735050" watchObservedRunningTime="2026-07-21 21:02:14.183287496 +0000 UTC m=+92.118037160" Jul 21 21:02:14.603172 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:02:14.603085 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-5945b5cfd-lfqxs"] Jul 21 21:02:14.607657 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:02:14.607634 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-5945b5cfd-lfqxs" Jul 21 21:02:14.611183 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:02:14.611157 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\"" Jul 21 21:02:14.611300 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:02:14.611183 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Jul 21 21:02:14.611300 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:02:14.611159 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\"" Jul 21 21:02:14.611300 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:02:14.611163 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-zvcgx\"" Jul 21 21:02:14.615311 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:02:14.615290 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-5945b5cfd-lfqxs"] Jul 21 21:02:14.734439 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:02:14.734385 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/705f56be-95fb-478b-90ff-f89ebac1baa7-metrics-client-ca\") pod \"prometheus-operator-5945b5cfd-lfqxs\" (UID: \"705f56be-95fb-478b-90ff-f89ebac1baa7\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-lfqxs" Jul 21 21:02:14.734633 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:02:14.734460 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jxvm5\" (UniqueName: \"kubernetes.io/projected/705f56be-95fb-478b-90ff-f89ebac1baa7-kube-api-access-jxvm5\") pod \"prometheus-operator-5945b5cfd-lfqxs\" (UID: \"705f56be-95fb-478b-90ff-f89ebac1baa7\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-lfqxs" Jul 21 21:02:14.734633 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:02:14.734589 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/705f56be-95fb-478b-90ff-f89ebac1baa7-prometheus-operator-tls\") pod \"prometheus-operator-5945b5cfd-lfqxs\" (UID: \"705f56be-95fb-478b-90ff-f89ebac1baa7\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-lfqxs" Jul 21 21:02:14.734633 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:02:14.734626 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/705f56be-95fb-478b-90ff-f89ebac1baa7-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-5945b5cfd-lfqxs\" (UID: \"705f56be-95fb-478b-90ff-f89ebac1baa7\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-lfqxs" Jul 21 21:02:14.835176 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:02:14.835141 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/705f56be-95fb-478b-90ff-f89ebac1baa7-metrics-client-ca\") pod \"prometheus-operator-5945b5cfd-lfqxs\" (UID: \"705f56be-95fb-478b-90ff-f89ebac1baa7\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-lfqxs" Jul 21 21:02:14.835374 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:02:14.835209 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jxvm5\" (UniqueName: \"kubernetes.io/projected/705f56be-95fb-478b-90ff-f89ebac1baa7-kube-api-access-jxvm5\") pod \"prometheus-operator-5945b5cfd-lfqxs\" (UID: \"705f56be-95fb-478b-90ff-f89ebac1baa7\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-lfqxs" Jul 21 21:02:14.835374 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:02:14.835280 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/705f56be-95fb-478b-90ff-f89ebac1baa7-prometheus-operator-tls\") pod \"prometheus-operator-5945b5cfd-lfqxs\" (UID: \"705f56be-95fb-478b-90ff-f89ebac1baa7\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-lfqxs" Jul 21 21:02:14.835374 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:02:14.835308 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/705f56be-95fb-478b-90ff-f89ebac1baa7-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-5945b5cfd-lfqxs\" (UID: \"705f56be-95fb-478b-90ff-f89ebac1baa7\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-lfqxs" Jul 21 21:02:14.835987 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:02:14.835956 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/705f56be-95fb-478b-90ff-f89ebac1baa7-metrics-client-ca\") pod \"prometheus-operator-5945b5cfd-lfqxs\" (UID: \"705f56be-95fb-478b-90ff-f89ebac1baa7\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-lfqxs" Jul 21 21:02:14.838040 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:02:14.838021 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/705f56be-95fb-478b-90ff-f89ebac1baa7-prometheus-operator-tls\") pod \"prometheus-operator-5945b5cfd-lfqxs\" (UID: \"705f56be-95fb-478b-90ff-f89ebac1baa7\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-lfqxs" Jul 21 21:02:14.838150 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:02:14.838049 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/705f56be-95fb-478b-90ff-f89ebac1baa7-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-5945b5cfd-lfqxs\" (UID: \"705f56be-95fb-478b-90ff-f89ebac1baa7\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-lfqxs" Jul 21 21:02:14.843512 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:02:14.843489 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jxvm5\" (UniqueName: \"kubernetes.io/projected/705f56be-95fb-478b-90ff-f89ebac1baa7-kube-api-access-jxvm5\") pod \"prometheus-operator-5945b5cfd-lfqxs\" (UID: \"705f56be-95fb-478b-90ff-f89ebac1baa7\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-lfqxs" Jul 21 21:02:14.919072 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:02:14.919041 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-5945b5cfd-lfqxs" Jul 21 21:02:18.093137 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:02:18.093104 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-6c4d67b688-6m6c6" Jul 21 21:02:18.093519 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:02:18.093162 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-6c4d67b688-6m6c6" Jul 21 21:02:18.098157 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:02:18.098135 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-6c4d67b688-6m6c6" Jul 21 21:02:18.169486 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:02:18.169450 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-6c4d67b688-6m6c6" Jul 21 21:02:21.125738 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:02:21.125711 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-5945b5cfd-lfqxs"] Jul 21 21:02:21.129408 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:02:21.129382 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod705f56be_95fb_478b_90ff_f89ebac1baa7.slice/crio-2e30605ffdb5b8a3778e0ac17964fa4934effa144b1d8a99375707bcb7187c3a WatchSource:0}: Error finding container 2e30605ffdb5b8a3778e0ac17964fa4934effa144b1d8a99375707bcb7187c3a: Status 404 returned error can't find the container with id 2e30605ffdb5b8a3778e0ac17964fa4934effa144b1d8a99375707bcb7187c3a Jul 21 21:02:21.179093 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:02:21.179065 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-5945b5cfd-lfqxs" event={"ID":"705f56be-95fb-478b-90ff-f89ebac1baa7","Type":"ContainerStarted","Data":"2e30605ffdb5b8a3778e0ac17964fa4934effa144b1d8a99375707bcb7187c3a"} Jul 21 21:02:22.185441 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:02:22.185395 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-5bc67dcd46-gqslk" event={"ID":"e5751a54-5d9c-4f26-9007-74c141b8ace9","Type":"ContainerStarted","Data":"a477b01284dac937ba46a706d47dc1aff56db16b14a345ab05c769554b554b15"} Jul 21 21:02:22.185930 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:02:22.185725 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-5bc67dcd46-gqslk" Jul 21 21:02:22.199951 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:02:22.199926 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-5bc67dcd46-gqslk" Jul 21 21:02:22.210432 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:02:22.210373 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/downloads-5bc67dcd46-gqslk" podStartSLOduration=1.50225656 podStartE2EDuration="19.210352928s" totalImagesPullingTime="17.708096368s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 21:02:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 21:02:22.208918143 +0000 UTC m=+100.143667808" watchObservedRunningTime="2026-07-21 21:02:22.210352928 +0000 UTC m=+100.145102596" Jul 21 21:02:25.040336 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:02:25.040262 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-4nvs6" Jul 21 21:02:25.199710 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:02:25.199669 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-5945b5cfd-lfqxs" event={"ID":"705f56be-95fb-478b-90ff-f89ebac1baa7","Type":"ContainerStarted","Data":"be11ad6f6717e07cb63e7d15a0932e28496778654e02edb054d61285ec4b5d2d"} Jul 21 21:02:25.199874 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:02:25.199734 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-5945b5cfd-lfqxs" event={"ID":"705f56be-95fb-478b-90ff-f89ebac1baa7","Type":"ContainerStarted","Data":"af98ea75f30244dacc1efdb66cabc657dd4b5599ddd82e7e64b5c0eeb53b28f8"} Jul 21 21:02:25.222447 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:02:25.222395 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-5945b5cfd-lfqxs" podStartSLOduration=7.66523089 podStartE2EDuration="11.222380834s" totalImagesPullingTime="3.557149944s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 21:02:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 21:02:25.222159605 +0000 UTC m=+103.156909310" watchObservedRunningTime="2026-07-21 21:02:25.222380834 +0000 UTC m=+103.157130499" Jul 21 21:02:26.113147 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:02:26.113112 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6c4d67b688-6m6c6"] Jul 21 21:02:27.001578 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:02:27.001540 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-56c5594674-229p4"] Jul 21 21:02:27.043688 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:02:27.043652 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-56c5594674-229p4"] Jul 21 21:02:27.043877 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:02:27.043791 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-56c5594674-229p4" Jul 21 21:02:27.055893 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:02:27.055870 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\"" Jul 21 21:02:27.056023 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:02:27.055869 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-xflqb\"" Jul 21 21:02:27.056729 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:02:27.056710 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\"" Jul 21 21:02:27.065851 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:02:27.065831 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\"" Jul 21 21:02:27.074775 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:02:27.074752 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-cc8rp"] Jul 21 21:02:27.094295 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:02:27.094273 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-cc8rp" Jul 21 21:02:27.097084 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:02:27.097061 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Jul 21 21:02:27.097211 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:02:27.097100 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Jul 21 21:02:27.097211 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:02:27.097166 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Jul 21 21:02:27.097471 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:02:27.097450 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-6hrnm\"" Jul 21 21:02:27.137396 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:02:27.137354 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/083847dd-e890-49f1-8873-5da46f1417e3-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-56c5594674-229p4\" (UID: \"083847dd-e890-49f1-8873-5da46f1417e3\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-229p4" Jul 21 21:02:27.137827 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:02:27.137406 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/90e9470b-56e7-4b66-9e86-622407c3b7c1-sys\") pod \"node-exporter-cc8rp\" (UID: \"90e9470b-56e7-4b66-9e86-622407c3b7c1\") " pod="openshift-monitoring/node-exporter-cc8rp" Jul 21 21:02:27.137827 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:02:27.137432 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/90e9470b-56e7-4b66-9e86-622407c3b7c1-node-exporter-tls\") pod \"node-exporter-cc8rp\" (UID: \"90e9470b-56e7-4b66-9e86-622407c3b7c1\") " pod="openshift-monitoring/node-exporter-cc8rp" Jul 21 21:02:27.137827 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:02:27.137461 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2tb7r\" (UniqueName: \"kubernetes.io/projected/083847dd-e890-49f1-8873-5da46f1417e3-kube-api-access-2tb7r\") pod \"kube-state-metrics-56c5594674-229p4\" (UID: \"083847dd-e890-49f1-8873-5da46f1417e3\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-229p4" Jul 21 21:02:27.137827 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:02:27.137493 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/90e9470b-56e7-4b66-9e86-622407c3b7c1-metrics-client-ca\") pod \"node-exporter-cc8rp\" (UID: \"90e9470b-56e7-4b66-9e86-622407c3b7c1\") " pod="openshift-monitoring/node-exporter-cc8rp" Jul 21 21:02:27.137827 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:02:27.137511 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/083847dd-e890-49f1-8873-5da46f1417e3-volume-directive-shadow\") pod \"kube-state-metrics-56c5594674-229p4\" (UID: \"083847dd-e890-49f1-8873-5da46f1417e3\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-229p4" Jul 21 21:02:27.137827 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:02:27.137604 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/90e9470b-56e7-4b66-9e86-622407c3b7c1-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-cc8rp\" (UID: \"90e9470b-56e7-4b66-9e86-622407c3b7c1\") " pod="openshift-monitoring/node-exporter-cc8rp" Jul 21 21:02:27.137827 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:02:27.137642 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/083847dd-e890-49f1-8873-5da46f1417e3-metrics-client-ca\") pod \"kube-state-metrics-56c5594674-229p4\" (UID: \"083847dd-e890-49f1-8873-5da46f1417e3\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-229p4" Jul 21 21:02:27.137827 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:02:27.137668 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/90e9470b-56e7-4b66-9e86-622407c3b7c1-node-exporter-textfile\") pod \"node-exporter-cc8rp\" (UID: \"90e9470b-56e7-4b66-9e86-622407c3b7c1\") " pod="openshift-monitoring/node-exporter-cc8rp" Jul 21 21:02:27.137827 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:02:27.137755 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/90e9470b-56e7-4b66-9e86-622407c3b7c1-node-exporter-wtmp\") pod \"node-exporter-cc8rp\" (UID: \"90e9470b-56e7-4b66-9e86-622407c3b7c1\") " pod="openshift-monitoring/node-exporter-cc8rp" Jul 21 21:02:27.137827 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:02:27.137793 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/90e9470b-56e7-4b66-9e86-622407c3b7c1-node-exporter-accelerators-collector-config\") pod \"node-exporter-cc8rp\" (UID: \"90e9470b-56e7-4b66-9e86-622407c3b7c1\") " pod="openshift-monitoring/node-exporter-cc8rp" Jul 21 21:02:27.137827 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:02:27.137818 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/083847dd-e890-49f1-8873-5da46f1417e3-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-56c5594674-229p4\" (UID: \"083847dd-e890-49f1-8873-5da46f1417e3\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-229p4" Jul 21 21:02:27.138254 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:02:27.137854 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/90e9470b-56e7-4b66-9e86-622407c3b7c1-root\") pod \"node-exporter-cc8rp\" (UID: \"90e9470b-56e7-4b66-9e86-622407c3b7c1\") " pod="openshift-monitoring/node-exporter-cc8rp" Jul 21 21:02:27.138254 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:02:27.137900 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lrrnd\" (UniqueName: \"kubernetes.io/projected/90e9470b-56e7-4b66-9e86-622407c3b7c1-kube-api-access-lrrnd\") pod \"node-exporter-cc8rp\" (UID: \"90e9470b-56e7-4b66-9e86-622407c3b7c1\") " pod="openshift-monitoring/node-exporter-cc8rp" Jul 21 21:02:27.138254 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:02:27.137949 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/083847dd-e890-49f1-8873-5da46f1417e3-kube-state-metrics-tls\") pod \"kube-state-metrics-56c5594674-229p4\" (UID: \"083847dd-e890-49f1-8873-5da46f1417e3\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-229p4" Jul 21 21:02:27.239133 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:02:27.239094 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/90e9470b-56e7-4b66-9e86-622407c3b7c1-node-exporter-tls\") pod \"node-exporter-cc8rp\" (UID: \"90e9470b-56e7-4b66-9e86-622407c3b7c1\") " pod="openshift-monitoring/node-exporter-cc8rp" Jul 21 21:02:27.239133 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:02:27.239139 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2tb7r\" (UniqueName: \"kubernetes.io/projected/083847dd-e890-49f1-8873-5da46f1417e3-kube-api-access-2tb7r\") pod \"kube-state-metrics-56c5594674-229p4\" (UID: \"083847dd-e890-49f1-8873-5da46f1417e3\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-229p4" Jul 21 21:02:27.239380 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:02:27.239172 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/90e9470b-56e7-4b66-9e86-622407c3b7c1-metrics-client-ca\") pod \"node-exporter-cc8rp\" (UID: \"90e9470b-56e7-4b66-9e86-622407c3b7c1\") " pod="openshift-monitoring/node-exporter-cc8rp" Jul 21 21:02:27.239380 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:02:27.239195 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/083847dd-e890-49f1-8873-5da46f1417e3-volume-directive-shadow\") pod \"kube-state-metrics-56c5594674-229p4\" (UID: \"083847dd-e890-49f1-8873-5da46f1417e3\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-229p4" Jul 21 21:02:27.239380 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:02:27.239240 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/90e9470b-56e7-4b66-9e86-622407c3b7c1-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-cc8rp\" (UID: \"90e9470b-56e7-4b66-9e86-622407c3b7c1\") " pod="openshift-monitoring/node-exporter-cc8rp" Jul 21 21:02:27.239380 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:02:27.239273 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/083847dd-e890-49f1-8873-5da46f1417e3-metrics-client-ca\") pod \"kube-state-metrics-56c5594674-229p4\" (UID: \"083847dd-e890-49f1-8873-5da46f1417e3\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-229p4" Jul 21 21:02:27.239380 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:02:27.239304 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/90e9470b-56e7-4b66-9e86-622407c3b7c1-node-exporter-textfile\") pod \"node-exporter-cc8rp\" (UID: \"90e9470b-56e7-4b66-9e86-622407c3b7c1\") " pod="openshift-monitoring/node-exporter-cc8rp" Jul 21 21:02:27.239380 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:02:27.239349 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/90e9470b-56e7-4b66-9e86-622407c3b7c1-node-exporter-wtmp\") pod \"node-exporter-cc8rp\" (UID: \"90e9470b-56e7-4b66-9e86-622407c3b7c1\") " pod="openshift-monitoring/node-exporter-cc8rp" Jul 21 21:02:27.239380 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:02:27.239378 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/90e9470b-56e7-4b66-9e86-622407c3b7c1-node-exporter-accelerators-collector-config\") pod \"node-exporter-cc8rp\" (UID: \"90e9470b-56e7-4b66-9e86-622407c3b7c1\") " pod="openshift-monitoring/node-exporter-cc8rp" Jul 21 21:02:27.239754 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:02:27.239409 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/083847dd-e890-49f1-8873-5da46f1417e3-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-56c5594674-229p4\" (UID: \"083847dd-e890-49f1-8873-5da46f1417e3\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-229p4" Jul 21 21:02:27.239754 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:02:27.239441 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/90e9470b-56e7-4b66-9e86-622407c3b7c1-root\") pod \"node-exporter-cc8rp\" (UID: \"90e9470b-56e7-4b66-9e86-622407c3b7c1\") " pod="openshift-monitoring/node-exporter-cc8rp" Jul 21 21:02:27.239754 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:02:27.239465 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lrrnd\" (UniqueName: \"kubernetes.io/projected/90e9470b-56e7-4b66-9e86-622407c3b7c1-kube-api-access-lrrnd\") pod \"node-exporter-cc8rp\" (UID: \"90e9470b-56e7-4b66-9e86-622407c3b7c1\") " pod="openshift-monitoring/node-exporter-cc8rp" Jul 21 21:02:27.239754 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:02:27.239492 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/083847dd-e890-49f1-8873-5da46f1417e3-kube-state-metrics-tls\") pod \"kube-state-metrics-56c5594674-229p4\" (UID: \"083847dd-e890-49f1-8873-5da46f1417e3\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-229p4" Jul 21 21:02:27.239754 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:02:27.239547 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/083847dd-e890-49f1-8873-5da46f1417e3-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-56c5594674-229p4\" (UID: \"083847dd-e890-49f1-8873-5da46f1417e3\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-229p4" Jul 21 21:02:27.239754 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:02:27.239588 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/90e9470b-56e7-4b66-9e86-622407c3b7c1-sys\") pod \"node-exporter-cc8rp\" (UID: \"90e9470b-56e7-4b66-9e86-622407c3b7c1\") " pod="openshift-monitoring/node-exporter-cc8rp" Jul 21 21:02:27.239754 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:02:27.239678 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/90e9470b-56e7-4b66-9e86-622407c3b7c1-sys\") pod \"node-exporter-cc8rp\" (UID: \"90e9470b-56e7-4b66-9e86-622407c3b7c1\") " pod="openshift-monitoring/node-exporter-cc8rp" Jul 21 21:02:27.240095 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:02:27.239785 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/083847dd-e890-49f1-8873-5da46f1417e3-volume-directive-shadow\") pod \"kube-state-metrics-56c5594674-229p4\" (UID: \"083847dd-e890-49f1-8873-5da46f1417e3\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-229p4" Jul 21 21:02:27.240095 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:02:27.239964 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/90e9470b-56e7-4b66-9e86-622407c3b7c1-metrics-client-ca\") pod \"node-exporter-cc8rp\" (UID: \"90e9470b-56e7-4b66-9e86-622407c3b7c1\") " pod="openshift-monitoring/node-exporter-cc8rp" Jul 21 21:02:27.240095 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:02:27.240071 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/90e9470b-56e7-4b66-9e86-622407c3b7c1-node-exporter-accelerators-collector-config\") pod \"node-exporter-cc8rp\" (UID: \"90e9470b-56e7-4b66-9e86-622407c3b7c1\") " pod="openshift-monitoring/node-exporter-cc8rp" Jul 21 21:02:27.240933 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:02:27.240686 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/083847dd-e890-49f1-8873-5da46f1417e3-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-56c5594674-229p4\" (UID: \"083847dd-e890-49f1-8873-5da46f1417e3\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-229p4" Jul 21 21:02:27.240933 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:02:27.240787 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/90e9470b-56e7-4b66-9e86-622407c3b7c1-root\") pod \"node-exporter-cc8rp\" (UID: \"90e9470b-56e7-4b66-9e86-622407c3b7c1\") " pod="openshift-monitoring/node-exporter-cc8rp" Jul 21 21:02:27.240933 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:02:27.240790 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/90e9470b-56e7-4b66-9e86-622407c3b7c1-node-exporter-textfile\") pod \"node-exporter-cc8rp\" (UID: \"90e9470b-56e7-4b66-9e86-622407c3b7c1\") " pod="openshift-monitoring/node-exporter-cc8rp" Jul 21 21:02:27.240933 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:02:27.240836 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/90e9470b-56e7-4b66-9e86-622407c3b7c1-node-exporter-wtmp\") pod \"node-exporter-cc8rp\" (UID: \"90e9470b-56e7-4b66-9e86-622407c3b7c1\") " pod="openshift-monitoring/node-exporter-cc8rp" Jul 21 21:02:27.241498 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:02:27.241476 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/083847dd-e890-49f1-8873-5da46f1417e3-metrics-client-ca\") pod \"kube-state-metrics-56c5594674-229p4\" (UID: \"083847dd-e890-49f1-8873-5da46f1417e3\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-229p4" Jul 21 21:02:27.243025 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:02:27.242984 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/90e9470b-56e7-4b66-9e86-622407c3b7c1-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-cc8rp\" (UID: \"90e9470b-56e7-4b66-9e86-622407c3b7c1\") " pod="openshift-monitoring/node-exporter-cc8rp" Jul 21 21:02:27.243297 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:02:27.243144 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/083847dd-e890-49f1-8873-5da46f1417e3-kube-state-metrics-tls\") pod \"kube-state-metrics-56c5594674-229p4\" (UID: \"083847dd-e890-49f1-8873-5da46f1417e3\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-229p4" Jul 21 21:02:27.243381 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:02:27.243306 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/083847dd-e890-49f1-8873-5da46f1417e3-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-56c5594674-229p4\" (UID: \"083847dd-e890-49f1-8873-5da46f1417e3\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-229p4" Jul 21 21:02:27.243506 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:02:27.243467 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/90e9470b-56e7-4b66-9e86-622407c3b7c1-node-exporter-tls\") pod \"node-exporter-cc8rp\" (UID: \"90e9470b-56e7-4b66-9e86-622407c3b7c1\") " pod="openshift-monitoring/node-exporter-cc8rp" Jul 21 21:02:27.248845 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:02:27.248810 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2tb7r\" (UniqueName: \"kubernetes.io/projected/083847dd-e890-49f1-8873-5da46f1417e3-kube-api-access-2tb7r\") pod \"kube-state-metrics-56c5594674-229p4\" (UID: \"083847dd-e890-49f1-8873-5da46f1417e3\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-229p4" Jul 21 21:02:27.250169 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:02:27.250145 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lrrnd\" (UniqueName: \"kubernetes.io/projected/90e9470b-56e7-4b66-9e86-622407c3b7c1-kube-api-access-lrrnd\") pod \"node-exporter-cc8rp\" (UID: \"90e9470b-56e7-4b66-9e86-622407c3b7c1\") " pod="openshift-monitoring/node-exporter-cc8rp" Jul 21 21:02:27.355086 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:02:27.354999 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-56c5594674-229p4" Jul 21 21:02:27.411711 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:02:27.411684 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-cc8rp" Jul 21 21:02:27.464918 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:02:27.464866 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod90e9470b_56e7_4b66_9e86_622407c3b7c1.slice/crio-a1ce468f56c3bfea41c1bad182fedfa9be626b9960ecf4a30a91a375ba71161e WatchSource:0}: Error finding container a1ce468f56c3bfea41c1bad182fedfa9be626b9960ecf4a30a91a375ba71161e: Status 404 returned error can't find the container with id a1ce468f56c3bfea41c1bad182fedfa9be626b9960ecf4a30a91a375ba71161e Jul 21 21:02:27.542673 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:02:27.542633 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-56c5594674-229p4"] Jul 21 21:02:27.544855 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:02:27.544826 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod083847dd_e890_49f1_8873_5da46f1417e3.slice/crio-49823f9a1860e9d614432724e11c7b68f70e5c6219c818dde0e47c5c63f9f245 WatchSource:0}: Error finding container 49823f9a1860e9d614432724e11c7b68f70e5c6219c818dde0e47c5c63f9f245: Status 404 returned error can't find the container with id 49823f9a1860e9d614432724e11c7b68f70e5c6219c818dde0e47c5c63f9f245 Jul 21 21:02:28.213520 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:02:28.213478 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-cc8rp" event={"ID":"90e9470b-56e7-4b66-9e86-622407c3b7c1","Type":"ContainerStarted","Data":"a1ce468f56c3bfea41c1bad182fedfa9be626b9960ecf4a30a91a375ba71161e"} Jul 21 21:02:28.215258 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:02:28.215223 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-56c5594674-229p4" event={"ID":"083847dd-e890-49f1-8873-5da46f1417e3","Type":"ContainerStarted","Data":"49823f9a1860e9d614432724e11c7b68f70e5c6219c818dde0e47c5c63f9f245"} Jul 21 21:02:31.229749 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:02:31.229661 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-56c5594674-229p4" event={"ID":"083847dd-e890-49f1-8873-5da46f1417e3","Type":"ContainerStarted","Data":"baa9beb5f63dd1a7b2a817932b430bbd79f9397851a1c0ca28f0ff280279bfa6"} Jul 21 21:02:31.229749 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:02:31.229731 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-56c5594674-229p4" event={"ID":"083847dd-e890-49f1-8873-5da46f1417e3","Type":"ContainerStarted","Data":"4923f22258b9ca431bb1091ebed3440841d6bc63082f2c74e581a14a75931754"} Jul 21 21:02:31.230247 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:02:31.229760 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-56c5594674-229p4" event={"ID":"083847dd-e890-49f1-8873-5da46f1417e3","Type":"ContainerStarted","Data":"bc9b65c0b3139523fa31f01884dbdde0773a67e0478eae57ee3d34788de6aa5b"} Jul 21 21:02:31.231215 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:02:31.231187 2575 generic.go:356] "Generic (PLEG): container finished" podID="90e9470b-56e7-4b66-9e86-622407c3b7c1" containerID="149af17191ea665767670b62cb3e3086183e126aec315da1d21b14048a114ea2" exitCode=0 Jul 21 21:02:31.231302 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:02:31.231238 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-cc8rp" event={"ID":"90e9470b-56e7-4b66-9e86-622407c3b7c1","Type":"ContainerDied","Data":"149af17191ea665767670b62cb3e3086183e126aec315da1d21b14048a114ea2"} Jul 21 21:02:31.249882 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:02:31.249830 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-56c5594674-229p4" podStartSLOduration=2.464273867 podStartE2EDuration="5.249811273s" totalImagesPullingTime="2.785537406s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 21:02:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 21:02:31.248493634 +0000 UTC m=+109.183243337" watchObservedRunningTime="2026-07-21 21:02:31.249811273 +0000 UTC m=+109.184560943" Jul 21 21:02:32.236789 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:02:32.236754 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-cc8rp" event={"ID":"90e9470b-56e7-4b66-9e86-622407c3b7c1","Type":"ContainerStarted","Data":"70d6e6cb12f623b73184e37d9353c39da881159edc947ee5d0bab127382f3784"} Jul 21 21:02:32.237258 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:02:32.236834 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-cc8rp" event={"ID":"90e9470b-56e7-4b66-9e86-622407c3b7c1","Type":"ContainerStarted","Data":"1899d462b755d875d56b24faaf6e44445f9a2333d5482e61afdd8f5b1fe88acf"} Jul 21 21:02:32.258604 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:02:32.258562 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-cc8rp" podStartSLOduration=1.930478861 podStartE2EDuration="5.258544059s" totalImagesPullingTime="3.218936317s" totalInitContainerRuntime="109.128881ms" isStatefulPod=true podCreationTimestamp="2026-07-21 21:02:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 21:02:32.257628443 +0000 UTC m=+110.192378108" watchObservedRunningTime="2026-07-21 21:02:32.258544059 +0000 UTC m=+110.193293724" Jul 21 21:02:33.165920 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:02:33.165884 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 21 21:02:33.181433 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:02:33.181376 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 21 21:02:33.183739 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:02:33.183712 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Jul 21 21:02:33.183922 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:02:33.183893 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 21 21:02:33.184012 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:02:33.183955 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Jul 21 21:02:33.184012 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:02:33.183995 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-86ibs7einh8vo\"" Jul 21 21:02:33.184477 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:02:33.184455 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Jul 21 21:02:33.184697 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:02:33.184563 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Jul 21 21:02:33.184761 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:02:33.184749 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Jul 21 21:02:33.184761 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:02:33.184755 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Jul 21 21:02:33.184860 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:02:33.184827 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Jul 21 21:02:33.184980 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:02:33.184900 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Jul 21 21:02:33.184980 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:02:33.184963 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Jul 21 21:02:33.185317 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:02:33.185164 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Jul 21 21:02:33.185317 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:02:33.185258 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Jul 21 21:02:33.185509 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:02:33.185322 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Jul 21 21:02:33.185509 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:02:33.185406 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-jlstg\"" Jul 21 21:02:33.185509 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:02:33.185456 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Jul 21 21:02:33.185714 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:02:33.185678 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Jul 21 21:02:33.192166 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:02:33.192150 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Jul 21 21:02:33.294732 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:02:33.294694 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/24fd0b69-36fa-4f00-89ab-77b624152a32-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"24fd0b69-36fa-4f00-89ab-77b624152a32\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 21:02:33.294732 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:02:33.294731 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/24fd0b69-36fa-4f00-89ab-77b624152a32-web-config\") pod \"prometheus-k8s-0\" (UID: \"24fd0b69-36fa-4f00-89ab-77b624152a32\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 21:02:33.295204 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:02:33.294816 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/24fd0b69-36fa-4f00-89ab-77b624152a32-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"24fd0b69-36fa-4f00-89ab-77b624152a32\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 21:02:33.295204 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:02:33.294886 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/24fd0b69-36fa-4f00-89ab-77b624152a32-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"24fd0b69-36fa-4f00-89ab-77b624152a32\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 21:02:33.295204 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:02:33.294917 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/24fd0b69-36fa-4f00-89ab-77b624152a32-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"24fd0b69-36fa-4f00-89ab-77b624152a32\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 21:02:33.295204 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:02:33.294947 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/24fd0b69-36fa-4f00-89ab-77b624152a32-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"24fd0b69-36fa-4f00-89ab-77b624152a32\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 21:02:33.295204 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:02:33.294968 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/24fd0b69-36fa-4f00-89ab-77b624152a32-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"24fd0b69-36fa-4f00-89ab-77b624152a32\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 21:02:33.295204 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:02:33.295008 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/24fd0b69-36fa-4f00-89ab-77b624152a32-config\") pod \"prometheus-k8s-0\" (UID: \"24fd0b69-36fa-4f00-89ab-77b624152a32\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 21:02:33.295204 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:02:33.295027 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/24fd0b69-36fa-4f00-89ab-77b624152a32-config-out\") pod \"prometheus-k8s-0\" (UID: \"24fd0b69-36fa-4f00-89ab-77b624152a32\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 21:02:33.295204 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:02:33.295074 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/24fd0b69-36fa-4f00-89ab-77b624152a32-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"24fd0b69-36fa-4f00-89ab-77b624152a32\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 21:02:33.295204 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:02:33.295114 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/24fd0b69-36fa-4f00-89ab-77b624152a32-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"24fd0b69-36fa-4f00-89ab-77b624152a32\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 21:02:33.295204 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:02:33.295161 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/24fd0b69-36fa-4f00-89ab-77b624152a32-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"24fd0b69-36fa-4f00-89ab-77b624152a32\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 21:02:33.295204 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:02:33.295180 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/24fd0b69-36fa-4f00-89ab-77b624152a32-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"24fd0b69-36fa-4f00-89ab-77b624152a32\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 21:02:33.295826 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:02:33.295260 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/24fd0b69-36fa-4f00-89ab-77b624152a32-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"24fd0b69-36fa-4f00-89ab-77b624152a32\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 21:02:33.295826 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:02:33.295301 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/24fd0b69-36fa-4f00-89ab-77b624152a32-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"24fd0b69-36fa-4f00-89ab-77b624152a32\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 21:02:33.295826 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:02:33.295327 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c5fml\" (UniqueName: \"kubernetes.io/projected/24fd0b69-36fa-4f00-89ab-77b624152a32-kube-api-access-c5fml\") pod \"prometheus-k8s-0\" (UID: \"24fd0b69-36fa-4f00-89ab-77b624152a32\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 21:02:33.295826 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:02:33.295352 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/24fd0b69-36fa-4f00-89ab-77b624152a32-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"24fd0b69-36fa-4f00-89ab-77b624152a32\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 21:02:33.295826 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:02:33.295422 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/24fd0b69-36fa-4f00-89ab-77b624152a32-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"24fd0b69-36fa-4f00-89ab-77b624152a32\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 21:02:33.295826 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:02:33.295443 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/24fd0b69-36fa-4f00-89ab-77b624152a32-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"24fd0b69-36fa-4f00-89ab-77b624152a32\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 21:02:33.295826 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:02:33.295491 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/24fd0b69-36fa-4f00-89ab-77b624152a32-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"24fd0b69-36fa-4f00-89ab-77b624152a32\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 21:02:33.396358 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:02:33.396104 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/24fd0b69-36fa-4f00-89ab-77b624152a32-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"24fd0b69-36fa-4f00-89ab-77b624152a32\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 21:02:33.396358 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:02:33.396186 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/24fd0b69-36fa-4f00-89ab-77b624152a32-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"24fd0b69-36fa-4f00-89ab-77b624152a32\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 21:02:33.396358 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:02:33.396213 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/24fd0b69-36fa-4f00-89ab-77b624152a32-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"24fd0b69-36fa-4f00-89ab-77b624152a32\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 21:02:33.396358 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:02:33.396246 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/24fd0b69-36fa-4f00-89ab-77b624152a32-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"24fd0b69-36fa-4f00-89ab-77b624152a32\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 21:02:33.396358 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:02:33.396276 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/24fd0b69-36fa-4f00-89ab-77b624152a32-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"24fd0b69-36fa-4f00-89ab-77b624152a32\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 21:02:33.396358 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:02:33.396305 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/24fd0b69-36fa-4f00-89ab-77b624152a32-web-config\") pod \"prometheus-k8s-0\" (UID: \"24fd0b69-36fa-4f00-89ab-77b624152a32\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 21:02:33.396358 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:02:33.396344 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/24fd0b69-36fa-4f00-89ab-77b624152a32-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"24fd0b69-36fa-4f00-89ab-77b624152a32\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 21:02:33.396886 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:02:33.396393 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/24fd0b69-36fa-4f00-89ab-77b624152a32-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"24fd0b69-36fa-4f00-89ab-77b624152a32\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 21:02:33.396886 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:02:33.396413 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/24fd0b69-36fa-4f00-89ab-77b624152a32-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"24fd0b69-36fa-4f00-89ab-77b624152a32\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 21:02:33.396886 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:02:33.396435 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/24fd0b69-36fa-4f00-89ab-77b624152a32-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"24fd0b69-36fa-4f00-89ab-77b624152a32\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 21:02:33.396886 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:02:33.396456 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/24fd0b69-36fa-4f00-89ab-77b624152a32-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"24fd0b69-36fa-4f00-89ab-77b624152a32\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 21:02:33.396886 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:02:33.396481 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/24fd0b69-36fa-4f00-89ab-77b624152a32-config\") pod \"prometheus-k8s-0\" (UID: \"24fd0b69-36fa-4f00-89ab-77b624152a32\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 21:02:33.396886 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:02:33.396500 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/24fd0b69-36fa-4f00-89ab-77b624152a32-config-out\") pod \"prometheus-k8s-0\" (UID: \"24fd0b69-36fa-4f00-89ab-77b624152a32\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 21:02:33.396886 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:02:33.396541 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/24fd0b69-36fa-4f00-89ab-77b624152a32-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"24fd0b69-36fa-4f00-89ab-77b624152a32\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 21:02:33.396886 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:02:33.396564 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/24fd0b69-36fa-4f00-89ab-77b624152a32-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"24fd0b69-36fa-4f00-89ab-77b624152a32\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 21:02:33.396886 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:02:33.396611 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/24fd0b69-36fa-4f00-89ab-77b624152a32-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"24fd0b69-36fa-4f00-89ab-77b624152a32\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 21:02:33.396886 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:02:33.396637 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/24fd0b69-36fa-4f00-89ab-77b624152a32-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"24fd0b69-36fa-4f00-89ab-77b624152a32\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 21:02:33.396886 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:02:33.396692 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/24fd0b69-36fa-4f00-89ab-77b624152a32-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"24fd0b69-36fa-4f00-89ab-77b624152a32\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 21:02:33.396886 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:02:33.396725 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/24fd0b69-36fa-4f00-89ab-77b624152a32-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"24fd0b69-36fa-4f00-89ab-77b624152a32\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 21:02:33.396886 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:02:33.396749 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c5fml\" (UniqueName: \"kubernetes.io/projected/24fd0b69-36fa-4f00-89ab-77b624152a32-kube-api-access-c5fml\") pod \"prometheus-k8s-0\" (UID: \"24fd0b69-36fa-4f00-89ab-77b624152a32\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 21:02:33.397550 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:02:33.397299 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/24fd0b69-36fa-4f00-89ab-77b624152a32-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"24fd0b69-36fa-4f00-89ab-77b624152a32\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 21:02:33.398491 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:02:33.397666 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/24fd0b69-36fa-4f00-89ab-77b624152a32-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"24fd0b69-36fa-4f00-89ab-77b624152a32\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 21:02:33.398491 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:02:33.398171 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/24fd0b69-36fa-4f00-89ab-77b624152a32-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"24fd0b69-36fa-4f00-89ab-77b624152a32\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 21:02:33.398491 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:02:33.398438 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/24fd0b69-36fa-4f00-89ab-77b624152a32-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"24fd0b69-36fa-4f00-89ab-77b624152a32\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 21:02:33.399465 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:02:33.399439 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/24fd0b69-36fa-4f00-89ab-77b624152a32-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"24fd0b69-36fa-4f00-89ab-77b624152a32\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 21:02:33.400232 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:02:33.399901 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/24fd0b69-36fa-4f00-89ab-77b624152a32-config\") pod \"prometheus-k8s-0\" (UID: \"24fd0b69-36fa-4f00-89ab-77b624152a32\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 21:02:33.400232 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:02:33.399970 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/24fd0b69-36fa-4f00-89ab-77b624152a32-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"24fd0b69-36fa-4f00-89ab-77b624152a32\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 21:02:33.400382 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:02:33.400360 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/24fd0b69-36fa-4f00-89ab-77b624152a32-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"24fd0b69-36fa-4f00-89ab-77b624152a32\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 21:02:33.400968 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:02:33.400937 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/24fd0b69-36fa-4f00-89ab-77b624152a32-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"24fd0b69-36fa-4f00-89ab-77b624152a32\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 21:02:33.401063 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:02:33.401001 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/24fd0b69-36fa-4f00-89ab-77b624152a32-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"24fd0b69-36fa-4f00-89ab-77b624152a32\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 21:02:33.401063 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:02:33.401023 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/24fd0b69-36fa-4f00-89ab-77b624152a32-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"24fd0b69-36fa-4f00-89ab-77b624152a32\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 21:02:33.401331 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:02:33.401302 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/24fd0b69-36fa-4f00-89ab-77b624152a32-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"24fd0b69-36fa-4f00-89ab-77b624152a32\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 21:02:33.401466 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:02:33.401442 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/24fd0b69-36fa-4f00-89ab-77b624152a32-config-out\") pod \"prometheus-k8s-0\" (UID: \"24fd0b69-36fa-4f00-89ab-77b624152a32\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 21:02:33.401728 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:02:33.401705 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/24fd0b69-36fa-4f00-89ab-77b624152a32-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"24fd0b69-36fa-4f00-89ab-77b624152a32\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 21:02:33.402000 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:02:33.401940 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/24fd0b69-36fa-4f00-89ab-77b624152a32-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"24fd0b69-36fa-4f00-89ab-77b624152a32\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 21:02:33.403108 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:02:33.403081 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/24fd0b69-36fa-4f00-89ab-77b624152a32-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"24fd0b69-36fa-4f00-89ab-77b624152a32\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 21:02:33.403369 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:02:33.403350 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/24fd0b69-36fa-4f00-89ab-77b624152a32-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"24fd0b69-36fa-4f00-89ab-77b624152a32\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 21:02:33.403431 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:02:33.403408 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/24fd0b69-36fa-4f00-89ab-77b624152a32-web-config\") pod \"prometheus-k8s-0\" (UID: \"24fd0b69-36fa-4f00-89ab-77b624152a32\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 21:02:33.403488 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:02:33.403450 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/24fd0b69-36fa-4f00-89ab-77b624152a32-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"24fd0b69-36fa-4f00-89ab-77b624152a32\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 21:02:33.406633 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:02:33.406609 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c5fml\" (UniqueName: \"kubernetes.io/projected/24fd0b69-36fa-4f00-89ab-77b624152a32-kube-api-access-c5fml\") pod \"prometheus-k8s-0\" (UID: \"24fd0b69-36fa-4f00-89ab-77b624152a32\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 21 21:02:33.494077 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:02:33.493810 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 21 21:02:33.651667 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:02:33.651642 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 21 21:02:33.653893 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:02:33.653863 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod24fd0b69_36fa_4f00_89ab_77b624152a32.slice/crio-4ea4c7ea7454a691611e143301e510d775cee1852399476f5a13097318621e56 WatchSource:0}: Error finding container 4ea4c7ea7454a691611e143301e510d775cee1852399476f5a13097318621e56: Status 404 returned error can't find the container with id 4ea4c7ea7454a691611e143301e510d775cee1852399476f5a13097318621e56 Jul 21 21:02:34.250800 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:02:34.250763 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"24fd0b69-36fa-4f00-89ab-77b624152a32","Type":"ContainerStarted","Data":"4ea4c7ea7454a691611e143301e510d775cee1852399476f5a13097318621e56"} Jul 21 21:02:36.259204 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:02:36.259171 2575 generic.go:356] "Generic (PLEG): container finished" podID="24fd0b69-36fa-4f00-89ab-77b624152a32" containerID="807b1f5dc9682af5bcdd890682ebf45f4896d0caabca63b157bac06e2165be20" exitCode=0 Jul 21 21:02:36.259610 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:02:36.259250 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"24fd0b69-36fa-4f00-89ab-77b624152a32","Type":"ContainerDied","Data":"807b1f5dc9682af5bcdd890682ebf45f4896d0caabca63b157bac06e2165be20"} Jul 21 21:02:41.284215 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:02:41.284181 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"24fd0b69-36fa-4f00-89ab-77b624152a32","Type":"ContainerStarted","Data":"f6e28a695e1420c6b47787a4c98893d9edda4dc3aeddffd0380698aba36b075f"} Jul 21 21:02:41.284678 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:02:41.284246 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"24fd0b69-36fa-4f00-89ab-77b624152a32","Type":"ContainerStarted","Data":"1c8ea95dfcaa50170bf6c95802bd19eb399d37e61e6ac161e25c19ce6267ca20"} Jul 21 21:02:44.302841 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:02:44.302760 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"24fd0b69-36fa-4f00-89ab-77b624152a32","Type":"ContainerStarted","Data":"7977c1a059b4925dd21170de61ba784c55ccdd9a22d57254c38acaa6eaf4da8c"} Jul 21 21:02:44.302841 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:02:44.302823 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"24fd0b69-36fa-4f00-89ab-77b624152a32","Type":"ContainerStarted","Data":"bdc4e667375f0dc2663bb727374c8945586540a23f4bec6dd7b424843362e02b"} Jul 21 21:02:44.303209 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:02:44.302848 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"24fd0b69-36fa-4f00-89ab-77b624152a32","Type":"ContainerStarted","Data":"090d71ce540f186b12a6aa9729b48ca4221e45f9f59a83c65bcce079783adecf"} Jul 21 21:02:44.303209 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:02:44.302877 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"24fd0b69-36fa-4f00-89ab-77b624152a32","Type":"ContainerStarted","Data":"329dfd0045ce964231af3febe00d5fdbfcb5db338fe723496ed46924098ab097"} Jul 21 21:02:44.334411 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:02:44.334356 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.298660801 podStartE2EDuration="11.334335415s" totalImagesPullingTime="9.012803875s" totalInitContainerRuntime="22.870739ms" isStatefulPod=false podCreationTimestamp="2026-07-21 21:02:33 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 21:02:44.331976828 +0000 UTC m=+122.266726494" watchObservedRunningTime="2026-07-21 21:02:44.334335415 +0000 UTC m=+122.269085082" Jul 21 21:02:48.494941 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:02:48.494902 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Jul 21 21:02:51.137188 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:02:51.137129 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-6c4d67b688-6m6c6" podUID="311eb160-46a8-4d6e-b448-5e6c0a6b776f" containerName="console" containerID="cri-o://6ce1f4923c6552fcf392b759d2ee589db5cc9cde39f1203be115464d5dcc0e06" gracePeriod=15 Jul 21 21:02:51.331246 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:02:51.331214 2575 generic.go:356] "Generic (PLEG): container finished" podID="311eb160-46a8-4d6e-b448-5e6c0a6b776f" containerID="6ce1f4923c6552fcf392b759d2ee589db5cc9cde39f1203be115464d5dcc0e06" exitCode=0 Jul 21 21:02:51.331369 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:02:51.331284 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6c4d67b688-6m6c6" event={"ID":"311eb160-46a8-4d6e-b448-5e6c0a6b776f","Type":"ContainerDied","Data":"6ce1f4923c6552fcf392b759d2ee589db5cc9cde39f1203be115464d5dcc0e06"} Jul 21 21:02:51.428334 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:02:51.428309 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6c4d67b688-6m6c6" Jul 21 21:02:51.564782 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:02:51.564751 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/311eb160-46a8-4d6e-b448-5e6c0a6b776f-console-serving-cert\") pod \"311eb160-46a8-4d6e-b448-5e6c0a6b776f\" (UID: \"311eb160-46a8-4d6e-b448-5e6c0a6b776f\") " Jul 21 21:02:51.564936 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:02:51.564790 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fxvmk\" (UniqueName: \"kubernetes.io/projected/311eb160-46a8-4d6e-b448-5e6c0a6b776f-kube-api-access-fxvmk\") pod \"311eb160-46a8-4d6e-b448-5e6c0a6b776f\" (UID: \"311eb160-46a8-4d6e-b448-5e6c0a6b776f\") " Jul 21 21:02:51.564936 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:02:51.564823 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/311eb160-46a8-4d6e-b448-5e6c0a6b776f-service-ca\") pod \"311eb160-46a8-4d6e-b448-5e6c0a6b776f\" (UID: \"311eb160-46a8-4d6e-b448-5e6c0a6b776f\") " Jul 21 21:02:51.564936 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:02:51.564847 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/311eb160-46a8-4d6e-b448-5e6c0a6b776f-console-config\") pod \"311eb160-46a8-4d6e-b448-5e6c0a6b776f\" (UID: \"311eb160-46a8-4d6e-b448-5e6c0a6b776f\") " Jul 21 21:02:51.564936 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:02:51.564893 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/311eb160-46a8-4d6e-b448-5e6c0a6b776f-oauth-serving-cert\") pod \"311eb160-46a8-4d6e-b448-5e6c0a6b776f\" (UID: \"311eb160-46a8-4d6e-b448-5e6c0a6b776f\") " Jul 21 21:02:51.564936 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:02:51.564924 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/311eb160-46a8-4d6e-b448-5e6c0a6b776f-console-oauth-config\") pod \"311eb160-46a8-4d6e-b448-5e6c0a6b776f\" (UID: \"311eb160-46a8-4d6e-b448-5e6c0a6b776f\") " Jul 21 21:02:51.565347 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:02:51.565322 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/311eb160-46a8-4d6e-b448-5e6c0a6b776f-service-ca" (OuterVolumeSpecName: "service-ca") pod "311eb160-46a8-4d6e-b448-5e6c0a6b776f" (UID: "311eb160-46a8-4d6e-b448-5e6c0a6b776f"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 21 21:02:51.565426 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:02:51.565364 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/311eb160-46a8-4d6e-b448-5e6c0a6b776f-console-config" (OuterVolumeSpecName: "console-config") pod "311eb160-46a8-4d6e-b448-5e6c0a6b776f" (UID: "311eb160-46a8-4d6e-b448-5e6c0a6b776f"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 21 21:02:51.565426 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:02:51.565366 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/311eb160-46a8-4d6e-b448-5e6c0a6b776f-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "311eb160-46a8-4d6e-b448-5e6c0a6b776f" (UID: "311eb160-46a8-4d6e-b448-5e6c0a6b776f"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 21 21:02:51.567130 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:02:51.567104 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/311eb160-46a8-4d6e-b448-5e6c0a6b776f-kube-api-access-fxvmk" (OuterVolumeSpecName: "kube-api-access-fxvmk") pod "311eb160-46a8-4d6e-b448-5e6c0a6b776f" (UID: "311eb160-46a8-4d6e-b448-5e6c0a6b776f"). InnerVolumeSpecName "kube-api-access-fxvmk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 21 21:02:51.567130 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:02:51.567112 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/311eb160-46a8-4d6e-b448-5e6c0a6b776f-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "311eb160-46a8-4d6e-b448-5e6c0a6b776f" (UID: "311eb160-46a8-4d6e-b448-5e6c0a6b776f"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 21 21:02:51.567130 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:02:51.567122 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/311eb160-46a8-4d6e-b448-5e6c0a6b776f-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "311eb160-46a8-4d6e-b448-5e6c0a6b776f" (UID: "311eb160-46a8-4d6e-b448-5e6c0a6b776f"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 21 21:02:51.665716 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:02:51.665634 2575 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/311eb160-46a8-4d6e-b448-5e6c0a6b776f-console-serving-cert\") on node \"ip-10-0-140-97.ec2.internal\" DevicePath \"\"" Jul 21 21:02:51.665716 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:02:51.665670 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fxvmk\" (UniqueName: \"kubernetes.io/projected/311eb160-46a8-4d6e-b448-5e6c0a6b776f-kube-api-access-fxvmk\") on node \"ip-10-0-140-97.ec2.internal\" DevicePath \"\"" Jul 21 21:02:51.665716 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:02:51.665681 2575 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/311eb160-46a8-4d6e-b448-5e6c0a6b776f-service-ca\") on node \"ip-10-0-140-97.ec2.internal\" DevicePath \"\"" Jul 21 21:02:51.665716 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:02:51.665690 2575 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/311eb160-46a8-4d6e-b448-5e6c0a6b776f-console-config\") on node \"ip-10-0-140-97.ec2.internal\" DevicePath \"\"" Jul 21 21:02:51.665716 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:02:51.665699 2575 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/311eb160-46a8-4d6e-b448-5e6c0a6b776f-oauth-serving-cert\") on node \"ip-10-0-140-97.ec2.internal\" DevicePath \"\"" Jul 21 21:02:51.665716 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:02:51.665707 2575 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/311eb160-46a8-4d6e-b448-5e6c0a6b776f-console-oauth-config\") on node \"ip-10-0-140-97.ec2.internal\" DevicePath \"\"" Jul 21 21:02:52.271085 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:02:52.271050 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/67b28cbf-d3f3-4d32-b87f-f1e586e2faca-metrics-certs\") pod \"network-metrics-daemon-6fx5v\" (UID: \"67b28cbf-d3f3-4d32-b87f-f1e586e2faca\") " pod="openshift-multus/network-metrics-daemon-6fx5v" Jul 21 21:02:52.273279 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:02:52.273259 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/67b28cbf-d3f3-4d32-b87f-f1e586e2faca-metrics-certs\") pod \"network-metrics-daemon-6fx5v\" (UID: \"67b28cbf-d3f3-4d32-b87f-f1e586e2faca\") " pod="openshift-multus/network-metrics-daemon-6fx5v" Jul 21 21:02:52.336882 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:02:52.336859 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6c4d67b688-6m6c6" Jul 21 21:02:52.336882 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:02:52.336872 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6c4d67b688-6m6c6" event={"ID":"311eb160-46a8-4d6e-b448-5e6c0a6b776f","Type":"ContainerDied","Data":"5ece47c0432fcf5226ac4092a7ba4f8509030fbbf0e6e0e8508e6957033852b8"} Jul 21 21:02:52.337064 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:02:52.336925 2575 scope.go:117] "RemoveContainer" containerID="6ce1f4923c6552fcf392b759d2ee589db5cc9cde39f1203be115464d5dcc0e06" Jul 21 21:02:52.358629 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:02:52.358600 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6c4d67b688-6m6c6"] Jul 21 21:02:52.362320 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:02:52.362299 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-6c4d67b688-6m6c6"] Jul 21 21:02:52.557174 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:02:52.557096 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-cvwws\"" Jul 21 21:02:52.565001 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:02:52.564984 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6fx5v" Jul 21 21:02:52.724151 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:02:52.724128 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-6fx5v"] Jul 21 21:02:52.726345 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:02:52.726315 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod67b28cbf_d3f3_4d32_b87f_f1e586e2faca.slice/crio-4b25d91c9f4cd49c67a0e0c9974d48f3b47ff467b40c01d2a94c85cdab20df48 WatchSource:0}: Error finding container 4b25d91c9f4cd49c67a0e0c9974d48f3b47ff467b40c01d2a94c85cdab20df48: Status 404 returned error can't find the container with id 4b25d91c9f4cd49c67a0e0c9974d48f3b47ff467b40c01d2a94c85cdab20df48 Jul 21 21:02:52.743943 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:02:52.743915 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="311eb160-46a8-4d6e-b448-5e6c0a6b776f" path="/var/lib/kubelet/pods/311eb160-46a8-4d6e-b448-5e6c0a6b776f/volumes" Jul 21 21:02:53.343847 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:02:53.343802 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-6fx5v" event={"ID":"67b28cbf-d3f3-4d32-b87f-f1e586e2faca","Type":"ContainerStarted","Data":"4b25d91c9f4cd49c67a0e0c9974d48f3b47ff467b40c01d2a94c85cdab20df48"} Jul 21 21:02:54.350199 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:02:54.350112 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-6fx5v" event={"ID":"67b28cbf-d3f3-4d32-b87f-f1e586e2faca","Type":"ContainerStarted","Data":"ca11465515d85fd894dc8b6d6754a79155424ad210c0a9bd2d8e4a9e93b5d92b"} Jul 21 21:02:54.350199 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:02:54.350162 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-6fx5v" event={"ID":"67b28cbf-d3f3-4d32-b87f-f1e586e2faca","Type":"ContainerStarted","Data":"055d563fb346a752cb6c7dd689f89ffcc75326ad12730002fe4c3ccdb5a24d72"} Jul 21 21:02:54.366150 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:02:54.366101 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-6fx5v" podStartSLOduration=130.997644828 podStartE2EDuration="2m12.366083831s" totalImagesPullingTime="1.368439003s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 21:00:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 21:02:54.365117313 +0000 UTC m=+132.299866992" watchObservedRunningTime="2026-07-21 21:02:54.366083831 +0000 UTC m=+132.300833497" Jul 21 21:02:58.365591 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:02:58.365559 2575 generic.go:356] "Generic (PLEG): container finished" podID="ca2c8246-50a5-48fb-be89-a95b401523a7" containerID="934d73c3726f025ff75e9210cd3e3f1086c309270229e07e5930fec16b6acd0e" exitCode=0 Jul 21 21:02:58.365970 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:02:58.365615 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-7zc94" event={"ID":"ca2c8246-50a5-48fb-be89-a95b401523a7","Type":"ContainerDied","Data":"934d73c3726f025ff75e9210cd3e3f1086c309270229e07e5930fec16b6acd0e"} Jul 21 21:02:58.365970 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:02:58.365958 2575 scope.go:117] "RemoveContainer" containerID="934d73c3726f025ff75e9210cd3e3f1086c309270229e07e5930fec16b6acd0e" Jul 21 21:02:59.371975 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:02:59.371942 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-7zc94" event={"ID":"ca2c8246-50a5-48fb-be89-a95b401523a7","Type":"ContainerStarted","Data":"1bb4b0e33cca7ac70aa785b3e1c3a87f1566cb5f6af0593119e325c797f42c85"} Jul 21 21:03:08.412749 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:03:08.412716 2575 generic.go:356] "Generic (PLEG): container finished" podID="8a30add9-0efd-4cf9-99df-17229907f235" containerID="5f9ba926f889efc7afb3b0d2f432352d9d650da9b36592accb3322eb3c813b8d" exitCode=0 Jul 21 21:03:08.413271 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:03:08.412765 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-b2hpw" event={"ID":"8a30add9-0efd-4cf9-99df-17229907f235","Type":"ContainerDied","Data":"5f9ba926f889efc7afb3b0d2f432352d9d650da9b36592accb3322eb3c813b8d"} Jul 21 21:03:08.413271 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:03:08.413163 2575 scope.go:117] "RemoveContainer" containerID="5f9ba926f889efc7afb3b0d2f432352d9d650da9b36592accb3322eb3c813b8d" Jul 21 21:03:09.419238 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:03:09.419200 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-b2hpw" event={"ID":"8a30add9-0efd-4cf9-99df-17229907f235","Type":"ContainerStarted","Data":"38292b84b8645a72897fec0c4ff536faab9c70ec6a790eaa1e1ec78d1bcf988e"} Jul 21 21:03:33.495341 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:03:33.495305 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0" Jul 21 21:03:33.510920 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:03:33.510896 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0" Jul 21 21:03:33.525202 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:03:33.525180 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0" Jul 21 21:05:42.419582 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:05:42.419554 2575 kubelet.go:1648] "Image garbage collection succeeded" Jul 21 21:05:55.667328 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:05:55.667285 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-pxdxj"] Jul 21 21:05:55.669030 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:05:55.667702 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="311eb160-46a8-4d6e-b448-5e6c0a6b776f" containerName="console" Jul 21 21:05:55.669030 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:05:55.667720 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="311eb160-46a8-4d6e-b448-5e6c0a6b776f" containerName="console" Jul 21 21:05:55.669030 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:05:55.667805 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="311eb160-46a8-4d6e-b448-5e6c0a6b776f" containerName="console" Jul 21 21:05:55.669823 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:05:55.669803 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-pxdxj" Jul 21 21:05:55.672385 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:05:55.672353 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\"" Jul 21 21:05:55.672672 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:05:55.672652 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\"" Jul 21 21:05:55.673436 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:05:55.673419 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"openshift-lws-operator-dockercfg-fw5bz\"" Jul 21 21:05:55.682324 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:05:55.682304 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-pxdxj"] Jul 21 21:05:55.818632 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:05:55.818602 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-flgf2\" (UniqueName: \"kubernetes.io/projected/edaed366-c509-46cb-8d00-555f055259e2-kube-api-access-flgf2\") pod \"openshift-lws-operator-fd8ccff4c-pxdxj\" (UID: \"edaed366-c509-46cb-8d00-555f055259e2\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-pxdxj" Jul 21 21:05:55.818776 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:05:55.818679 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/edaed366-c509-46cb-8d00-555f055259e2-tmp\") pod \"openshift-lws-operator-fd8ccff4c-pxdxj\" (UID: \"edaed366-c509-46cb-8d00-555f055259e2\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-pxdxj" Jul 21 21:05:55.919726 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:05:55.919655 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-flgf2\" (UniqueName: \"kubernetes.io/projected/edaed366-c509-46cb-8d00-555f055259e2-kube-api-access-flgf2\") pod \"openshift-lws-operator-fd8ccff4c-pxdxj\" (UID: \"edaed366-c509-46cb-8d00-555f055259e2\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-pxdxj" Jul 21 21:05:55.919726 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:05:55.919724 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/edaed366-c509-46cb-8d00-555f055259e2-tmp\") pod \"openshift-lws-operator-fd8ccff4c-pxdxj\" (UID: \"edaed366-c509-46cb-8d00-555f055259e2\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-pxdxj" Jul 21 21:05:55.920054 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:05:55.920039 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/edaed366-c509-46cb-8d00-555f055259e2-tmp\") pod \"openshift-lws-operator-fd8ccff4c-pxdxj\" (UID: \"edaed366-c509-46cb-8d00-555f055259e2\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-pxdxj" Jul 21 21:05:55.927617 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:05:55.927589 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-flgf2\" (UniqueName: \"kubernetes.io/projected/edaed366-c509-46cb-8d00-555f055259e2-kube-api-access-flgf2\") pod \"openshift-lws-operator-fd8ccff4c-pxdxj\" (UID: \"edaed366-c509-46cb-8d00-555f055259e2\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-pxdxj" Jul 21 21:05:55.979125 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:05:55.979103 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-pxdxj" Jul 21 21:05:56.100146 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:05:56.100112 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-pxdxj"] Jul 21 21:05:56.103047 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:05:56.103022 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podedaed366_c509_46cb_8d00_555f055259e2.slice/crio-b7d48a7724c91139ad5ef44a329cdf0a53bc158d4cc9db47b3b7de6877786320 WatchSource:0}: Error finding container b7d48a7724c91139ad5ef44a329cdf0a53bc158d4cc9db47b3b7de6877786320: Status 404 returned error can't find the container with id b7d48a7724c91139ad5ef44a329cdf0a53bc158d4cc9db47b3b7de6877786320 Jul 21 21:05:56.104752 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:05:56.104731 2575 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 21 21:05:57.032812 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:05:57.032740 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-pxdxj" event={"ID":"edaed366-c509-46cb-8d00-555f055259e2","Type":"ContainerStarted","Data":"b7d48a7724c91139ad5ef44a329cdf0a53bc158d4cc9db47b3b7de6877786320"} Jul 21 21:05:59.043499 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:05:59.043412 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-pxdxj" event={"ID":"edaed366-c509-46cb-8d00-555f055259e2","Type":"ContainerStarted","Data":"1932f47497c38ac0ff1c86a4ac6213413990678da1f42b00f29867987a005190"} Jul 21 21:05:59.066201 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:05:59.066152 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-pxdxj" podStartSLOduration=1.485828297 podStartE2EDuration="4.066134221s" totalImagesPullingTime="2.580305924s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 21:05:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 21:05:59.065705381 +0000 UTC m=+317.000455047" watchObservedRunningTime="2026-07-21 21:05:59.066134221 +0000 UTC m=+317.000883889" Jul 21 21:06:14.933823 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:06:14.933783 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-694dcbb678-vqdp9"] Jul 21 21:06:14.937987 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:06:14.937966 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-694dcbb678-vqdp9" Jul 21 21:06:14.940647 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:06:14.940622 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Jul 21 21:06:14.940758 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:06:14.940661 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Jul 21 21:06:14.941153 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:06:14.941134 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Jul 21 21:06:14.941224 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:06:14.941186 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Jul 21 21:06:14.941326 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:06:14.941305 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-ddkqb\"" Jul 21 21:06:14.954813 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:06:14.954782 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-694dcbb678-vqdp9"] Jul 21 21:06:15.083853 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:06:15.083808 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/a2248fc1-a1c0-461a-87dd-8f6f9cb0d634-manifests\") pod \"opendatahub-operator-controller-manager-694dcbb678-vqdp9\" (UID: \"a2248fc1-a1c0-461a-87dd-8f6f9cb0d634\") " pod="opendatahub/opendatahub-operator-controller-manager-694dcbb678-vqdp9" Jul 21 21:06:15.083853 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:06:15.083848 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a2248fc1-a1c0-461a-87dd-8f6f9cb0d634-tmp\") pod \"opendatahub-operator-controller-manager-694dcbb678-vqdp9\" (UID: \"a2248fc1-a1c0-461a-87dd-8f6f9cb0d634\") " pod="opendatahub/opendatahub-operator-controller-manager-694dcbb678-vqdp9" Jul 21 21:06:15.084055 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:06:15.083961 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/a2248fc1-a1c0-461a-87dd-8f6f9cb0d634-webhook-cert\") pod \"opendatahub-operator-controller-manager-694dcbb678-vqdp9\" (UID: \"a2248fc1-a1c0-461a-87dd-8f6f9cb0d634\") " pod="opendatahub/opendatahub-operator-controller-manager-694dcbb678-vqdp9" Jul 21 21:06:15.084055 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:06:15.084018 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7fmws\" (UniqueName: \"kubernetes.io/projected/a2248fc1-a1c0-461a-87dd-8f6f9cb0d634-kube-api-access-7fmws\") pod \"opendatahub-operator-controller-manager-694dcbb678-vqdp9\" (UID: \"a2248fc1-a1c0-461a-87dd-8f6f9cb0d634\") " pod="opendatahub/opendatahub-operator-controller-manager-694dcbb678-vqdp9" Jul 21 21:06:15.084055 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:06:15.084052 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/a2248fc1-a1c0-461a-87dd-8f6f9cb0d634-apiservice-cert\") pod \"opendatahub-operator-controller-manager-694dcbb678-vqdp9\" (UID: \"a2248fc1-a1c0-461a-87dd-8f6f9cb0d634\") " pod="opendatahub/opendatahub-operator-controller-manager-694dcbb678-vqdp9" Jul 21 21:06:15.185249 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:06:15.185158 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/a2248fc1-a1c0-461a-87dd-8f6f9cb0d634-apiservice-cert\") pod \"opendatahub-operator-controller-manager-694dcbb678-vqdp9\" (UID: \"a2248fc1-a1c0-461a-87dd-8f6f9cb0d634\") " pod="opendatahub/opendatahub-operator-controller-manager-694dcbb678-vqdp9" Jul 21 21:06:15.185249 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:06:15.185208 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/a2248fc1-a1c0-461a-87dd-8f6f9cb0d634-manifests\") pod \"opendatahub-operator-controller-manager-694dcbb678-vqdp9\" (UID: \"a2248fc1-a1c0-461a-87dd-8f6f9cb0d634\") " pod="opendatahub/opendatahub-operator-controller-manager-694dcbb678-vqdp9" Jul 21 21:06:15.185249 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:06:15.185226 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a2248fc1-a1c0-461a-87dd-8f6f9cb0d634-tmp\") pod \"opendatahub-operator-controller-manager-694dcbb678-vqdp9\" (UID: \"a2248fc1-a1c0-461a-87dd-8f6f9cb0d634\") " pod="opendatahub/opendatahub-operator-controller-manager-694dcbb678-vqdp9" Jul 21 21:06:15.185469 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:06:15.185270 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/a2248fc1-a1c0-461a-87dd-8f6f9cb0d634-webhook-cert\") pod \"opendatahub-operator-controller-manager-694dcbb678-vqdp9\" (UID: \"a2248fc1-a1c0-461a-87dd-8f6f9cb0d634\") " pod="opendatahub/opendatahub-operator-controller-manager-694dcbb678-vqdp9" Jul 21 21:06:15.185469 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:06:15.185299 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7fmws\" (UniqueName: \"kubernetes.io/projected/a2248fc1-a1c0-461a-87dd-8f6f9cb0d634-kube-api-access-7fmws\") pod \"opendatahub-operator-controller-manager-694dcbb678-vqdp9\" (UID: \"a2248fc1-a1c0-461a-87dd-8f6f9cb0d634\") " pod="opendatahub/opendatahub-operator-controller-manager-694dcbb678-vqdp9" Jul 21 21:06:15.185682 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:06:15.185654 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/a2248fc1-a1c0-461a-87dd-8f6f9cb0d634-manifests\") pod \"opendatahub-operator-controller-manager-694dcbb678-vqdp9\" (UID: \"a2248fc1-a1c0-461a-87dd-8f6f9cb0d634\") " pod="opendatahub/opendatahub-operator-controller-manager-694dcbb678-vqdp9" Jul 21 21:06:15.187592 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:06:15.187564 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a2248fc1-a1c0-461a-87dd-8f6f9cb0d634-tmp\") pod \"opendatahub-operator-controller-manager-694dcbb678-vqdp9\" (UID: \"a2248fc1-a1c0-461a-87dd-8f6f9cb0d634\") " pod="opendatahub/opendatahub-operator-controller-manager-694dcbb678-vqdp9" Jul 21 21:06:15.187810 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:06:15.187791 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/a2248fc1-a1c0-461a-87dd-8f6f9cb0d634-apiservice-cert\") pod \"opendatahub-operator-controller-manager-694dcbb678-vqdp9\" (UID: \"a2248fc1-a1c0-461a-87dd-8f6f9cb0d634\") " pod="opendatahub/opendatahub-operator-controller-manager-694dcbb678-vqdp9" Jul 21 21:06:15.187938 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:06:15.187916 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/a2248fc1-a1c0-461a-87dd-8f6f9cb0d634-webhook-cert\") pod \"opendatahub-operator-controller-manager-694dcbb678-vqdp9\" (UID: \"a2248fc1-a1c0-461a-87dd-8f6f9cb0d634\") " pod="opendatahub/opendatahub-operator-controller-manager-694dcbb678-vqdp9" Jul 21 21:06:15.195279 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:06:15.195252 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7fmws\" (UniqueName: \"kubernetes.io/projected/a2248fc1-a1c0-461a-87dd-8f6f9cb0d634-kube-api-access-7fmws\") pod \"opendatahub-operator-controller-manager-694dcbb678-vqdp9\" (UID: \"a2248fc1-a1c0-461a-87dd-8f6f9cb0d634\") " pod="opendatahub/opendatahub-operator-controller-manager-694dcbb678-vqdp9" Jul 21 21:06:15.250000 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:06:15.249964 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-694dcbb678-vqdp9" Jul 21 21:06:15.388477 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:06:15.388437 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-694dcbb678-vqdp9"] Jul 21 21:06:15.390879 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:06:15.390847 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda2248fc1_a1c0_461a_87dd_8f6f9cb0d634.slice/crio-e01c79938de733b4127bf1c512da814b9341124a2e8ea46114893856b1076851 WatchSource:0}: Error finding container e01c79938de733b4127bf1c512da814b9341124a2e8ea46114893856b1076851: Status 404 returned error can't find the container with id e01c79938de733b4127bf1c512da814b9341124a2e8ea46114893856b1076851 Jul 21 21:06:16.108997 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:06:16.108958 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-694dcbb678-vqdp9" event={"ID":"a2248fc1-a1c0-461a-87dd-8f6f9cb0d634","Type":"ContainerStarted","Data":"e01c79938de733b4127bf1c512da814b9341124a2e8ea46114893856b1076851"} Jul 21 21:06:19.124384 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:06:19.124344 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-694dcbb678-vqdp9" event={"ID":"a2248fc1-a1c0-461a-87dd-8f6f9cb0d634","Type":"ContainerStarted","Data":"bfa25f339271c8fbee1dcbd725db9502b5589e3b0ec05b5f7f725beac1177a83"} Jul 21 21:06:20.130831 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:06:20.130793 2575 generic.go:356] "Generic (PLEG): container finished" podID="a2248fc1-a1c0-461a-87dd-8f6f9cb0d634" containerID="bfa25f339271c8fbee1dcbd725db9502b5589e3b0ec05b5f7f725beac1177a83" exitCode=0 Jul 21 21:06:20.131226 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:06:20.130842 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-694dcbb678-vqdp9" event={"ID":"a2248fc1-a1c0-461a-87dd-8f6f9cb0d634","Type":"ContainerDied","Data":"bfa25f339271c8fbee1dcbd725db9502b5589e3b0ec05b5f7f725beac1177a83"} Jul 21 21:06:21.136473 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:06:21.136435 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-694dcbb678-vqdp9" event={"ID":"a2248fc1-a1c0-461a-87dd-8f6f9cb0d634","Type":"ContainerStarted","Data":"60fb39595d172b9616f21f477b963438e357136583ae43edf892345edf37ae47"} Jul 21 21:06:21.136863 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:06:21.136629 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-694dcbb678-vqdp9" Jul 21 21:06:21.159729 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:06:21.159678 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-694dcbb678-vqdp9" podStartSLOduration=2.546000142 podStartE2EDuration="7.159660849s" totalImagesPullingTime="3.447417857s" totalInitContainerRuntime="1.16624285s" isStatefulPod=false podCreationTimestamp="2026-07-21 21:06:14 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 21:06:21.158424127 +0000 UTC m=+339.093173790" watchObservedRunningTime="2026-07-21 21:06:21.159660849 +0000 UTC m=+339.094410514" Jul 21 21:06:32.142679 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:06:32.142642 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-694dcbb678-vqdp9" Jul 21 21:06:34.648303 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:06:34.648220 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-8f965b796-8wsc2"] Jul 21 21:06:34.651390 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:06:34.651374 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-8f965b796-8wsc2" Jul 21 21:06:34.654139 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:06:34.654116 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-tdtgn\"" Jul 21 21:06:34.654139 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:06:34.654132 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\"" Jul 21 21:06:34.655015 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:06:34.654996 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\"" Jul 21 21:06:34.655116 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:06:34.655087 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\"" Jul 21 21:06:34.664602 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:06:34.664578 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-8f965b796-8wsc2"] Jul 21 21:06:34.751330 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:06:34.751298 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/cbc0cc0e-a3cf-48d3-9417-64d1cda9f6ab-manager-config\") pod \"lws-controller-manager-8f965b796-8wsc2\" (UID: \"cbc0cc0e-a3cf-48d3-9417-64d1cda9f6ab\") " pod="openshift-lws-operator/lws-controller-manager-8f965b796-8wsc2" Jul 21 21:06:34.751492 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:06:34.751335 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/cbc0cc0e-a3cf-48d3-9417-64d1cda9f6ab-metrics-cert\") pod \"lws-controller-manager-8f965b796-8wsc2\" (UID: \"cbc0cc0e-a3cf-48d3-9417-64d1cda9f6ab\") " pod="openshift-lws-operator/lws-controller-manager-8f965b796-8wsc2" Jul 21 21:06:34.751492 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:06:34.751425 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hgccq\" (UniqueName: \"kubernetes.io/projected/cbc0cc0e-a3cf-48d3-9417-64d1cda9f6ab-kube-api-access-hgccq\") pod \"lws-controller-manager-8f965b796-8wsc2\" (UID: \"cbc0cc0e-a3cf-48d3-9417-64d1cda9f6ab\") " pod="openshift-lws-operator/lws-controller-manager-8f965b796-8wsc2" Jul 21 21:06:34.751492 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:06:34.751463 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/cbc0cc0e-a3cf-48d3-9417-64d1cda9f6ab-cert\") pod \"lws-controller-manager-8f965b796-8wsc2\" (UID: \"cbc0cc0e-a3cf-48d3-9417-64d1cda9f6ab\") " pod="openshift-lws-operator/lws-controller-manager-8f965b796-8wsc2" Jul 21 21:06:34.852182 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:06:34.852140 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/cbc0cc0e-a3cf-48d3-9417-64d1cda9f6ab-manager-config\") pod \"lws-controller-manager-8f965b796-8wsc2\" (UID: \"cbc0cc0e-a3cf-48d3-9417-64d1cda9f6ab\") " pod="openshift-lws-operator/lws-controller-manager-8f965b796-8wsc2" Jul 21 21:06:34.852348 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:06:34.852240 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/cbc0cc0e-a3cf-48d3-9417-64d1cda9f6ab-metrics-cert\") pod \"lws-controller-manager-8f965b796-8wsc2\" (UID: \"cbc0cc0e-a3cf-48d3-9417-64d1cda9f6ab\") " pod="openshift-lws-operator/lws-controller-manager-8f965b796-8wsc2" Jul 21 21:06:34.852402 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:06:34.852355 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hgccq\" (UniqueName: \"kubernetes.io/projected/cbc0cc0e-a3cf-48d3-9417-64d1cda9f6ab-kube-api-access-hgccq\") pod \"lws-controller-manager-8f965b796-8wsc2\" (UID: \"cbc0cc0e-a3cf-48d3-9417-64d1cda9f6ab\") " pod="openshift-lws-operator/lws-controller-manager-8f965b796-8wsc2" Jul 21 21:06:34.852455 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:06:34.852404 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/cbc0cc0e-a3cf-48d3-9417-64d1cda9f6ab-cert\") pod \"lws-controller-manager-8f965b796-8wsc2\" (UID: \"cbc0cc0e-a3cf-48d3-9417-64d1cda9f6ab\") " pod="openshift-lws-operator/lws-controller-manager-8f965b796-8wsc2" Jul 21 21:06:34.853000 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:06:34.852969 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/cbc0cc0e-a3cf-48d3-9417-64d1cda9f6ab-manager-config\") pod \"lws-controller-manager-8f965b796-8wsc2\" (UID: \"cbc0cc0e-a3cf-48d3-9417-64d1cda9f6ab\") " pod="openshift-lws-operator/lws-controller-manager-8f965b796-8wsc2" Jul 21 21:06:34.854763 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:06:34.854743 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/cbc0cc0e-a3cf-48d3-9417-64d1cda9f6ab-cert\") pod \"lws-controller-manager-8f965b796-8wsc2\" (UID: \"cbc0cc0e-a3cf-48d3-9417-64d1cda9f6ab\") " pod="openshift-lws-operator/lws-controller-manager-8f965b796-8wsc2" Jul 21 21:06:34.854936 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:06:34.854917 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/cbc0cc0e-a3cf-48d3-9417-64d1cda9f6ab-metrics-cert\") pod \"lws-controller-manager-8f965b796-8wsc2\" (UID: \"cbc0cc0e-a3cf-48d3-9417-64d1cda9f6ab\") " pod="openshift-lws-operator/lws-controller-manager-8f965b796-8wsc2" Jul 21 21:06:34.860398 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:06:34.860367 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hgccq\" (UniqueName: \"kubernetes.io/projected/cbc0cc0e-a3cf-48d3-9417-64d1cda9f6ab-kube-api-access-hgccq\") pod \"lws-controller-manager-8f965b796-8wsc2\" (UID: \"cbc0cc0e-a3cf-48d3-9417-64d1cda9f6ab\") " pod="openshift-lws-operator/lws-controller-manager-8f965b796-8wsc2" Jul 21 21:06:34.960829 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:06:34.960746 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-8f965b796-8wsc2" Jul 21 21:06:35.093631 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:06:35.093605 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-8f965b796-8wsc2"] Jul 21 21:06:35.095927 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:06:35.095901 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcbc0cc0e_a3cf_48d3_9417_64d1cda9f6ab.slice/crio-3b6919beeca14886e1ce3b7058b0d5a076cc247245d64447c53338199412b9c0 WatchSource:0}: Error finding container 3b6919beeca14886e1ce3b7058b0d5a076cc247245d64447c53338199412b9c0: Status 404 returned error can't find the container with id 3b6919beeca14886e1ce3b7058b0d5a076cc247245d64447c53338199412b9c0 Jul 21 21:06:35.196677 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:06:35.196640 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-8f965b796-8wsc2" event={"ID":"cbc0cc0e-a3cf-48d3-9417-64d1cda9f6ab","Type":"ContainerStarted","Data":"3b6919beeca14886e1ce3b7058b0d5a076cc247245d64447c53338199412b9c0"} Jul 21 21:06:37.209493 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:06:37.209458 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-8f965b796-8wsc2" event={"ID":"cbc0cc0e-a3cf-48d3-9417-64d1cda9f6ab","Type":"ContainerStarted","Data":"8fd7e953115870962fa427eac67988fb6e89854e56ab4f98bde4e55aadfdaae9"} Jul 21 21:06:37.209895 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:06:37.209566 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-8f965b796-8wsc2" Jul 21 21:06:37.226845 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:06:37.226797 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-8f965b796-8wsc2" podStartSLOduration=1.79915604 podStartE2EDuration="3.226781213s" totalImagesPullingTime="1.427625173s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 21:06:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 21:06:37.225524882 +0000 UTC m=+355.160274548" watchObservedRunningTime="2026-07-21 21:06:37.226781213 +0000 UTC m=+355.161530878" Jul 21 21:06:48.217232 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:06:48.217199 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-8f965b796-8wsc2" Jul 21 21:06:52.941127 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:06:52.941083 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4v97t9"] Jul 21 21:06:52.949269 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:06:52.949239 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4v97t9" Jul 21 21:06:52.951778 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:06:52.951742 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Jul 21 21:06:52.951943 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:06:52.951839 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"data-science-gateway-data-science-gateway-class-dockercfg-jvm46\"" Jul 21 21:06:52.957772 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:06:52.957745 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4v97t9"] Jul 21 21:06:53.114881 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:06:53.114831 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4573f5c2-4d6e-4c0d-a850-55b738a81de7-istio-token\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4v97t9\" (UID: \"4573f5c2-4d6e-4c0d-a850-55b738a81de7\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4v97t9" Jul 21 21:06:53.114881 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:06:53.114888 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4573f5c2-4d6e-4c0d-a850-55b738a81de7-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4v97t9\" (UID: \"4573f5c2-4d6e-4c0d-a850-55b738a81de7\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4v97t9" Jul 21 21:06:53.115108 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:06:53.114905 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4573f5c2-4d6e-4c0d-a850-55b738a81de7-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4v97t9\" (UID: \"4573f5c2-4d6e-4c0d-a850-55b738a81de7\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4v97t9" Jul 21 21:06:53.115108 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:06:53.114922 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4573f5c2-4d6e-4c0d-a850-55b738a81de7-istio-data\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4v97t9\" (UID: \"4573f5c2-4d6e-4c0d-a850-55b738a81de7\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4v97t9" Jul 21 21:06:53.115108 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:06:53.114942 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4573f5c2-4d6e-4c0d-a850-55b738a81de7-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4v97t9\" (UID: \"4573f5c2-4d6e-4c0d-a850-55b738a81de7\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4v97t9" Jul 21 21:06:53.115108 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:06:53.115039 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4573f5c2-4d6e-4c0d-a850-55b738a81de7-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4v97t9\" (UID: \"4573f5c2-4d6e-4c0d-a850-55b738a81de7\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4v97t9" Jul 21 21:06:53.115108 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:06:53.115079 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jmkz6\" (UniqueName: \"kubernetes.io/projected/4573f5c2-4d6e-4c0d-a850-55b738a81de7-kube-api-access-jmkz6\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4v97t9\" (UID: \"4573f5c2-4d6e-4c0d-a850-55b738a81de7\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4v97t9" Jul 21 21:06:53.115108 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:06:53.115101 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4573f5c2-4d6e-4c0d-a850-55b738a81de7-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4v97t9\" (UID: \"4573f5c2-4d6e-4c0d-a850-55b738a81de7\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4v97t9" Jul 21 21:06:53.115318 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:06:53.115121 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4573f5c2-4d6e-4c0d-a850-55b738a81de7-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4v97t9\" (UID: \"4573f5c2-4d6e-4c0d-a850-55b738a81de7\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4v97t9" Jul 21 21:06:53.215858 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:06:53.215756 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jmkz6\" (UniqueName: \"kubernetes.io/projected/4573f5c2-4d6e-4c0d-a850-55b738a81de7-kube-api-access-jmkz6\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4v97t9\" (UID: \"4573f5c2-4d6e-4c0d-a850-55b738a81de7\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4v97t9" Jul 21 21:06:53.215858 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:06:53.215803 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4573f5c2-4d6e-4c0d-a850-55b738a81de7-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4v97t9\" (UID: \"4573f5c2-4d6e-4c0d-a850-55b738a81de7\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4v97t9" Jul 21 21:06:53.215858 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:06:53.215825 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4573f5c2-4d6e-4c0d-a850-55b738a81de7-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4v97t9\" (UID: \"4573f5c2-4d6e-4c0d-a850-55b738a81de7\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4v97t9" Jul 21 21:06:53.215858 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:06:53.215854 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4573f5c2-4d6e-4c0d-a850-55b738a81de7-istio-token\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4v97t9\" (UID: \"4573f5c2-4d6e-4c0d-a850-55b738a81de7\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4v97t9" Jul 21 21:06:53.216186 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:06:53.215884 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4573f5c2-4d6e-4c0d-a850-55b738a81de7-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4v97t9\" (UID: \"4573f5c2-4d6e-4c0d-a850-55b738a81de7\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4v97t9" Jul 21 21:06:53.216186 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:06:53.215899 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4573f5c2-4d6e-4c0d-a850-55b738a81de7-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4v97t9\" (UID: \"4573f5c2-4d6e-4c0d-a850-55b738a81de7\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4v97t9" Jul 21 21:06:53.216186 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:06:53.215918 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4573f5c2-4d6e-4c0d-a850-55b738a81de7-istio-data\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4v97t9\" (UID: \"4573f5c2-4d6e-4c0d-a850-55b738a81de7\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4v97t9" Jul 21 21:06:53.216186 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:06:53.215949 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4573f5c2-4d6e-4c0d-a850-55b738a81de7-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4v97t9\" (UID: \"4573f5c2-4d6e-4c0d-a850-55b738a81de7\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4v97t9" Jul 21 21:06:53.216186 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:06:53.216022 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4573f5c2-4d6e-4c0d-a850-55b738a81de7-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4v97t9\" (UID: \"4573f5c2-4d6e-4c0d-a850-55b738a81de7\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4v97t9" Jul 21 21:06:53.216426 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:06:53.216267 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4573f5c2-4d6e-4c0d-a850-55b738a81de7-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4v97t9\" (UID: \"4573f5c2-4d6e-4c0d-a850-55b738a81de7\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4v97t9" Jul 21 21:06:53.216480 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:06:53.216453 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4573f5c2-4d6e-4c0d-a850-55b738a81de7-istio-data\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4v97t9\" (UID: \"4573f5c2-4d6e-4c0d-a850-55b738a81de7\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4v97t9" Jul 21 21:06:53.216571 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:06:53.216496 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4573f5c2-4d6e-4c0d-a850-55b738a81de7-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4v97t9\" (UID: \"4573f5c2-4d6e-4c0d-a850-55b738a81de7\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4v97t9" Jul 21 21:06:53.216629 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:06:53.216593 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4573f5c2-4d6e-4c0d-a850-55b738a81de7-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4v97t9\" (UID: \"4573f5c2-4d6e-4c0d-a850-55b738a81de7\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4v97t9" Jul 21 21:06:53.216794 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:06:53.216773 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4573f5c2-4d6e-4c0d-a850-55b738a81de7-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4v97t9\" (UID: \"4573f5c2-4d6e-4c0d-a850-55b738a81de7\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4v97t9" Jul 21 21:06:53.218270 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:06:53.218251 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4573f5c2-4d6e-4c0d-a850-55b738a81de7-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4v97t9\" (UID: \"4573f5c2-4d6e-4c0d-a850-55b738a81de7\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4v97t9" Jul 21 21:06:53.218561 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:06:53.218542 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4573f5c2-4d6e-4c0d-a850-55b738a81de7-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4v97t9\" (UID: \"4573f5c2-4d6e-4c0d-a850-55b738a81de7\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4v97t9" Jul 21 21:06:53.224302 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:06:53.224262 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4573f5c2-4d6e-4c0d-a850-55b738a81de7-istio-token\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4v97t9\" (UID: \"4573f5c2-4d6e-4c0d-a850-55b738a81de7\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4v97t9" Jul 21 21:06:53.224462 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:06:53.224418 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jmkz6\" (UniqueName: \"kubernetes.io/projected/4573f5c2-4d6e-4c0d-a850-55b738a81de7-kube-api-access-jmkz6\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4v97t9\" (UID: \"4573f5c2-4d6e-4c0d-a850-55b738a81de7\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4v97t9" Jul 21 21:06:53.262128 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:06:53.262080 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4v97t9" Jul 21 21:06:53.422482 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:06:53.422452 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4v97t9"] Jul 21 21:06:53.424367 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:06:53.424338 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4573f5c2_4d6e_4c0d_a850_55b738a81de7.slice/crio-5d9859559184ea58de15d7538f1fc9fbf741f0171878494aaa7aa98c1be7120d WatchSource:0}: Error finding container 5d9859559184ea58de15d7538f1fc9fbf741f0171878494aaa7aa98c1be7120d: Status 404 returned error can't find the container with id 5d9859559184ea58de15d7538f1fc9fbf741f0171878494aaa7aa98c1be7120d Jul 21 21:06:54.286668 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:06:54.286604 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4v97t9" event={"ID":"4573f5c2-4d6e-4c0d-a850-55b738a81de7","Type":"ContainerStarted","Data":"5d9859559184ea58de15d7538f1fc9fbf741f0171878494aaa7aa98c1be7120d"} Jul 21 21:06:56.733858 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:06:56.733815 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 21 21:06:56.734250 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:06:56.733908 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 21 21:06:56.734250 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:06:56.733959 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 21 21:06:57.303615 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:06:57.303573 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4v97t9" event={"ID":"4573f5c2-4d6e-4c0d-a850-55b738a81de7","Type":"ContainerStarted","Data":"d7f4daebc59cafeb5711b0e26a4fcd6f28c7ea60cc162a11dd2e7dbfb894278e"} Jul 21 21:06:57.325674 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:06:57.325614 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4v97t9" podStartSLOduration=2.018606878 podStartE2EDuration="5.325594197s" totalImagesPullingTime="3.306987319s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 21:06:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 21:06:57.324221364 +0000 UTC m=+375.258971042" watchObservedRunningTime="2026-07-21 21:06:57.325594197 +0000 UTC m=+375.260343864" Jul 21 21:06:58.263214 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:06:58.263162 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4v97t9" Jul 21 21:06:58.268130 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:06:58.268100 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4v97t9" Jul 21 21:06:58.308622 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:06:58.308588 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4v97t9" Jul 21 21:06:58.309479 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:06:58.309460 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4v97t9" Jul 21 21:07:35.746606 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:07:35.746571 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-9tv8c"] Jul 21 21:07:35.749663 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:07:35.749643 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-9tv8c" Jul 21 21:07:35.752368 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:07:35.752342 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-catalog-dockercfg-pwxrd\"" Jul 21 21:07:35.753243 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:07:35.753209 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Jul 21 21:07:35.753350 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:07:35.753263 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Jul 21 21:07:35.760736 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:07:35.760708 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-9tv8c"] Jul 21 21:07:35.869945 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:07:35.869907 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-87x69\" (UniqueName: \"kubernetes.io/projected/a3d39d6d-5038-41f2-bd14-1a5fc5a063d7-kube-api-access-87x69\") pod \"kuadrant-operator-catalog-9tv8c\" (UID: \"a3d39d6d-5038-41f2-bd14-1a5fc5a063d7\") " pod="kuadrant-system/kuadrant-operator-catalog-9tv8c" Jul 21 21:07:35.970947 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:07:35.970913 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-87x69\" (UniqueName: \"kubernetes.io/projected/a3d39d6d-5038-41f2-bd14-1a5fc5a063d7-kube-api-access-87x69\") pod \"kuadrant-operator-catalog-9tv8c\" (UID: \"a3d39d6d-5038-41f2-bd14-1a5fc5a063d7\") " pod="kuadrant-system/kuadrant-operator-catalog-9tv8c" Jul 21 21:07:35.978732 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:07:35.978708 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-87x69\" (UniqueName: \"kubernetes.io/projected/a3d39d6d-5038-41f2-bd14-1a5fc5a063d7-kube-api-access-87x69\") pod \"kuadrant-operator-catalog-9tv8c\" (UID: \"a3d39d6d-5038-41f2-bd14-1a5fc5a063d7\") " pod="kuadrant-system/kuadrant-operator-catalog-9tv8c" Jul 21 21:07:36.060769 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:07:36.060713 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-9tv8c" Jul 21 21:07:36.109941 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:07:36.109905 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-9tv8c"] Jul 21 21:07:36.187053 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:07:36.187027 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-9tv8c"] Jul 21 21:07:36.189516 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:07:36.189487 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda3d39d6d_5038_41f2_bd14_1a5fc5a063d7.slice/crio-3a318cf380f8ff825b1900aca83d72647a397ef794e706f3693c1e104d52cac9 WatchSource:0}: Error finding container 3a318cf380f8ff825b1900aca83d72647a397ef794e706f3693c1e104d52cac9: Status 404 returned error can't find the container with id 3a318cf380f8ff825b1900aca83d72647a397ef794e706f3693c1e104d52cac9 Jul 21 21:07:36.323617 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:07:36.323553 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-d4ms7"] Jul 21 21:07:36.327784 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:07:36.327766 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-d4ms7" Jul 21 21:07:36.334427 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:07:36.334395 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-d4ms7"] Jul 21 21:07:36.374280 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:07:36.374250 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rw92m\" (UniqueName: \"kubernetes.io/projected/a50b9c14-3c3f-47f9-bacd-3cbdc890fb93-kube-api-access-rw92m\") pod \"kuadrant-operator-catalog-d4ms7\" (UID: \"a50b9c14-3c3f-47f9-bacd-3cbdc890fb93\") " pod="kuadrant-system/kuadrant-operator-catalog-d4ms7" Jul 21 21:07:36.469280 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:07:36.469249 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-9tv8c" event={"ID":"a3d39d6d-5038-41f2-bd14-1a5fc5a063d7","Type":"ContainerStarted","Data":"3a318cf380f8ff825b1900aca83d72647a397ef794e706f3693c1e104d52cac9"} Jul 21 21:07:36.475170 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:07:36.475153 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rw92m\" (UniqueName: \"kubernetes.io/projected/a50b9c14-3c3f-47f9-bacd-3cbdc890fb93-kube-api-access-rw92m\") pod \"kuadrant-operator-catalog-d4ms7\" (UID: \"a50b9c14-3c3f-47f9-bacd-3cbdc890fb93\") " pod="kuadrant-system/kuadrant-operator-catalog-d4ms7" Jul 21 21:07:36.483197 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:07:36.483175 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rw92m\" (UniqueName: \"kubernetes.io/projected/a50b9c14-3c3f-47f9-bacd-3cbdc890fb93-kube-api-access-rw92m\") pod \"kuadrant-operator-catalog-d4ms7\" (UID: \"a50b9c14-3c3f-47f9-bacd-3cbdc890fb93\") " pod="kuadrant-system/kuadrant-operator-catalog-d4ms7" Jul 21 21:07:36.638674 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:07:36.638642 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-d4ms7" Jul 21 21:07:36.768242 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:07:36.768087 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-d4ms7"] Jul 21 21:07:36.770676 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:07:36.770651 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda50b9c14_3c3f_47f9_bacd_3cbdc890fb93.slice/crio-6ec697f6981b63cbd051e2e9e78dbec32d40306b3b393ddf5f36b207e31f9668 WatchSource:0}: Error finding container 6ec697f6981b63cbd051e2e9e78dbec32d40306b3b393ddf5f36b207e31f9668: Status 404 returned error can't find the container with id 6ec697f6981b63cbd051e2e9e78dbec32d40306b3b393ddf5f36b207e31f9668 Jul 21 21:07:37.477021 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:07:37.476982 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-d4ms7" event={"ID":"a50b9c14-3c3f-47f9-bacd-3cbdc890fb93","Type":"ContainerStarted","Data":"6ec697f6981b63cbd051e2e9e78dbec32d40306b3b393ddf5f36b207e31f9668"} Jul 21 21:07:39.485999 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:07:39.485958 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-9tv8c" event={"ID":"a3d39d6d-5038-41f2-bd14-1a5fc5a063d7","Type":"ContainerStarted","Data":"3bb1c5bb06cb63d036e9424f15e2158608b047d20e9507511551eb9fa26fa8dd"} Jul 21 21:07:39.486424 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:07:39.486007 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-catalog-9tv8c" podUID="a3d39d6d-5038-41f2-bd14-1a5fc5a063d7" containerName="registry-server" containerID="cri-o://3bb1c5bb06cb63d036e9424f15e2158608b047d20e9507511551eb9fa26fa8dd" gracePeriod=2 Jul 21 21:07:39.487386 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:07:39.487363 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-d4ms7" event={"ID":"a50b9c14-3c3f-47f9-bacd-3cbdc890fb93","Type":"ContainerStarted","Data":"f1bf70541699e7f52eb638c6154c636ed2c96c8ffe4a9c30956273e013c9c288"} Jul 21 21:07:39.503498 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:07:39.503458 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-catalog-9tv8c" podStartSLOduration=2.161735041 podStartE2EDuration="4.503445371s" totalImagesPullingTime="2.34171033s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 21:07:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 21:07:39.502386375 +0000 UTC m=+417.437136059" watchObservedRunningTime="2026-07-21 21:07:39.503445371 +0000 UTC m=+417.438195036" Jul 21 21:07:39.518932 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:07:39.518893 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-catalog-d4ms7" podStartSLOduration=1.754919176 podStartE2EDuration="3.518879942s" totalImagesPullingTime="1.763960766s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 21:07:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 21:07:39.517391694 +0000 UTC m=+417.452141372" watchObservedRunningTime="2026-07-21 21:07:39.518879942 +0000 UTC m=+417.453629608" Jul 21 21:07:39.727131 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:07:39.727105 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-9tv8c" Jul 21 21:07:39.806456 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:07:39.806391 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-87x69\" (UniqueName: \"kubernetes.io/projected/a3d39d6d-5038-41f2-bd14-1a5fc5a063d7-kube-api-access-87x69\") pod \"a3d39d6d-5038-41f2-bd14-1a5fc5a063d7\" (UID: \"a3d39d6d-5038-41f2-bd14-1a5fc5a063d7\") " Jul 21 21:07:39.808445 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:07:39.808418 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a3d39d6d-5038-41f2-bd14-1a5fc5a063d7-kube-api-access-87x69" (OuterVolumeSpecName: "kube-api-access-87x69") pod "a3d39d6d-5038-41f2-bd14-1a5fc5a063d7" (UID: "a3d39d6d-5038-41f2-bd14-1a5fc5a063d7"). InnerVolumeSpecName "kube-api-access-87x69". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 21 21:07:39.907751 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:07:39.907724 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-87x69\" (UniqueName: \"kubernetes.io/projected/a3d39d6d-5038-41f2-bd14-1a5fc5a063d7-kube-api-access-87x69\") on node \"ip-10-0-140-97.ec2.internal\" DevicePath \"\"" Jul 21 21:07:40.493138 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:07:40.493108 2575 generic.go:356] "Generic (PLEG): container finished" podID="a3d39d6d-5038-41f2-bd14-1a5fc5a063d7" containerID="3bb1c5bb06cb63d036e9424f15e2158608b047d20e9507511551eb9fa26fa8dd" exitCode=0 Jul 21 21:07:40.493582 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:07:40.493149 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-9tv8c" Jul 21 21:07:40.493582 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:07:40.493188 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-9tv8c" event={"ID":"a3d39d6d-5038-41f2-bd14-1a5fc5a063d7","Type":"ContainerDied","Data":"3bb1c5bb06cb63d036e9424f15e2158608b047d20e9507511551eb9fa26fa8dd"} Jul 21 21:07:40.493582 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:07:40.493234 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-9tv8c" event={"ID":"a3d39d6d-5038-41f2-bd14-1a5fc5a063d7","Type":"ContainerDied","Data":"3a318cf380f8ff825b1900aca83d72647a397ef794e706f3693c1e104d52cac9"} Jul 21 21:07:40.493582 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:07:40.493254 2575 scope.go:117] "RemoveContainer" containerID="3bb1c5bb06cb63d036e9424f15e2158608b047d20e9507511551eb9fa26fa8dd" Jul 21 21:07:40.502330 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:07:40.502310 2575 scope.go:117] "RemoveContainer" containerID="3bb1c5bb06cb63d036e9424f15e2158608b047d20e9507511551eb9fa26fa8dd" Jul 21 21:07:40.502593 ip-10-0-140-97 kubenswrapper[2575]: E0721 21:07:40.502574 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3bb1c5bb06cb63d036e9424f15e2158608b047d20e9507511551eb9fa26fa8dd\": container with ID starting with 3bb1c5bb06cb63d036e9424f15e2158608b047d20e9507511551eb9fa26fa8dd not found: ID does not exist" containerID="3bb1c5bb06cb63d036e9424f15e2158608b047d20e9507511551eb9fa26fa8dd" Jul 21 21:07:40.502666 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:07:40.502605 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3bb1c5bb06cb63d036e9424f15e2158608b047d20e9507511551eb9fa26fa8dd"} err="failed to get container status \"3bb1c5bb06cb63d036e9424f15e2158608b047d20e9507511551eb9fa26fa8dd\": rpc error: code = NotFound desc = could not find container \"3bb1c5bb06cb63d036e9424f15e2158608b047d20e9507511551eb9fa26fa8dd\": container with ID starting with 3bb1c5bb06cb63d036e9424f15e2158608b047d20e9507511551eb9fa26fa8dd not found: ID does not exist" Jul 21 21:07:40.513629 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:07:40.513603 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-9tv8c"] Jul 21 21:07:40.518352 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:07:40.518329 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-9tv8c"] Jul 21 21:07:40.739882 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:07:40.739850 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a3d39d6d-5038-41f2-bd14-1a5fc5a063d7" path="/var/lib/kubelet/pods/a3d39d6d-5038-41f2-bd14-1a5fc5a063d7/volumes" Jul 21 21:07:46.639276 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:07:46.639245 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kuadrant-system/kuadrant-operator-catalog-d4ms7" Jul 21 21:07:46.639812 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:07:46.639302 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-catalog-d4ms7" Jul 21 21:07:46.661776 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:07:46.661751 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kuadrant-system/kuadrant-operator-catalog-d4ms7" Jul 21 21:07:47.545073 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:07:47.545045 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-catalog-d4ms7" Jul 21 21:08:03.515784 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:08:03.515756 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-zvjq5"] Jul 21 21:08:03.516121 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:08:03.516066 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a3d39d6d-5038-41f2-bd14-1a5fc5a063d7" containerName="registry-server" Jul 21 21:08:03.516121 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:08:03.516077 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="a3d39d6d-5038-41f2-bd14-1a5fc5a063d7" containerName="registry-server" Jul 21 21:08:03.516188 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:08:03.516136 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="a3d39d6d-5038-41f2-bd14-1a5fc5a063d7" containerName="registry-server" Jul 21 21:08:03.520977 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:08:03.520953 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-zvjq5" Jul 21 21:08:03.523971 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:08:03.523949 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-dpnx5\"" Jul 21 21:08:03.529790 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:08:03.529769 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-zvjq5"] Jul 21 21:08:03.702649 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:08:03.702616 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pz2j2\" (UniqueName: \"kubernetes.io/projected/230bad68-0d05-442b-9c51-d7f525de37aa-kube-api-access-pz2j2\") pod \"kuadrant-operator-controller-manager-8684c9ff49-zvjq5\" (UID: \"230bad68-0d05-442b-9c51-d7f525de37aa\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-zvjq5" Jul 21 21:08:03.702834 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:08:03.702748 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/230bad68-0d05-442b-9c51-d7f525de37aa-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-zvjq5\" (UID: \"230bad68-0d05-442b-9c51-d7f525de37aa\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-zvjq5" Jul 21 21:08:03.803395 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:08:03.803306 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/230bad68-0d05-442b-9c51-d7f525de37aa-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-zvjq5\" (UID: \"230bad68-0d05-442b-9c51-d7f525de37aa\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-zvjq5" Jul 21 21:08:03.803395 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:08:03.803352 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pz2j2\" (UniqueName: \"kubernetes.io/projected/230bad68-0d05-442b-9c51-d7f525de37aa-kube-api-access-pz2j2\") pod \"kuadrant-operator-controller-manager-8684c9ff49-zvjq5\" (UID: \"230bad68-0d05-442b-9c51-d7f525de37aa\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-zvjq5" Jul 21 21:08:03.803806 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:08:03.803785 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/230bad68-0d05-442b-9c51-d7f525de37aa-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-zvjq5\" (UID: \"230bad68-0d05-442b-9c51-d7f525de37aa\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-zvjq5" Jul 21 21:08:03.812707 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:08:03.812687 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pz2j2\" (UniqueName: \"kubernetes.io/projected/230bad68-0d05-442b-9c51-d7f525de37aa-kube-api-access-pz2j2\") pod \"kuadrant-operator-controller-manager-8684c9ff49-zvjq5\" (UID: \"230bad68-0d05-442b-9c51-d7f525de37aa\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-zvjq5" Jul 21 21:08:03.832802 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:08:03.832774 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-zvjq5" Jul 21 21:08:03.963602 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:08:03.963559 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-zvjq5"] Jul 21 21:08:03.966155 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:08:03.966129 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod230bad68_0d05_442b_9c51_d7f525de37aa.slice/crio-36593aa8c18988c3702c3d23b071541aef876d4974c3d083e6416153062433c3 WatchSource:0}: Error finding container 36593aa8c18988c3702c3d23b071541aef876d4974c3d083e6416153062433c3: Status 404 returned error can't find the container with id 36593aa8c18988c3702c3d23b071541aef876d4974c3d083e6416153062433c3 Jul 21 21:08:04.592783 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:08:04.592712 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-zvjq5" event={"ID":"230bad68-0d05-442b-9c51-d7f525de37aa","Type":"ContainerStarted","Data":"36593aa8c18988c3702c3d23b071541aef876d4974c3d083e6416153062433c3"} Jul 21 21:08:08.057365 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:08:08.057328 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-b45rs"] Jul 21 21:08:08.059896 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:08:08.059870 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-75cc8d5cb7-b45rs" Jul 21 21:08:08.062460 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:08:08.062435 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-operator-dockercfg-srpqf\"" Jul 21 21:08:08.073315 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:08:08.073292 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-b45rs"] Jul 21 21:08:08.241595 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:08:08.241549 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8xdps\" (UniqueName: \"kubernetes.io/projected/296877a3-26aa-437f-9970-c5d99bfdbd94-kube-api-access-8xdps\") pod \"authorino-operator-75cc8d5cb7-b45rs\" (UID: \"296877a3-26aa-437f-9970-c5d99bfdbd94\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-b45rs" Jul 21 21:08:08.343081 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:08:08.342991 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8xdps\" (UniqueName: \"kubernetes.io/projected/296877a3-26aa-437f-9970-c5d99bfdbd94-kube-api-access-8xdps\") pod \"authorino-operator-75cc8d5cb7-b45rs\" (UID: \"296877a3-26aa-437f-9970-c5d99bfdbd94\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-b45rs" Jul 21 21:08:08.360366 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:08:08.360339 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8xdps\" (UniqueName: \"kubernetes.io/projected/296877a3-26aa-437f-9970-c5d99bfdbd94-kube-api-access-8xdps\") pod \"authorino-operator-75cc8d5cb7-b45rs\" (UID: \"296877a3-26aa-437f-9970-c5d99bfdbd94\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-b45rs" Jul 21 21:08:08.372971 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:08:08.372944 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-75cc8d5cb7-b45rs" Jul 21 21:08:09.410919 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:08:09.410895 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-b45rs"] Jul 21 21:08:09.412936 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:08:09.412912 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod296877a3_26aa_437f_9970_c5d99bfdbd94.slice/crio-b1f257999d9213806991c4ca3325550f71f86c206725e1689af9045de163d423 WatchSource:0}: Error finding container b1f257999d9213806991c4ca3325550f71f86c206725e1689af9045de163d423: Status 404 returned error can't find the container with id b1f257999d9213806991c4ca3325550f71f86c206725e1689af9045de163d423 Jul 21 21:08:09.618825 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:08:09.618781 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-zvjq5" event={"ID":"230bad68-0d05-442b-9c51-d7f525de37aa","Type":"ContainerStarted","Data":"68c5e511b9420297df38068010632973ad0b8fa235b79dda2d3ddc9d18429abc"} Jul 21 21:08:09.619003 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:08:09.618888 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-zvjq5" Jul 21 21:08:09.620358 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:08:09.620320 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-75cc8d5cb7-b45rs" event={"ID":"296877a3-26aa-437f-9970-c5d99bfdbd94","Type":"ContainerStarted","Data":"b1f257999d9213806991c4ca3325550f71f86c206725e1689af9045de163d423"} Jul 21 21:08:09.649323 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:08:09.649269 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-zvjq5" podStartSLOduration=1.300070414 podStartE2EDuration="6.649249869s" totalImagesPullingTime="5.349179455s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 21:08:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 21:08:09.648001544 +0000 UTC m=+447.582751256" watchObservedRunningTime="2026-07-21 21:08:09.649249869 +0000 UTC m=+447.583999535" Jul 21 21:08:12.637449 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:08:12.637411 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-75cc8d5cb7-b45rs" event={"ID":"296877a3-26aa-437f-9970-c5d99bfdbd94","Type":"ContainerStarted","Data":"d1d3b0a9c34137b86891fa943a6cc405dff043672a9bd6a9cc6f75f38f7cd34e"} Jul 21 21:08:12.637877 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:08:12.637558 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/authorino-operator-75cc8d5cb7-b45rs" Jul 21 21:08:12.656660 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:08:12.656619 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-operator-75cc8d5cb7-b45rs" podStartSLOduration=2.496584616 podStartE2EDuration="4.656604488s" totalImagesPullingTime="2.160019872s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 21:08:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 21:08:12.6547547 +0000 UTC m=+450.589504377" watchObservedRunningTime="2026-07-21 21:08:12.656604488 +0000 UTC m=+450.591354150" Jul 21 21:08:20.627157 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:08:20.627129 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-zvjq5" Jul 21 21:08:22.437712 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:08:22.437673 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-bsdm4"] Jul 21 21:08:22.442134 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:08:22.442110 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-bsdm4" Jul 21 21:08:22.458060 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:08:22.458031 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-bsdm4"] Jul 21 21:08:22.499946 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:08:22.499919 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-zvjq5"] Jul 21 21:08:22.500085 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:08:22.500068 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-zvjq5" podUID="230bad68-0d05-442b-9c51-d7f525de37aa" containerName="manager" containerID="cri-o://68c5e511b9420297df38068010632973ad0b8fa235b79dda2d3ddc9d18429abc" gracePeriod=2 Jul 21 21:08:22.510630 ip-10-0-140-97 kubenswrapper[2575]: E0721 21:08:22.509874 2575 status_manager.go:1041] "Failed to update status for pod" err="failed to patch status \"{\\\"metadata\\\":{\\\"uid\\\":\\\"230bad68-0d05-442b-9c51-d7f525de37aa\\\"},\\\"status\\\":{\\\"$setElementOrder/conditions\\\":[{\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"type\\\":\\\"Initialized\\\"},{\\\"type\\\":\\\"Ready\\\"},{\\\"type\\\":\\\"ContainersReady\\\"},{\\\"type\\\":\\\"PodScheduled\\\"}],\\\"conditions\\\":[{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Initialized\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Ready\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"ContainersReady\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodScheduled\\\"}],\\\"observedGeneration\\\":2}}\" for pod \"kuadrant-system\"/\"kuadrant-operator-controller-manager-8684c9ff49-zvjq5\": pods \"kuadrant-operator-controller-manager-8684c9ff49-zvjq5\" not found" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-zvjq5" Jul 21 21:08:22.515122 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:08:22.515085 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-zvjq5"] Jul 21 21:08:22.522066 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:08:22.522033 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-bsdm4"] Jul 21 21:08:22.522391 ip-10-0-140-97 kubenswrapper[2575]: E0721 21:08:22.522366 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[extensions-socket-volume kube-api-access-k9lhr], unattached volumes=[], failed to process volumes=[extensions-socket-volume kube-api-access-k9lhr]: context canceled" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-bsdm4" podUID="8101afa1-97e4-42f6-9a75-e91b076eec9e" Jul 21 21:08:22.532093 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:08:22.532069 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-689nk"] Jul 21 21:08:22.532370 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:08:22.532356 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="230bad68-0d05-442b-9c51-d7f525de37aa" containerName="manager" Jul 21 21:08:22.532370 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:08:22.532370 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="230bad68-0d05-442b-9c51-d7f525de37aa" containerName="manager" Jul 21 21:08:22.532498 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:08:22.532437 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="230bad68-0d05-442b-9c51-d7f525de37aa" containerName="manager" Jul 21 21:08:22.534582 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:08:22.534564 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-689nk" Jul 21 21:08:22.539481 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:08:22.539457 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-bsdm4"] Jul 21 21:08:22.548649 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:08:22.548618 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-689nk"] Jul 21 21:08:22.550796 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:08:22.550777 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-6gqzm"] Jul 21 21:08:22.553616 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:08:22.553591 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-6gqzm" Jul 21 21:08:22.565038 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:08:22.565011 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k9lhr\" (UniqueName: \"kubernetes.io/projected/8101afa1-97e4-42f6-9a75-e91b076eec9e-kube-api-access-k9lhr\") pod \"kuadrant-operator-controller-manager-86878c769f-bsdm4\" (UID: \"8101afa1-97e4-42f6-9a75-e91b076eec9e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-bsdm4" Jul 21 21:08:22.565268 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:08:22.565232 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/b9db7195-53bd-48ac-8cdb-8683fb86b717-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-689nk\" (UID: \"b9db7195-53bd-48ac-8cdb-8683fb86b717\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-689nk" Jul 21 21:08:22.565385 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:08:22.565276 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2bv99\" (UniqueName: \"kubernetes.io/projected/b9db7195-53bd-48ac-8cdb-8683fb86b717-kube-api-access-2bv99\") pod \"kuadrant-operator-controller-manager-8684c9ff49-689nk\" (UID: \"b9db7195-53bd-48ac-8cdb-8683fb86b717\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-689nk" Jul 21 21:08:22.565441 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:08:22.565381 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/8101afa1-97e4-42f6-9a75-e91b076eec9e-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-bsdm4\" (UID: \"8101afa1-97e4-42f6-9a75-e91b076eec9e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-bsdm4" Jul 21 21:08:22.567399 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:08:22.567373 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-6gqzm"] Jul 21 21:08:22.580991 ip-10-0-140-97 kubenswrapper[2575]: E0721 21:08:22.580954 2575 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-zvjq5\" is forbidden: User \"system:node:ip-10-0-140-97.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-140-97.ec2.internal' and this object" podUID="230bad68-0d05-442b-9c51-d7f525de37aa" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-zvjq5" Jul 21 21:08:22.618319 ip-10-0-140-97 kubenswrapper[2575]: E0721 21:08:22.618284 2575 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-zvjq5\" is forbidden: User \"system:node:ip-10-0-140-97.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-140-97.ec2.internal' and this object" podUID="230bad68-0d05-442b-9c51-d7f525de37aa" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-zvjq5" Jul 21 21:08:22.666409 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:08:22.666375 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/8a7d2814-c13b-4fc4-bd0d-9739d5bab9d4-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-6gqzm\" (UID: \"8a7d2814-c13b-4fc4-bd0d-9739d5bab9d4\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-6gqzm" Jul 21 21:08:22.666409 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:08:22.666412 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wbc6m\" (UniqueName: \"kubernetes.io/projected/8a7d2814-c13b-4fc4-bd0d-9739d5bab9d4-kube-api-access-wbc6m\") pod \"kuadrant-operator-controller-manager-86878c769f-6gqzm\" (UID: \"8a7d2814-c13b-4fc4-bd0d-9739d5bab9d4\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-6gqzm" Jul 21 21:08:22.666655 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:08:22.666524 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/8101afa1-97e4-42f6-9a75-e91b076eec9e-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-bsdm4\" (UID: \"8101afa1-97e4-42f6-9a75-e91b076eec9e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-bsdm4" Jul 21 21:08:22.666655 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:08:22.666613 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k9lhr\" (UniqueName: \"kubernetes.io/projected/8101afa1-97e4-42f6-9a75-e91b076eec9e-kube-api-access-k9lhr\") pod \"kuadrant-operator-controller-manager-86878c769f-bsdm4\" (UID: \"8101afa1-97e4-42f6-9a75-e91b076eec9e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-bsdm4" Jul 21 21:08:22.666756 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:08:22.666666 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/b9db7195-53bd-48ac-8cdb-8683fb86b717-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-689nk\" (UID: \"b9db7195-53bd-48ac-8cdb-8683fb86b717\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-689nk" Jul 21 21:08:22.666854 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:08:22.666830 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2bv99\" (UniqueName: \"kubernetes.io/projected/b9db7195-53bd-48ac-8cdb-8683fb86b717-kube-api-access-2bv99\") pod \"kuadrant-operator-controller-manager-8684c9ff49-689nk\" (UID: \"b9db7195-53bd-48ac-8cdb-8683fb86b717\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-689nk" Jul 21 21:08:22.666942 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:08:22.666871 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/8101afa1-97e4-42f6-9a75-e91b076eec9e-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-bsdm4\" (UID: \"8101afa1-97e4-42f6-9a75-e91b076eec9e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-bsdm4" Jul 21 21:08:22.667052 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:08:22.667034 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/b9db7195-53bd-48ac-8cdb-8683fb86b717-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-689nk\" (UID: \"b9db7195-53bd-48ac-8cdb-8683fb86b717\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-689nk" Jul 21 21:08:22.671468 ip-10-0-140-97 kubenswrapper[2575]: E0721 21:08:22.671451 2575 projected.go:196] Error preparing data for projected volume kube-api-access-k9lhr for pod kuadrant-system/kuadrant-operator-controller-manager-86878c769f-bsdm4: failed to fetch token: pod "kuadrant-operator-controller-manager-86878c769f-bsdm4" not found Jul 21 21:08:22.671548 ip-10-0-140-97 kubenswrapper[2575]: E0721 21:08:22.671517 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8101afa1-97e4-42f6-9a75-e91b076eec9e-kube-api-access-k9lhr podName:8101afa1-97e4-42f6-9a75-e91b076eec9e nodeName:}" failed. No retries permitted until 2026-07-21 21:08:23.171496986 +0000 UTC m=+461.106246635 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-k9lhr" (UniqueName: "kubernetes.io/projected/8101afa1-97e4-42f6-9a75-e91b076eec9e-kube-api-access-k9lhr") pod "kuadrant-operator-controller-manager-86878c769f-bsdm4" (UID: "8101afa1-97e4-42f6-9a75-e91b076eec9e") : failed to fetch token: pod "kuadrant-operator-controller-manager-86878c769f-bsdm4" not found Jul 21 21:08:22.675206 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:08:22.675185 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2bv99\" (UniqueName: \"kubernetes.io/projected/b9db7195-53bd-48ac-8cdb-8683fb86b717-kube-api-access-2bv99\") pod \"kuadrant-operator-controller-manager-8684c9ff49-689nk\" (UID: \"b9db7195-53bd-48ac-8cdb-8683fb86b717\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-689nk" Jul 21 21:08:22.683633 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:08:22.683608 2575 generic.go:356] "Generic (PLEG): container finished" podID="230bad68-0d05-442b-9c51-d7f525de37aa" containerID="68c5e511b9420297df38068010632973ad0b8fa235b79dda2d3ddc9d18429abc" exitCode=0 Jul 21 21:08:22.683738 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:08:22.683680 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-bsdm4" Jul 21 21:08:22.686157 ip-10-0-140-97 kubenswrapper[2575]: E0721 21:08:22.686073 2575 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-zvjq5\" is forbidden: User \"system:node:ip-10-0-140-97.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-140-97.ec2.internal' and this object" podUID="230bad68-0d05-442b-9c51-d7f525de37aa" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-zvjq5" Jul 21 21:08:22.688153 ip-10-0-140-97 kubenswrapper[2575]: E0721 21:08:22.688088 2575 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-86878c769f-bsdm4\" is forbidden: User \"system:node:ip-10-0-140-97.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-140-97.ec2.internal' and this object" podUID="8101afa1-97e4-42f6-9a75-e91b076eec9e" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-bsdm4" Jul 21 21:08:22.691856 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:08:22.691840 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-bsdm4" Jul 21 21:08:22.693770 ip-10-0-140-97 kubenswrapper[2575]: E0721 21:08:22.693746 2575 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-zvjq5\" is forbidden: User \"system:node:ip-10-0-140-97.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-140-97.ec2.internal' and this object" podUID="230bad68-0d05-442b-9c51-d7f525de37aa" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-zvjq5" Jul 21 21:08:22.695623 ip-10-0-140-97 kubenswrapper[2575]: E0721 21:08:22.695597 2575 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-86878c769f-bsdm4\" is forbidden: User \"system:node:ip-10-0-140-97.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-140-97.ec2.internal' and this object" podUID="8101afa1-97e4-42f6-9a75-e91b076eec9e" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-bsdm4" Jul 21 21:08:22.739898 ip-10-0-140-97 kubenswrapper[2575]: E0721 21:08:22.739870 2575 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-zvjq5\" is forbidden: User \"system:node:ip-10-0-140-97.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-140-97.ec2.internal' and this object" podUID="230bad68-0d05-442b-9c51-d7f525de37aa" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-zvjq5" Jul 21 21:08:22.741659 ip-10-0-140-97 kubenswrapper[2575]: E0721 21:08:22.741628 2575 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-86878c769f-bsdm4\" is forbidden: User \"system:node:ip-10-0-140-97.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-140-97.ec2.internal' and this object" podUID="8101afa1-97e4-42f6-9a75-e91b076eec9e" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-bsdm4" Jul 21 21:08:22.749868 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:08:22.749844 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-zvjq5" Jul 21 21:08:22.768110 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:08:22.768082 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/8101afa1-97e4-42f6-9a75-e91b076eec9e-extensions-socket-volume\") pod \"8101afa1-97e4-42f6-9a75-e91b076eec9e\" (UID: \"8101afa1-97e4-42f6-9a75-e91b076eec9e\") " Jul 21 21:08:22.768226 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:08:22.768213 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wbc6m\" (UniqueName: \"kubernetes.io/projected/8a7d2814-c13b-4fc4-bd0d-9739d5bab9d4-kube-api-access-wbc6m\") pod \"kuadrant-operator-controller-manager-86878c769f-6gqzm\" (UID: \"8a7d2814-c13b-4fc4-bd0d-9739d5bab9d4\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-6gqzm" Jul 21 21:08:22.768316 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:08:22.768304 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/8a7d2814-c13b-4fc4-bd0d-9739d5bab9d4-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-6gqzm\" (UID: \"8a7d2814-c13b-4fc4-bd0d-9739d5bab9d4\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-6gqzm" Jul 21 21:08:22.768356 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:08:22.768334 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k9lhr\" (UniqueName: \"kubernetes.io/projected/8101afa1-97e4-42f6-9a75-e91b076eec9e-kube-api-access-k9lhr\") on node \"ip-10-0-140-97.ec2.internal\" DevicePath \"\"" Jul 21 21:08:22.768356 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:08:22.768301 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8101afa1-97e4-42f6-9a75-e91b076eec9e-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "8101afa1-97e4-42f6-9a75-e91b076eec9e" (UID: "8101afa1-97e4-42f6-9a75-e91b076eec9e"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 21:08:22.768608 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:08:22.768589 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/8a7d2814-c13b-4fc4-bd0d-9739d5bab9d4-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-6gqzm\" (UID: \"8a7d2814-c13b-4fc4-bd0d-9739d5bab9d4\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-6gqzm" Jul 21 21:08:22.776934 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:08:22.776915 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wbc6m\" (UniqueName: \"kubernetes.io/projected/8a7d2814-c13b-4fc4-bd0d-9739d5bab9d4-kube-api-access-wbc6m\") pod \"kuadrant-operator-controller-manager-86878c769f-6gqzm\" (UID: \"8a7d2814-c13b-4fc4-bd0d-9739d5bab9d4\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-6gqzm" Jul 21 21:08:22.868766 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:08:22.868746 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/230bad68-0d05-442b-9c51-d7f525de37aa-extensions-socket-volume\") pod \"230bad68-0d05-442b-9c51-d7f525de37aa\" (UID: \"230bad68-0d05-442b-9c51-d7f525de37aa\") " Jul 21 21:08:22.868875 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:08:22.868844 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pz2j2\" (UniqueName: \"kubernetes.io/projected/230bad68-0d05-442b-9c51-d7f525de37aa-kube-api-access-pz2j2\") pod \"230bad68-0d05-442b-9c51-d7f525de37aa\" (UID: \"230bad68-0d05-442b-9c51-d7f525de37aa\") " Jul 21 21:08:22.868959 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:08:22.868947 2575 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/8101afa1-97e4-42f6-9a75-e91b076eec9e-extensions-socket-volume\") on node \"ip-10-0-140-97.ec2.internal\" DevicePath \"\"" Jul 21 21:08:22.869201 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:08:22.869179 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/230bad68-0d05-442b-9c51-d7f525de37aa-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "230bad68-0d05-442b-9c51-d7f525de37aa" (UID: "230bad68-0d05-442b-9c51-d7f525de37aa"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 21:08:22.870736 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:08:22.870715 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/230bad68-0d05-442b-9c51-d7f525de37aa-kube-api-access-pz2j2" (OuterVolumeSpecName: "kube-api-access-pz2j2") pod "230bad68-0d05-442b-9c51-d7f525de37aa" (UID: "230bad68-0d05-442b-9c51-d7f525de37aa"). InnerVolumeSpecName "kube-api-access-pz2j2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 21 21:08:22.875870 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:08:22.875853 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-689nk" Jul 21 21:08:22.883851 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:08:22.883833 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-6gqzm" Jul 21 21:08:22.970073 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:08:22.970050 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pz2j2\" (UniqueName: \"kubernetes.io/projected/230bad68-0d05-442b-9c51-d7f525de37aa-kube-api-access-pz2j2\") on node \"ip-10-0-140-97.ec2.internal\" DevicePath \"\"" Jul 21 21:08:22.970136 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:08:22.970076 2575 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/230bad68-0d05-442b-9c51-d7f525de37aa-extensions-socket-volume\") on node \"ip-10-0-140-97.ec2.internal\" DevicePath \"\"" Jul 21 21:08:23.015241 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:08:23.015209 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-689nk"] Jul 21 21:08:23.017840 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:08:23.017815 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb9db7195_53bd_48ac_8cdb_8683fb86b717.slice/crio-8c78be8927916bd72269fb9312f05e8677fc1cc08dc5d689014c2973944fce65 WatchSource:0}: Error finding container 8c78be8927916bd72269fb9312f05e8677fc1cc08dc5d689014c2973944fce65: Status 404 returned error can't find the container with id 8c78be8927916bd72269fb9312f05e8677fc1cc08dc5d689014c2973944fce65 Jul 21 21:08:23.038305 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:08:23.038284 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-6gqzm"] Jul 21 21:08:23.047917 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:08:23.047893 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8a7d2814_c13b_4fc4_bd0d_9739d5bab9d4.slice/crio-f482372395c397141d268b341772b5e521db00c72c0c1e3d3f5e8e6e954f01c7 WatchSource:0}: Error finding container f482372395c397141d268b341772b5e521db00c72c0c1e3d3f5e8e6e954f01c7: Status 404 returned error can't find the container with id f482372395c397141d268b341772b5e521db00c72c0c1e3d3f5e8e6e954f01c7 Jul 21 21:08:23.643752 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:08:23.643724 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/authorino-operator-75cc8d5cb7-b45rs" Jul 21 21:08:23.693350 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:08:23.693323 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-zvjq5" Jul 21 21:08:23.693350 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:08:23.693345 2575 scope.go:117] "RemoveContainer" containerID="68c5e511b9420297df38068010632973ad0b8fa235b79dda2d3ddc9d18429abc" Jul 21 21:08:23.695909 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:08:23.695878 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-689nk" event={"ID":"b9db7195-53bd-48ac-8cdb-8683fb86b717","Type":"ContainerStarted","Data":"b0150c78de6ef96bf12292c8cdd692e42ec148570cf19d13cfbd596ec9b3d87f"} Jul 21 21:08:23.696037 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:08:23.695996 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-689nk" Jul 21 21:08:23.696037 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:08:23.696024 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-689nk" event={"ID":"b9db7195-53bd-48ac-8cdb-8683fb86b717","Type":"ContainerStarted","Data":"8c78be8927916bd72269fb9312f05e8677fc1cc08dc5d689014c2973944fce65"} Jul 21 21:08:23.697687 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:08:23.697660 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-6gqzm" event={"ID":"8a7d2814-c13b-4fc4-bd0d-9739d5bab9d4","Type":"ContainerStarted","Data":"62352d1abfb749c96b1e280536782f5bda0be32638ab3a78e1ec4af8f7ae2e01"} Jul 21 21:08:23.697853 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:08:23.697705 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-bsdm4" Jul 21 21:08:23.697853 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:08:23.697711 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-6gqzm" event={"ID":"8a7d2814-c13b-4fc4-bd0d-9739d5bab9d4","Type":"ContainerStarted","Data":"f482372395c397141d268b341772b5e521db00c72c0c1e3d3f5e8e6e954f01c7"} Jul 21 21:08:23.697853 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:08:23.697743 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-6gqzm" Jul 21 21:08:23.719037 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:08:23.718985 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-689nk" podStartSLOduration=1.718967546 podStartE2EDuration="1.718967546s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 21:08:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 21:08:23.7164166 +0000 UTC m=+461.651166279" watchObservedRunningTime="2026-07-21 21:08:23.718967546 +0000 UTC m=+461.653717213" Jul 21 21:08:23.734599 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:08:23.734560 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-6gqzm" podStartSLOduration=1.734547015 podStartE2EDuration="1.734547015s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 21:08:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 21:08:23.733877377 +0000 UTC m=+461.668627057" watchObservedRunningTime="2026-07-21 21:08:23.734547015 +0000 UTC m=+461.669296673" Jul 21 21:08:23.735721 ip-10-0-140-97 kubenswrapper[2575]: E0721 21:08:23.735689 2575 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-86878c769f-bsdm4\" is forbidden: User \"system:node:ip-10-0-140-97.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-140-97.ec2.internal' and this object" podUID="8101afa1-97e4-42f6-9a75-e91b076eec9e" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-bsdm4" Jul 21 21:08:24.744146 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:08:24.744100 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="230bad68-0d05-442b-9c51-d7f525de37aa" path="/var/lib/kubelet/pods/230bad68-0d05-442b-9c51-d7f525de37aa/volumes" Jul 21 21:08:24.744563 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:08:24.744477 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8101afa1-97e4-42f6-9a75-e91b076eec9e" path="/var/lib/kubelet/pods/8101afa1-97e4-42f6-9a75-e91b076eec9e/volumes" Jul 21 21:08:34.705299 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:08:34.705268 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-689nk" Jul 21 21:08:34.705739 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:08:34.705345 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-6gqzm" Jul 21 21:08:34.794073 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:08:34.794043 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-689nk"] Jul 21 21:08:34.794246 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:08:34.794221 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-689nk" podUID="b9db7195-53bd-48ac-8cdb-8683fb86b717" containerName="manager" containerID="cri-o://b0150c78de6ef96bf12292c8cdd692e42ec148570cf19d13cfbd596ec9b3d87f" gracePeriod=10 Jul 21 21:08:35.044507 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:08:35.044484 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-689nk" Jul 21 21:08:35.173858 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:08:35.173833 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2bv99\" (UniqueName: \"kubernetes.io/projected/b9db7195-53bd-48ac-8cdb-8683fb86b717-kube-api-access-2bv99\") pod \"b9db7195-53bd-48ac-8cdb-8683fb86b717\" (UID: \"b9db7195-53bd-48ac-8cdb-8683fb86b717\") " Jul 21 21:08:35.174003 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:08:35.173946 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/b9db7195-53bd-48ac-8cdb-8683fb86b717-extensions-socket-volume\") pod \"b9db7195-53bd-48ac-8cdb-8683fb86b717\" (UID: \"b9db7195-53bd-48ac-8cdb-8683fb86b717\") " Jul 21 21:08:35.174320 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:08:35.174296 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b9db7195-53bd-48ac-8cdb-8683fb86b717-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "b9db7195-53bd-48ac-8cdb-8683fb86b717" (UID: "b9db7195-53bd-48ac-8cdb-8683fb86b717"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 21:08:35.175730 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:08:35.175708 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b9db7195-53bd-48ac-8cdb-8683fb86b717-kube-api-access-2bv99" (OuterVolumeSpecName: "kube-api-access-2bv99") pod "b9db7195-53bd-48ac-8cdb-8683fb86b717" (UID: "b9db7195-53bd-48ac-8cdb-8683fb86b717"). InnerVolumeSpecName "kube-api-access-2bv99". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 21 21:08:35.274972 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:08:35.274900 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2bv99\" (UniqueName: \"kubernetes.io/projected/b9db7195-53bd-48ac-8cdb-8683fb86b717-kube-api-access-2bv99\") on node \"ip-10-0-140-97.ec2.internal\" DevicePath \"\"" Jul 21 21:08:35.274972 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:08:35.274927 2575 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/b9db7195-53bd-48ac-8cdb-8683fb86b717-extensions-socket-volume\") on node \"ip-10-0-140-97.ec2.internal\" DevicePath \"\"" Jul 21 21:08:35.760292 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:08:35.760259 2575 generic.go:356] "Generic (PLEG): container finished" podID="b9db7195-53bd-48ac-8cdb-8683fb86b717" containerID="b0150c78de6ef96bf12292c8cdd692e42ec148570cf19d13cfbd596ec9b3d87f" exitCode=0 Jul 21 21:08:35.760716 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:08:35.760311 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-689nk" Jul 21 21:08:35.760716 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:08:35.760321 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-689nk" event={"ID":"b9db7195-53bd-48ac-8cdb-8683fb86b717","Type":"ContainerDied","Data":"b0150c78de6ef96bf12292c8cdd692e42ec148570cf19d13cfbd596ec9b3d87f"} Jul 21 21:08:35.760716 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:08:35.760362 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-689nk" event={"ID":"b9db7195-53bd-48ac-8cdb-8683fb86b717","Type":"ContainerDied","Data":"8c78be8927916bd72269fb9312f05e8677fc1cc08dc5d689014c2973944fce65"} Jul 21 21:08:35.760716 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:08:35.760382 2575 scope.go:117] "RemoveContainer" containerID="b0150c78de6ef96bf12292c8cdd692e42ec148570cf19d13cfbd596ec9b3d87f" Jul 21 21:08:35.769041 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:08:35.769002 2575 scope.go:117] "RemoveContainer" containerID="b0150c78de6ef96bf12292c8cdd692e42ec148570cf19d13cfbd596ec9b3d87f" Jul 21 21:08:35.769266 ip-10-0-140-97 kubenswrapper[2575]: E0721 21:08:35.769249 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b0150c78de6ef96bf12292c8cdd692e42ec148570cf19d13cfbd596ec9b3d87f\": container with ID starting with b0150c78de6ef96bf12292c8cdd692e42ec148570cf19d13cfbd596ec9b3d87f not found: ID does not exist" containerID="b0150c78de6ef96bf12292c8cdd692e42ec148570cf19d13cfbd596ec9b3d87f" Jul 21 21:08:35.769310 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:08:35.769275 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b0150c78de6ef96bf12292c8cdd692e42ec148570cf19d13cfbd596ec9b3d87f"} err="failed to get container status \"b0150c78de6ef96bf12292c8cdd692e42ec148570cf19d13cfbd596ec9b3d87f\": rpc error: code = NotFound desc = could not find container \"b0150c78de6ef96bf12292c8cdd692e42ec148570cf19d13cfbd596ec9b3d87f\": container with ID starting with b0150c78de6ef96bf12292c8cdd692e42ec148570cf19d13cfbd596ec9b3d87f not found: ID does not exist" Jul 21 21:08:35.789328 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:08:35.789306 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-689nk"] Jul 21 21:08:35.793006 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:08:35.792987 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-689nk"] Jul 21 21:08:36.739797 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:08:36.739764 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b9db7195-53bd-48ac-8cdb-8683fb86b717" path="/var/lib/kubelet/pods/b9db7195-53bd-48ac-8cdb-8683fb86b717/volumes" Jul 21 21:08:51.200575 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:08:51.200523 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-2pnp6"] Jul 21 21:08:51.201134 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:08:51.201017 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b9db7195-53bd-48ac-8cdb-8683fb86b717" containerName="manager" Jul 21 21:08:51.201134 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:08:51.201047 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="b9db7195-53bd-48ac-8cdb-8683fb86b717" containerName="manager" Jul 21 21:08:51.201372 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:08:51.201163 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="b9db7195-53bd-48ac-8cdb-8683fb86b717" containerName="manager" Jul 21 21:08:51.204741 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:08:51.204718 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-2pnp6" Jul 21 21:08:51.207413 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:08:51.207395 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-jtk4j\"" Jul 21 21:08:51.217498 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:08:51.217477 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-2pnp6"] Jul 21 21:08:51.314626 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:08:51.314590 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sbd9t\" (UniqueName: \"kubernetes.io/projected/34d429d3-1614-43f5-997f-492caa021783-kube-api-access-sbd9t\") pod \"maas-default-gateway-openshift-default-56654dcf7-2pnp6\" (UID: \"34d429d3-1614-43f5-997f-492caa021783\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-2pnp6" Jul 21 21:08:51.314787 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:08:51.314631 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/34d429d3-1614-43f5-997f-492caa021783-istio-data\") pod \"maas-default-gateway-openshift-default-56654dcf7-2pnp6\" (UID: \"34d429d3-1614-43f5-997f-492caa021783\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-2pnp6" Jul 21 21:08:51.314787 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:08:51.314658 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/34d429d3-1614-43f5-997f-492caa021783-credential-socket\") pod \"maas-default-gateway-openshift-default-56654dcf7-2pnp6\" (UID: \"34d429d3-1614-43f5-997f-492caa021783\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-2pnp6" Jul 21 21:08:51.314787 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:08:51.314677 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/34d429d3-1614-43f5-997f-492caa021783-istio-envoy\") pod \"maas-default-gateway-openshift-default-56654dcf7-2pnp6\" (UID: \"34d429d3-1614-43f5-997f-492caa021783\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-2pnp6" Jul 21 21:08:51.314787 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:08:51.314695 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/34d429d3-1614-43f5-997f-492caa021783-workload-certs\") pod \"maas-default-gateway-openshift-default-56654dcf7-2pnp6\" (UID: \"34d429d3-1614-43f5-997f-492caa021783\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-2pnp6" Jul 21 21:08:51.314787 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:08:51.314754 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/34d429d3-1614-43f5-997f-492caa021783-istio-token\") pod \"maas-default-gateway-openshift-default-56654dcf7-2pnp6\" (UID: \"34d429d3-1614-43f5-997f-492caa021783\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-2pnp6" Jul 21 21:08:51.314963 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:08:51.314800 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/34d429d3-1614-43f5-997f-492caa021783-workload-socket\") pod \"maas-default-gateway-openshift-default-56654dcf7-2pnp6\" (UID: \"34d429d3-1614-43f5-997f-492caa021783\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-2pnp6" Jul 21 21:08:51.314963 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:08:51.314822 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/34d429d3-1614-43f5-997f-492caa021783-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-56654dcf7-2pnp6\" (UID: \"34d429d3-1614-43f5-997f-492caa021783\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-2pnp6" Jul 21 21:08:51.314963 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:08:51.314872 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/34d429d3-1614-43f5-997f-492caa021783-istio-podinfo\") pod \"maas-default-gateway-openshift-default-56654dcf7-2pnp6\" (UID: \"34d429d3-1614-43f5-997f-492caa021783\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-2pnp6" Jul 21 21:08:51.415474 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:08:51.415437 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/34d429d3-1614-43f5-997f-492caa021783-istio-podinfo\") pod \"maas-default-gateway-openshift-default-56654dcf7-2pnp6\" (UID: \"34d429d3-1614-43f5-997f-492caa021783\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-2pnp6" Jul 21 21:08:51.415666 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:08:51.415486 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sbd9t\" (UniqueName: \"kubernetes.io/projected/34d429d3-1614-43f5-997f-492caa021783-kube-api-access-sbd9t\") pod \"maas-default-gateway-openshift-default-56654dcf7-2pnp6\" (UID: \"34d429d3-1614-43f5-997f-492caa021783\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-2pnp6" Jul 21 21:08:51.415666 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:08:51.415515 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/34d429d3-1614-43f5-997f-492caa021783-istio-data\") pod \"maas-default-gateway-openshift-default-56654dcf7-2pnp6\" (UID: \"34d429d3-1614-43f5-997f-492caa021783\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-2pnp6" Jul 21 21:08:51.415749 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:08:51.415688 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/34d429d3-1614-43f5-997f-492caa021783-credential-socket\") pod \"maas-default-gateway-openshift-default-56654dcf7-2pnp6\" (UID: \"34d429d3-1614-43f5-997f-492caa021783\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-2pnp6" Jul 21 21:08:51.415749 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:08:51.415726 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/34d429d3-1614-43f5-997f-492caa021783-istio-envoy\") pod \"maas-default-gateway-openshift-default-56654dcf7-2pnp6\" (UID: \"34d429d3-1614-43f5-997f-492caa021783\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-2pnp6" Jul 21 21:08:51.415832 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:08:51.415761 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/34d429d3-1614-43f5-997f-492caa021783-workload-certs\") pod \"maas-default-gateway-openshift-default-56654dcf7-2pnp6\" (UID: \"34d429d3-1614-43f5-997f-492caa021783\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-2pnp6" Jul 21 21:08:51.415832 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:08:51.415807 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/34d429d3-1614-43f5-997f-492caa021783-istio-token\") pod \"maas-default-gateway-openshift-default-56654dcf7-2pnp6\" (UID: \"34d429d3-1614-43f5-997f-492caa021783\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-2pnp6" Jul 21 21:08:51.415907 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:08:51.415869 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/34d429d3-1614-43f5-997f-492caa021783-workload-socket\") pod \"maas-default-gateway-openshift-default-56654dcf7-2pnp6\" (UID: \"34d429d3-1614-43f5-997f-492caa021783\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-2pnp6" Jul 21 21:08:51.415907 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:08:51.415901 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/34d429d3-1614-43f5-997f-492caa021783-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-56654dcf7-2pnp6\" (UID: \"34d429d3-1614-43f5-997f-492caa021783\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-2pnp6" Jul 21 21:08:51.416001 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:08:51.415943 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/34d429d3-1614-43f5-997f-492caa021783-istio-data\") pod \"maas-default-gateway-openshift-default-56654dcf7-2pnp6\" (UID: \"34d429d3-1614-43f5-997f-492caa021783\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-2pnp6" Jul 21 21:08:51.416050 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:08:51.416016 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/34d429d3-1614-43f5-997f-492caa021783-credential-socket\") pod \"maas-default-gateway-openshift-default-56654dcf7-2pnp6\" (UID: \"34d429d3-1614-43f5-997f-492caa021783\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-2pnp6" Jul 21 21:08:51.416159 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:08:51.416134 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/34d429d3-1614-43f5-997f-492caa021783-workload-certs\") pod \"maas-default-gateway-openshift-default-56654dcf7-2pnp6\" (UID: \"34d429d3-1614-43f5-997f-492caa021783\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-2pnp6" Jul 21 21:08:51.416238 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:08:51.416162 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/34d429d3-1614-43f5-997f-492caa021783-workload-socket\") pod \"maas-default-gateway-openshift-default-56654dcf7-2pnp6\" (UID: \"34d429d3-1614-43f5-997f-492caa021783\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-2pnp6" Jul 21 21:08:51.416470 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:08:51.416453 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/34d429d3-1614-43f5-997f-492caa021783-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-56654dcf7-2pnp6\" (UID: \"34d429d3-1614-43f5-997f-492caa021783\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-2pnp6" Jul 21 21:08:51.417852 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:08:51.417827 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/34d429d3-1614-43f5-997f-492caa021783-istio-envoy\") pod \"maas-default-gateway-openshift-default-56654dcf7-2pnp6\" (UID: \"34d429d3-1614-43f5-997f-492caa021783\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-2pnp6" Jul 21 21:08:51.417931 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:08:51.417905 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/34d429d3-1614-43f5-997f-492caa021783-istio-podinfo\") pod \"maas-default-gateway-openshift-default-56654dcf7-2pnp6\" (UID: \"34d429d3-1614-43f5-997f-492caa021783\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-2pnp6" Jul 21 21:08:51.426372 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:08:51.426353 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/34d429d3-1614-43f5-997f-492caa021783-istio-token\") pod \"maas-default-gateway-openshift-default-56654dcf7-2pnp6\" (UID: \"34d429d3-1614-43f5-997f-492caa021783\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-2pnp6" Jul 21 21:08:51.427426 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:08:51.427408 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sbd9t\" (UniqueName: \"kubernetes.io/projected/34d429d3-1614-43f5-997f-492caa021783-kube-api-access-sbd9t\") pod \"maas-default-gateway-openshift-default-56654dcf7-2pnp6\" (UID: \"34d429d3-1614-43f5-997f-492caa021783\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-2pnp6" Jul 21 21:08:51.515424 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:08:51.515338 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-2pnp6" Jul 21 21:08:51.652409 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:08:51.652364 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-2pnp6"] Jul 21 21:08:51.656648 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:08:51.656619 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod34d429d3_1614_43f5_997f_492caa021783.slice/crio-14151e2e2b61e64cf40c360b1962d0411983693f588fb57a9567cc3f8c1193e7 WatchSource:0}: Error finding container 14151e2e2b61e64cf40c360b1962d0411983693f588fb57a9567cc3f8c1193e7: Status 404 returned error can't find the container with id 14151e2e2b61e64cf40c360b1962d0411983693f588fb57a9567cc3f8c1193e7 Jul 21 21:08:51.658491 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:08:51.658457 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 21 21:08:51.658602 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:08:51.658571 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 21 21:08:51.658668 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:08:51.658617 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 21 21:08:51.832134 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:08:51.832027 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-2pnp6" event={"ID":"34d429d3-1614-43f5-997f-492caa021783","Type":"ContainerStarted","Data":"8638249f201491393008c05255ba0cee2f38be7e58bb737b76b568f0079a293b"} Jul 21 21:08:51.832134 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:08:51.832093 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-2pnp6" event={"ID":"34d429d3-1614-43f5-997f-492caa021783","Type":"ContainerStarted","Data":"14151e2e2b61e64cf40c360b1962d0411983693f588fb57a9567cc3f8c1193e7"} Jul 21 21:08:51.857350 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:08:51.857303 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-2pnp6" podStartSLOduration=0.857287429 podStartE2EDuration="857.287429ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 21:08:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 21:08:51.855168918 +0000 UTC m=+489.789918585" watchObservedRunningTime="2026-07-21 21:08:51.857287429 +0000 UTC m=+489.792037095" Jul 21 21:08:52.516116 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:08:52.516086 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-2pnp6" Jul 21 21:08:52.520676 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:08:52.520654 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-2pnp6" Jul 21 21:08:52.836211 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:08:52.836133 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-2pnp6" Jul 21 21:08:52.837046 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:08:52.837018 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-2pnp6" Jul 21 21:08:53.810956 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:08:53.810930 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-75d556bd77-6q64t"] Jul 21 21:08:53.813240 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:08:53.813221 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-6q64t" Jul 21 21:08:53.815560 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:08:53.815519 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-nkn2q\"" Jul 21 21:08:53.822640 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:08:53.822616 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-75d556bd77-6q64t"] Jul 21 21:08:53.937944 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:08:53.937906 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jk867\" (UniqueName: \"kubernetes.io/projected/fc71117e-c61e-4180-a106-cb6f06bf023e-kube-api-access-jk867\") pod \"authorino-75d556bd77-6q64t\" (UID: \"fc71117e-c61e-4180-a106-cb6f06bf023e\") " pod="kuadrant-system/authorino-75d556bd77-6q64t" Jul 21 21:08:54.039104 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:08:54.039062 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jk867\" (UniqueName: \"kubernetes.io/projected/fc71117e-c61e-4180-a106-cb6f06bf023e-kube-api-access-jk867\") pod \"authorino-75d556bd77-6q64t\" (UID: \"fc71117e-c61e-4180-a106-cb6f06bf023e\") " pod="kuadrant-system/authorino-75d556bd77-6q64t" Jul 21 21:08:54.047863 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:08:54.047838 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jk867\" (UniqueName: \"kubernetes.io/projected/fc71117e-c61e-4180-a106-cb6f06bf023e-kube-api-access-jk867\") pod \"authorino-75d556bd77-6q64t\" (UID: \"fc71117e-c61e-4180-a106-cb6f06bf023e\") " pod="kuadrant-system/authorino-75d556bd77-6q64t" Jul 21 21:08:54.123453 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:08:54.123422 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-6q64t" Jul 21 21:08:54.253249 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:08:54.253224 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-75d556bd77-6q64t"] Jul 21 21:08:54.255655 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:08:54.255629 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podfc71117e_c61e_4180_a106_cb6f06bf023e.slice/crio-d6fadf469de9892bfa1bb70b7fd721a82bddc8dbe4d0eab45a5035ff1058dc90 WatchSource:0}: Error finding container d6fadf469de9892bfa1bb70b7fd721a82bddc8dbe4d0eab45a5035ff1058dc90: Status 404 returned error can't find the container with id d6fadf469de9892bfa1bb70b7fd721a82bddc8dbe4d0eab45a5035ff1058dc90 Jul 21 21:08:54.846672 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:08:54.846631 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-6q64t" event={"ID":"fc71117e-c61e-4180-a106-cb6f06bf023e","Type":"ContainerStarted","Data":"d6fadf469de9892bfa1bb70b7fd721a82bddc8dbe4d0eab45a5035ff1058dc90"} Jul 21 21:08:57.863719 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:08:57.863683 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-6q64t" event={"ID":"fc71117e-c61e-4180-a106-cb6f06bf023e","Type":"ContainerStarted","Data":"f5953411135ce4cc2975a24cb3a020dd2ddb62e9a53e87217016f8be2b26c15d"} Jul 21 21:08:57.880473 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:08:57.880427 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-75d556bd77-6q64t" podStartSLOduration=1.8408048510000001 podStartE2EDuration="4.880414521s" totalImagesPullingTime="3.03960967s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 21:08:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 21:08:57.87968817 +0000 UTC m=+495.814437836" watchObservedRunningTime="2026-07-21 21:08:57.880414521 +0000 UTC m=+495.815164186" Jul 21 21:09:01.064278 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:09:01.064249 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-6675f745fd-fswbm"] Jul 21 21:09:01.066694 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:09:01.066673 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6675f745fd-fswbm" Jul 21 21:09:01.069492 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:09:01.069470 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Jul 21 21:09:01.069774 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:09:01.069758 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Jul 21 21:09:01.076273 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:09:01.076254 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Jul 21 21:09:01.076507 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:09:01.076494 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-fz5rw\"" Jul 21 21:09:01.076892 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:09:01.076870 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Jul 21 21:09:01.076984 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:09:01.076962 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Jul 21 21:09:01.079889 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:09:01.079872 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Jul 21 21:09:01.080643 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:09:01.080627 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6675f745fd-fswbm"] Jul 21 21:09:01.092288 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:09:01.092265 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/cef40b45-cb9f-4e56-adaa-f7be7e5f7c2d-console-config\") pod \"console-6675f745fd-fswbm\" (UID: \"cef40b45-cb9f-4e56-adaa-f7be7e5f7c2d\") " pod="openshift-console/console-6675f745fd-fswbm" Jul 21 21:09:01.092373 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:09:01.092313 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d6jkh\" (UniqueName: \"kubernetes.io/projected/cef40b45-cb9f-4e56-adaa-f7be7e5f7c2d-kube-api-access-d6jkh\") pod \"console-6675f745fd-fswbm\" (UID: \"cef40b45-cb9f-4e56-adaa-f7be7e5f7c2d\") " pod="openshift-console/console-6675f745fd-fswbm" Jul 21 21:09:01.092373 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:09:01.092343 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/cef40b45-cb9f-4e56-adaa-f7be7e5f7c2d-service-ca\") pod \"console-6675f745fd-fswbm\" (UID: \"cef40b45-cb9f-4e56-adaa-f7be7e5f7c2d\") " pod="openshift-console/console-6675f745fd-fswbm" Jul 21 21:09:01.092445 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:09:01.092381 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/cef40b45-cb9f-4e56-adaa-f7be7e5f7c2d-console-oauth-config\") pod \"console-6675f745fd-fswbm\" (UID: \"cef40b45-cb9f-4e56-adaa-f7be7e5f7c2d\") " pod="openshift-console/console-6675f745fd-fswbm" Jul 21 21:09:01.092478 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:09:01.092442 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/cef40b45-cb9f-4e56-adaa-f7be7e5f7c2d-oauth-serving-cert\") pod \"console-6675f745fd-fswbm\" (UID: \"cef40b45-cb9f-4e56-adaa-f7be7e5f7c2d\") " pod="openshift-console/console-6675f745fd-fswbm" Jul 21 21:09:01.092478 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:09:01.092467 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cef40b45-cb9f-4e56-adaa-f7be7e5f7c2d-trusted-ca-bundle\") pod \"console-6675f745fd-fswbm\" (UID: \"cef40b45-cb9f-4e56-adaa-f7be7e5f7c2d\") " pod="openshift-console/console-6675f745fd-fswbm" Jul 21 21:09:01.092564 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:09:01.092494 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/cef40b45-cb9f-4e56-adaa-f7be7e5f7c2d-console-serving-cert\") pod \"console-6675f745fd-fswbm\" (UID: \"cef40b45-cb9f-4e56-adaa-f7be7e5f7c2d\") " pod="openshift-console/console-6675f745fd-fswbm" Jul 21 21:09:01.193294 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:09:01.193261 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/cef40b45-cb9f-4e56-adaa-f7be7e5f7c2d-console-config\") pod \"console-6675f745fd-fswbm\" (UID: \"cef40b45-cb9f-4e56-adaa-f7be7e5f7c2d\") " pod="openshift-console/console-6675f745fd-fswbm" Jul 21 21:09:01.193439 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:09:01.193307 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d6jkh\" (UniqueName: \"kubernetes.io/projected/cef40b45-cb9f-4e56-adaa-f7be7e5f7c2d-kube-api-access-d6jkh\") pod \"console-6675f745fd-fswbm\" (UID: \"cef40b45-cb9f-4e56-adaa-f7be7e5f7c2d\") " pod="openshift-console/console-6675f745fd-fswbm" Jul 21 21:09:01.193439 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:09:01.193328 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/cef40b45-cb9f-4e56-adaa-f7be7e5f7c2d-service-ca\") pod \"console-6675f745fd-fswbm\" (UID: \"cef40b45-cb9f-4e56-adaa-f7be7e5f7c2d\") " pod="openshift-console/console-6675f745fd-fswbm" Jul 21 21:09:01.193439 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:09:01.193346 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/cef40b45-cb9f-4e56-adaa-f7be7e5f7c2d-console-oauth-config\") pod \"console-6675f745fd-fswbm\" (UID: \"cef40b45-cb9f-4e56-adaa-f7be7e5f7c2d\") " pod="openshift-console/console-6675f745fd-fswbm" Jul 21 21:09:01.193439 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:09:01.193367 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/cef40b45-cb9f-4e56-adaa-f7be7e5f7c2d-oauth-serving-cert\") pod \"console-6675f745fd-fswbm\" (UID: \"cef40b45-cb9f-4e56-adaa-f7be7e5f7c2d\") " pod="openshift-console/console-6675f745fd-fswbm" Jul 21 21:09:01.193439 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:09:01.193385 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cef40b45-cb9f-4e56-adaa-f7be7e5f7c2d-trusted-ca-bundle\") pod \"console-6675f745fd-fswbm\" (UID: \"cef40b45-cb9f-4e56-adaa-f7be7e5f7c2d\") " pod="openshift-console/console-6675f745fd-fswbm" Jul 21 21:09:01.193439 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:09:01.193410 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/cef40b45-cb9f-4e56-adaa-f7be7e5f7c2d-console-serving-cert\") pod \"console-6675f745fd-fswbm\" (UID: \"cef40b45-cb9f-4e56-adaa-f7be7e5f7c2d\") " pod="openshift-console/console-6675f745fd-fswbm" Jul 21 21:09:01.194197 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:09:01.194168 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/cef40b45-cb9f-4e56-adaa-f7be7e5f7c2d-service-ca\") pod \"console-6675f745fd-fswbm\" (UID: \"cef40b45-cb9f-4e56-adaa-f7be7e5f7c2d\") " pod="openshift-console/console-6675f745fd-fswbm" Jul 21 21:09:01.194197 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:09:01.194189 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/cef40b45-cb9f-4e56-adaa-f7be7e5f7c2d-oauth-serving-cert\") pod \"console-6675f745fd-fswbm\" (UID: \"cef40b45-cb9f-4e56-adaa-f7be7e5f7c2d\") " pod="openshift-console/console-6675f745fd-fswbm" Jul 21 21:09:01.194351 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:09:01.194189 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/cef40b45-cb9f-4e56-adaa-f7be7e5f7c2d-console-config\") pod \"console-6675f745fd-fswbm\" (UID: \"cef40b45-cb9f-4e56-adaa-f7be7e5f7c2d\") " pod="openshift-console/console-6675f745fd-fswbm" Jul 21 21:09:01.194351 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:09:01.194254 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cef40b45-cb9f-4e56-adaa-f7be7e5f7c2d-trusted-ca-bundle\") pod \"console-6675f745fd-fswbm\" (UID: \"cef40b45-cb9f-4e56-adaa-f7be7e5f7c2d\") " pod="openshift-console/console-6675f745fd-fswbm" Jul 21 21:09:01.195806 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:09:01.195784 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/cef40b45-cb9f-4e56-adaa-f7be7e5f7c2d-console-oauth-config\") pod \"console-6675f745fd-fswbm\" (UID: \"cef40b45-cb9f-4e56-adaa-f7be7e5f7c2d\") " pod="openshift-console/console-6675f745fd-fswbm" Jul 21 21:09:01.195898 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:09:01.195881 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/cef40b45-cb9f-4e56-adaa-f7be7e5f7c2d-console-serving-cert\") pod \"console-6675f745fd-fswbm\" (UID: \"cef40b45-cb9f-4e56-adaa-f7be7e5f7c2d\") " pod="openshift-console/console-6675f745fd-fswbm" Jul 21 21:09:01.201024 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:09:01.200996 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d6jkh\" (UniqueName: \"kubernetes.io/projected/cef40b45-cb9f-4e56-adaa-f7be7e5f7c2d-kube-api-access-d6jkh\") pod \"console-6675f745fd-fswbm\" (UID: \"cef40b45-cb9f-4e56-adaa-f7be7e5f7c2d\") " pod="openshift-console/console-6675f745fd-fswbm" Jul 21 21:09:01.375905 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:09:01.375874 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6675f745fd-fswbm" Jul 21 21:09:01.503917 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:09:01.503885 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6675f745fd-fswbm"] Jul 21 21:09:01.506808 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:09:01.506783 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcef40b45_cb9f_4e56_adaa_f7be7e5f7c2d.slice/crio-8b6bbf54a4a9a90b2e4fbb46de09400246f7b6335564cbd8b2612d87de2f989e WatchSource:0}: Error finding container 8b6bbf54a4a9a90b2e4fbb46de09400246f7b6335564cbd8b2612d87de2f989e: Status 404 returned error can't find the container with id 8b6bbf54a4a9a90b2e4fbb46de09400246f7b6335564cbd8b2612d87de2f989e Jul 21 21:09:01.880899 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:09:01.880864 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6675f745fd-fswbm" event={"ID":"cef40b45-cb9f-4e56-adaa-f7be7e5f7c2d","Type":"ContainerStarted","Data":"45b8275815a929becf463c2a4e6a62c5dd060057feb74b28265067b83abc2d78"} Jul 21 21:09:01.881081 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:09:01.880912 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6675f745fd-fswbm" event={"ID":"cef40b45-cb9f-4e56-adaa-f7be7e5f7c2d","Type":"ContainerStarted","Data":"8b6bbf54a4a9a90b2e4fbb46de09400246f7b6335564cbd8b2612d87de2f989e"} Jul 21 21:09:01.900058 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:09:01.900013 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-6675f745fd-fswbm" podStartSLOduration=0.899999745 podStartE2EDuration="899.999745ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 21:09:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 21:09:01.899418982 +0000 UTC m=+499.834168644" watchObservedRunningTime="2026-07-21 21:09:01.899999745 +0000 UTC m=+499.834749409" Jul 21 21:09:11.376875 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:09:11.376840 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-6675f745fd-fswbm" Jul 21 21:09:11.377301 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:09:11.376993 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-6675f745fd-fswbm" Jul 21 21:09:11.381332 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:09:11.381309 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-6675f745fd-fswbm" Jul 21 21:09:11.927814 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:09:11.927786 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-6675f745fd-fswbm" Jul 21 21:09:27.959515 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:09:27.959480 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-6fdcfcc664-r8bxd"] Jul 21 21:09:27.965234 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:09:27.965214 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6fdcfcc664-r8bxd" Jul 21 21:09:27.967972 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:09:27.967944 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\"" Jul 21 21:09:27.969996 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:09:27.969971 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6fdcfcc664-r8bxd"] Jul 21 21:09:28.128298 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:09:28.128272 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/e7dd5576-6e15-4e54-a782-1d2e8a505b5c-tls-cert\") pod \"authorino-6fdcfcc664-r8bxd\" (UID: \"e7dd5576-6e15-4e54-a782-1d2e8a505b5c\") " pod="kuadrant-system/authorino-6fdcfcc664-r8bxd" Jul 21 21:09:28.128298 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:09:28.128307 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-58dgx\" (UniqueName: \"kubernetes.io/projected/e7dd5576-6e15-4e54-a782-1d2e8a505b5c-kube-api-access-58dgx\") pod \"authorino-6fdcfcc664-r8bxd\" (UID: \"e7dd5576-6e15-4e54-a782-1d2e8a505b5c\") " pod="kuadrant-system/authorino-6fdcfcc664-r8bxd" Jul 21 21:09:28.228909 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:09:28.228837 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-58dgx\" (UniqueName: \"kubernetes.io/projected/e7dd5576-6e15-4e54-a782-1d2e8a505b5c-kube-api-access-58dgx\") pod \"authorino-6fdcfcc664-r8bxd\" (UID: \"e7dd5576-6e15-4e54-a782-1d2e8a505b5c\") " pod="kuadrant-system/authorino-6fdcfcc664-r8bxd" Jul 21 21:09:28.229050 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:09:28.228969 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/e7dd5576-6e15-4e54-a782-1d2e8a505b5c-tls-cert\") pod \"authorino-6fdcfcc664-r8bxd\" (UID: \"e7dd5576-6e15-4e54-a782-1d2e8a505b5c\") " pod="kuadrant-system/authorino-6fdcfcc664-r8bxd" Jul 21 21:09:28.231348 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:09:28.231329 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/e7dd5576-6e15-4e54-a782-1d2e8a505b5c-tls-cert\") pod \"authorino-6fdcfcc664-r8bxd\" (UID: \"e7dd5576-6e15-4e54-a782-1d2e8a505b5c\") " pod="kuadrant-system/authorino-6fdcfcc664-r8bxd" Jul 21 21:09:28.237705 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:09:28.237683 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-58dgx\" (UniqueName: \"kubernetes.io/projected/e7dd5576-6e15-4e54-a782-1d2e8a505b5c-kube-api-access-58dgx\") pod \"authorino-6fdcfcc664-r8bxd\" (UID: \"e7dd5576-6e15-4e54-a782-1d2e8a505b5c\") " pod="kuadrant-system/authorino-6fdcfcc664-r8bxd" Jul 21 21:09:28.275733 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:09:28.275709 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6fdcfcc664-r8bxd" Jul 21 21:09:28.407841 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:09:28.407817 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6fdcfcc664-r8bxd"] Jul 21 21:09:28.410029 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:09:28.410003 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode7dd5576_6e15_4e54_a782_1d2e8a505b5c.slice/crio-48c99eb2b7d0fac6ce0c3bcc06333a09beecf121ceb7f614ff2e9f3175ae5100 WatchSource:0}: Error finding container 48c99eb2b7d0fac6ce0c3bcc06333a09beecf121ceb7f614ff2e9f3175ae5100: Status 404 returned error can't find the container with id 48c99eb2b7d0fac6ce0c3bcc06333a09beecf121ceb7f614ff2e9f3175ae5100 Jul 21 21:09:28.997984 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:09:28.997939 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6fdcfcc664-r8bxd" event={"ID":"e7dd5576-6e15-4e54-a782-1d2e8a505b5c","Type":"ContainerStarted","Data":"fcd3dc0066df7cc40e9b0d20d838a0d95d7cebd38c680076a7e4f8e2e88e1e07"} Jul 21 21:09:28.997984 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:09:28.997988 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6fdcfcc664-r8bxd" event={"ID":"e7dd5576-6e15-4e54-a782-1d2e8a505b5c","Type":"ContainerStarted","Data":"48c99eb2b7d0fac6ce0c3bcc06333a09beecf121ceb7f614ff2e9f3175ae5100"} Jul 21 21:09:29.012742 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:09:29.012692 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-6fdcfcc664-r8bxd" podStartSLOduration=1.631473526 podStartE2EDuration="2.012676687s" totalImagesPullingTime="381.203161ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 21:09:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 21:09:29.011928587 +0000 UTC m=+526.946678253" watchObservedRunningTime="2026-07-21 21:09:29.012676687 +0000 UTC m=+526.947426351" Jul 21 21:09:29.035518 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:09:29.035490 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-75d556bd77-6q64t"] Jul 21 21:09:29.035701 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:09:29.035676 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-75d556bd77-6q64t" podUID="fc71117e-c61e-4180-a106-cb6f06bf023e" containerName="authorino" containerID="cri-o://f5953411135ce4cc2975a24cb3a020dd2ddb62e9a53e87217016f8be2b26c15d" gracePeriod=30 Jul 21 21:09:29.286059 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:09:29.286037 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-6q64t" Jul 21 21:09:29.439288 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:09:29.439252 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jk867\" (UniqueName: \"kubernetes.io/projected/fc71117e-c61e-4180-a106-cb6f06bf023e-kube-api-access-jk867\") pod \"fc71117e-c61e-4180-a106-cb6f06bf023e\" (UID: \"fc71117e-c61e-4180-a106-cb6f06bf023e\") " Jul 21 21:09:29.441239 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:09:29.441211 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fc71117e-c61e-4180-a106-cb6f06bf023e-kube-api-access-jk867" (OuterVolumeSpecName: "kube-api-access-jk867") pod "fc71117e-c61e-4180-a106-cb6f06bf023e" (UID: "fc71117e-c61e-4180-a106-cb6f06bf023e"). InnerVolumeSpecName "kube-api-access-jk867". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 21 21:09:29.540364 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:09:29.540334 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jk867\" (UniqueName: \"kubernetes.io/projected/fc71117e-c61e-4180-a106-cb6f06bf023e-kube-api-access-jk867\") on node \"ip-10-0-140-97.ec2.internal\" DevicePath \"\"" Jul 21 21:09:30.003388 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:09:30.003350 2575 generic.go:356] "Generic (PLEG): container finished" podID="fc71117e-c61e-4180-a106-cb6f06bf023e" containerID="f5953411135ce4cc2975a24cb3a020dd2ddb62e9a53e87217016f8be2b26c15d" exitCode=0 Jul 21 21:09:30.003830 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:09:30.003392 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-6q64t" Jul 21 21:09:30.003830 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:09:30.003437 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-6q64t" event={"ID":"fc71117e-c61e-4180-a106-cb6f06bf023e","Type":"ContainerDied","Data":"f5953411135ce4cc2975a24cb3a020dd2ddb62e9a53e87217016f8be2b26c15d"} Jul 21 21:09:30.003830 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:09:30.003485 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-6q64t" event={"ID":"fc71117e-c61e-4180-a106-cb6f06bf023e","Type":"ContainerDied","Data":"d6fadf469de9892bfa1bb70b7fd721a82bddc8dbe4d0eab45a5035ff1058dc90"} Jul 21 21:09:30.003830 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:09:30.003510 2575 scope.go:117] "RemoveContainer" containerID="f5953411135ce4cc2975a24cb3a020dd2ddb62e9a53e87217016f8be2b26c15d" Jul 21 21:09:30.012807 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:09:30.012789 2575 scope.go:117] "RemoveContainer" containerID="f5953411135ce4cc2975a24cb3a020dd2ddb62e9a53e87217016f8be2b26c15d" Jul 21 21:09:30.013037 ip-10-0-140-97 kubenswrapper[2575]: E0721 21:09:30.013012 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f5953411135ce4cc2975a24cb3a020dd2ddb62e9a53e87217016f8be2b26c15d\": container with ID starting with f5953411135ce4cc2975a24cb3a020dd2ddb62e9a53e87217016f8be2b26c15d not found: ID does not exist" containerID="f5953411135ce4cc2975a24cb3a020dd2ddb62e9a53e87217016f8be2b26c15d" Jul 21 21:09:30.013095 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:09:30.013046 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f5953411135ce4cc2975a24cb3a020dd2ddb62e9a53e87217016f8be2b26c15d"} err="failed to get container status \"f5953411135ce4cc2975a24cb3a020dd2ddb62e9a53e87217016f8be2b26c15d\": rpc error: code = NotFound desc = could not find container \"f5953411135ce4cc2975a24cb3a020dd2ddb62e9a53e87217016f8be2b26c15d\": container with ID starting with f5953411135ce4cc2975a24cb3a020dd2ddb62e9a53e87217016f8be2b26c15d not found: ID does not exist" Jul 21 21:09:30.024724 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:09:30.024703 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-75d556bd77-6q64t"] Jul 21 21:09:30.031192 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:09:30.027368 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-75d556bd77-6q64t"] Jul 21 21:09:30.741671 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:09:30.741636 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fc71117e-c61e-4180-a106-cb6f06bf023e" path="/var/lib/kubelet/pods/fc71117e-c61e-4180-a106-cb6f06bf023e/volumes" Jul 21 21:10:19.230614 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:10:19.230502 2575 generic.go:356] "Generic (PLEG): container finished" podID="34d429d3-1614-43f5-997f-492caa021783" containerID="8638249f201491393008c05255ba0cee2f38be7e58bb737b76b568f0079a293b" exitCode=0 Jul 21 21:10:19.230614 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:10:19.230585 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-2pnp6" event={"ID":"34d429d3-1614-43f5-997f-492caa021783","Type":"ContainerDied","Data":"8638249f201491393008c05255ba0cee2f38be7e58bb737b76b568f0079a293b"} Jul 21 21:10:19.231056 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:10:19.230954 2575 scope.go:117] "RemoveContainer" containerID="8638249f201491393008c05255ba0cee2f38be7e58bb737b76b568f0079a293b" Jul 21 21:10:19.234402 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:10:19.234363 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 21 21:10:19.234541 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:10:19.234458 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 21 21:10:19.234541 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:10:19.234511 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 21 21:10:20.236256 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:10:20.236226 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-2pnp6" event={"ID":"34d429d3-1614-43f5-997f-492caa021783","Type":"ContainerStarted","Data":"2a6be7eb85281938da740cfed0b16477d3a43f4e591d77561eae4b2307c18da3"} Jul 21 21:10:20.515592 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:10:20.515485 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-2pnp6" Jul 21 21:10:20.516657 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:10:20.516615 2575 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-56654dcf7-2pnp6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.32:15021/healthz/ready\": dial tcp 10.134.0.32:15021: connect: connection refused" start-of-body= Jul 21 21:10:20.516784 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:10:20.516679 2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-2pnp6" podUID="34d429d3-1614-43f5-997f-492caa021783" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.32:15021/healthz/ready\": dial tcp 10.134.0.32:15021: connect: connection refused" Jul 21 21:10:21.516691 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:10:21.516654 2575 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-56654dcf7-2pnp6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.32:15021/healthz/ready\": dial tcp 10.134.0.32:15021: connect: connection refused" start-of-body= Jul 21 21:10:21.517099 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:10:21.516722 2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-2pnp6" podUID="34d429d3-1614-43f5-997f-492caa021783" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.32:15021/healthz/ready\": dial tcp 10.134.0.32:15021: connect: connection refused" Jul 21 21:10:22.515907 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:10:22.515871 2575 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-56654dcf7-2pnp6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.32:15021/healthz/ready\": dial tcp 10.134.0.32:15021: connect: connection refused" start-of-body= Jul 21 21:10:22.516072 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:10:22.515927 2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-2pnp6" podUID="34d429d3-1614-43f5-997f-492caa021783" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.32:15021/healthz/ready\": dial tcp 10.134.0.32:15021: connect: connection refused" Jul 21 21:10:22.837010 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:10:22.836920 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-2pnp6" Jul 21 21:10:23.515809 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:10:23.515779 2575 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-56654dcf7-2pnp6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.32:15021/healthz/ready\": dial tcp 10.134.0.32:15021: connect: connection refused" start-of-body= Jul 21 21:10:23.515960 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:10:23.515828 2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-2pnp6" podUID="34d429d3-1614-43f5-997f-492caa021783" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.32:15021/healthz/ready\": dial tcp 10.134.0.32:15021: connect: connection refused" Jul 21 21:10:24.520306 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:10:24.520278 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-2pnp6" Jul 21 21:10:24.521148 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:10:24.521131 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-2pnp6" Jul 21 21:10:49.631081 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:10:49.631036 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-bz46f"] Jul 21 21:10:49.631737 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:10:49.631548 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fc71117e-c61e-4180-a106-cb6f06bf023e" containerName="authorino" Jul 21 21:10:49.631737 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:10:49.631572 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="fc71117e-c61e-4180-a106-cb6f06bf023e" containerName="authorino" Jul 21 21:10:49.631737 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:10:49.631710 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="fc71117e-c61e-4180-a106-cb6f06bf023e" containerName="authorino" Jul 21 21:10:49.634729 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:10:49.634708 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-bz46f" Jul 21 21:10:49.638067 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:10:49.638040 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Jul 21 21:10:49.638199 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:10:49.638061 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-t48v6\"" Jul 21 21:10:49.638199 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:10:49.638061 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-simulated-kserve-self-signed-certs\"" Jul 21 21:10:49.638199 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:10:49.638072 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Jul 21 21:10:49.646362 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:10:49.646342 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-bz46f"] Jul 21 21:10:49.718078 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:10:49.718052 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7f45b356-8eac-4587-9cf0-183e1a86555a-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-bz46f\" (UID: \"7f45b356-8eac-4587-9cf0-183e1a86555a\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-bz46f" Jul 21 21:10:49.718214 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:10:49.718084 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7f45b356-8eac-4587-9cf0-183e1a86555a-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-bz46f\" (UID: \"7f45b356-8eac-4587-9cf0-183e1a86555a\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-bz46f" Jul 21 21:10:49.718214 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:10:49.718116 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7f45b356-8eac-4587-9cf0-183e1a86555a-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-bz46f\" (UID: \"7f45b356-8eac-4587-9cf0-183e1a86555a\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-bz46f" Jul 21 21:10:49.718284 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:10:49.718228 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7f45b356-8eac-4587-9cf0-183e1a86555a-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-bz46f\" (UID: \"7f45b356-8eac-4587-9cf0-183e1a86555a\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-bz46f" Jul 21 21:10:49.718284 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:10:49.718271 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7f45b356-8eac-4587-9cf0-183e1a86555a-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-bz46f\" (UID: \"7f45b356-8eac-4587-9cf0-183e1a86555a\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-bz46f" Jul 21 21:10:49.718350 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:10:49.718328 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pht9j\" (UniqueName: \"kubernetes.io/projected/7f45b356-8eac-4587-9cf0-183e1a86555a-kube-api-access-pht9j\") pod \"e2e-distinct-simulated-kserve-59975d69f9-bz46f\" (UID: \"7f45b356-8eac-4587-9cf0-183e1a86555a\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-bz46f" Jul 21 21:10:49.819554 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:10:49.819498 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pht9j\" (UniqueName: \"kubernetes.io/projected/7f45b356-8eac-4587-9cf0-183e1a86555a-kube-api-access-pht9j\") pod \"e2e-distinct-simulated-kserve-59975d69f9-bz46f\" (UID: \"7f45b356-8eac-4587-9cf0-183e1a86555a\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-bz46f" Jul 21 21:10:49.819722 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:10:49.819570 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7f45b356-8eac-4587-9cf0-183e1a86555a-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-bz46f\" (UID: \"7f45b356-8eac-4587-9cf0-183e1a86555a\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-bz46f" Jul 21 21:10:49.819722 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:10:49.819599 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7f45b356-8eac-4587-9cf0-183e1a86555a-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-bz46f\" (UID: \"7f45b356-8eac-4587-9cf0-183e1a86555a\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-bz46f" Jul 21 21:10:49.819722 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:10:49.819636 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7f45b356-8eac-4587-9cf0-183e1a86555a-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-bz46f\" (UID: \"7f45b356-8eac-4587-9cf0-183e1a86555a\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-bz46f" Jul 21 21:10:49.819722 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:10:49.819687 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7f45b356-8eac-4587-9cf0-183e1a86555a-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-bz46f\" (UID: \"7f45b356-8eac-4587-9cf0-183e1a86555a\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-bz46f" Jul 21 21:10:49.819722 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:10:49.819720 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7f45b356-8eac-4587-9cf0-183e1a86555a-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-bz46f\" (UID: \"7f45b356-8eac-4587-9cf0-183e1a86555a\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-bz46f" Jul 21 21:10:49.820068 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:10:49.820041 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7f45b356-8eac-4587-9cf0-183e1a86555a-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-bz46f\" (UID: \"7f45b356-8eac-4587-9cf0-183e1a86555a\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-bz46f" Jul 21 21:10:49.820068 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:10:49.820057 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7f45b356-8eac-4587-9cf0-183e1a86555a-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-bz46f\" (UID: \"7f45b356-8eac-4587-9cf0-183e1a86555a\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-bz46f" Jul 21 21:10:49.820164 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:10:49.820108 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7f45b356-8eac-4587-9cf0-183e1a86555a-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-bz46f\" (UID: \"7f45b356-8eac-4587-9cf0-183e1a86555a\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-bz46f" Jul 21 21:10:49.821862 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:10:49.821846 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7f45b356-8eac-4587-9cf0-183e1a86555a-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-bz46f\" (UID: \"7f45b356-8eac-4587-9cf0-183e1a86555a\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-bz46f" Jul 21 21:10:49.822181 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:10:49.822166 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7f45b356-8eac-4587-9cf0-183e1a86555a-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-bz46f\" (UID: \"7f45b356-8eac-4587-9cf0-183e1a86555a\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-bz46f" Jul 21 21:10:49.828017 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:10:49.827999 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pht9j\" (UniqueName: \"kubernetes.io/projected/7f45b356-8eac-4587-9cf0-183e1a86555a-kube-api-access-pht9j\") pod \"e2e-distinct-simulated-kserve-59975d69f9-bz46f\" (UID: \"7f45b356-8eac-4587-9cf0-183e1a86555a\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-bz46f" Jul 21 21:10:49.945719 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:10:49.945643 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-bz46f" Jul 21 21:10:50.073702 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:10:50.073675 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-bz46f"] Jul 21 21:10:50.075680 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:10:50.075652 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7f45b356_8eac_4587_9cf0_183e1a86555a.slice/crio-c390e270f08d44b2663a90ec5271af5a1e634fa29095973b1c085a7aea370f5e WatchSource:0}: Error finding container c390e270f08d44b2663a90ec5271af5a1e634fa29095973b1c085a7aea370f5e: Status 404 returned error can't find the container with id c390e270f08d44b2663a90ec5271af5a1e634fa29095973b1c085a7aea370f5e Jul 21 21:10:50.384681 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:10:50.384634 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-bz46f" event={"ID":"7f45b356-8eac-4587-9cf0-183e1a86555a","Type":"ContainerStarted","Data":"c390e270f08d44b2663a90ec5271af5a1e634fa29095973b1c085a7aea370f5e"} Jul 21 21:10:51.397757 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:10:51.397716 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-bz46f" event={"ID":"7f45b356-8eac-4587-9cf0-183e1a86555a","Type":"ContainerStarted","Data":"5b330502e46cdc276806f05b8e20f7550a5b7a542e54ebb95d0182622f28816b"} Jul 21 21:10:51.415344 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:10:51.415293 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-bz46f" podStartSLOduration=1.254739125 podStartE2EDuration="2.415276585s" totalImagesPullingTime="1.16053746s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 21:10:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 21:10:51.413509842 +0000 UTC m=+609.348259508" watchObservedRunningTime="2026-07-21 21:10:51.415276585 +0000 UTC m=+609.350026257" Jul 21 21:10:59.033021 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:10:59.032983 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-tqkl4"] Jul 21 21:10:59.036399 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:10:59.036379 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-tqkl4" Jul 21 21:10:59.039420 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:10:59.039401 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-trlp-test-simulated-kserve-self-signed-certs\"" Jul 21 21:10:59.046619 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:10:59.046579 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-tqkl4"] Jul 21 21:10:59.106370 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:10:59.106329 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/87408656-7347-480e-9e6e-937e499c6f6a-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-tqkl4\" (UID: \"87408656-7347-480e-9e6e-937e499c6f6a\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-tqkl4" Jul 21 21:10:59.106575 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:10:59.106423 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/87408656-7347-480e-9e6e-937e499c6f6a-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-tqkl4\" (UID: \"87408656-7347-480e-9e6e-937e499c6f6a\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-tqkl4" Jul 21 21:10:59.106575 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:10:59.106454 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/87408656-7347-480e-9e6e-937e499c6f6a-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-tqkl4\" (UID: \"87408656-7347-480e-9e6e-937e499c6f6a\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-tqkl4" Jul 21 21:10:59.106575 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:10:59.106474 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/87408656-7347-480e-9e6e-937e499c6f6a-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-tqkl4\" (UID: \"87408656-7347-480e-9e6e-937e499c6f6a\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-tqkl4" Jul 21 21:10:59.106575 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:10:59.106510 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/87408656-7347-480e-9e6e-937e499c6f6a-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-tqkl4\" (UID: \"87408656-7347-480e-9e6e-937e499c6f6a\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-tqkl4" Jul 21 21:10:59.106733 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:10:59.106586 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4qxwm\" (UniqueName: \"kubernetes.io/projected/87408656-7347-480e-9e6e-937e499c6f6a-kube-api-access-4qxwm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-tqkl4\" (UID: \"87408656-7347-480e-9e6e-937e499c6f6a\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-tqkl4" Jul 21 21:10:59.207889 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:10:59.207850 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/87408656-7347-480e-9e6e-937e499c6f6a-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-tqkl4\" (UID: \"87408656-7347-480e-9e6e-937e499c6f6a\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-tqkl4" Jul 21 21:10:59.208066 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:10:59.207907 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/87408656-7347-480e-9e6e-937e499c6f6a-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-tqkl4\" (UID: \"87408656-7347-480e-9e6e-937e499c6f6a\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-tqkl4" Jul 21 21:10:59.208066 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:10:59.207937 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/87408656-7347-480e-9e6e-937e499c6f6a-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-tqkl4\" (UID: \"87408656-7347-480e-9e6e-937e499c6f6a\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-tqkl4" Jul 21 21:10:59.208066 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:10:59.207983 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/87408656-7347-480e-9e6e-937e499c6f6a-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-tqkl4\" (UID: \"87408656-7347-480e-9e6e-937e499c6f6a\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-tqkl4" Jul 21 21:10:59.208066 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:10:59.208029 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4qxwm\" (UniqueName: \"kubernetes.io/projected/87408656-7347-480e-9e6e-937e499c6f6a-kube-api-access-4qxwm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-tqkl4\" (UID: \"87408656-7347-480e-9e6e-937e499c6f6a\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-tqkl4" Jul 21 21:10:59.208284 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:10:59.208103 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/87408656-7347-480e-9e6e-937e499c6f6a-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-tqkl4\" (UID: \"87408656-7347-480e-9e6e-937e499c6f6a\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-tqkl4" Jul 21 21:10:59.208349 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:10:59.208293 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/87408656-7347-480e-9e6e-937e499c6f6a-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-tqkl4\" (UID: \"87408656-7347-480e-9e6e-937e499c6f6a\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-tqkl4" Jul 21 21:10:59.208407 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:10:59.208350 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/87408656-7347-480e-9e6e-937e499c6f6a-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-tqkl4\" (UID: \"87408656-7347-480e-9e6e-937e499c6f6a\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-tqkl4" Jul 21 21:10:59.208447 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:10:59.208400 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/87408656-7347-480e-9e6e-937e499c6f6a-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-tqkl4\" (UID: \"87408656-7347-480e-9e6e-937e499c6f6a\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-tqkl4" Jul 21 21:10:59.210891 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:10:59.210853 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/87408656-7347-480e-9e6e-937e499c6f6a-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-tqkl4\" (UID: \"87408656-7347-480e-9e6e-937e499c6f6a\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-tqkl4" Jul 21 21:10:59.211229 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:10:59.211206 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/87408656-7347-480e-9e6e-937e499c6f6a-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-tqkl4\" (UID: \"87408656-7347-480e-9e6e-937e499c6f6a\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-tqkl4" Jul 21 21:10:59.216246 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:10:59.216227 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4qxwm\" (UniqueName: \"kubernetes.io/projected/87408656-7347-480e-9e6e-937e499c6f6a-kube-api-access-4qxwm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-tqkl4\" (UID: \"87408656-7347-480e-9e6e-937e499c6f6a\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-tqkl4" Jul 21 21:10:59.348381 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:10:59.348298 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-tqkl4" Jul 21 21:10:59.483063 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:10:59.483038 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-tqkl4"] Jul 21 21:10:59.485640 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:10:59.485607 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod87408656_7347_480e_9e6e_937e499c6f6a.slice/crio-207242754dbb03b06cbb8a792f37bfd4e294f4d49d3e2c8d6d0c2340d530e944 WatchSource:0}: Error finding container 207242754dbb03b06cbb8a792f37bfd4e294f4d49d3e2c8d6d0c2340d530e944: Status 404 returned error can't find the container with id 207242754dbb03b06cbb8a792f37bfd4e294f4d49d3e2c8d6d0c2340d530e944 Jul 21 21:10:59.487430 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:10:59.487409 2575 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 21 21:10:59.946350 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:10:59.946311 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-bz46f" Jul 21 21:10:59.946525 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:10:59.946383 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-bz46f" Jul 21 21:10:59.959868 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:10:59.959803 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-bz46f" Jul 21 21:11:00.442465 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:11:00.442430 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-tqkl4" event={"ID":"87408656-7347-480e-9e6e-937e499c6f6a","Type":"ContainerStarted","Data":"2534df72cffb5e498ae6c891ef9f50074d3864f5a4923cc934164d35f366d8ea"} Jul 21 21:11:00.442841 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:11:00.442494 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-tqkl4" event={"ID":"87408656-7347-480e-9e6e-937e499c6f6a","Type":"ContainerStarted","Data":"207242754dbb03b06cbb8a792f37bfd4e294f4d49d3e2c8d6d0c2340d530e944"} Jul 21 21:11:00.456556 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:11:00.456507 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-bz46f" Jul 21 21:11:00.461430 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:11:00.461371 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-tqkl4" podStartSLOduration=1.247439035 podStartE2EDuration="1.461355374s" totalImagesPullingTime="213.916339ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 21:10:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 21:11:00.460420053 +0000 UTC m=+618.395169720" watchObservedRunningTime="2026-07-21 21:11:00.461355374 +0000 UTC m=+618.396105043" Jul 21 21:11:09.348582 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:11:09.348468 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-tqkl4" Jul 21 21:11:09.348582 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:11:09.348552 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-tqkl4" Jul 21 21:11:09.361190 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:11:09.361161 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-tqkl4" Jul 21 21:11:09.498020 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:11:09.497989 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-tqkl4" Jul 21 21:12:47.952436 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:12:47.952345 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6fdcfcc664-r8bxd"] Jul 21 21:12:47.953080 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:12:47.952556 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-6fdcfcc664-r8bxd" podUID="e7dd5576-6e15-4e54-a782-1d2e8a505b5c" containerName="authorino" containerID="cri-o://fcd3dc0066df7cc40e9b0d20d838a0d95d7cebd38c680076a7e4f8e2e88e1e07" gracePeriod=30 Jul 21 21:12:48.204794 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:12:48.204740 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6fdcfcc664-r8bxd" Jul 21 21:12:48.266371 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:12:48.266342 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-58dgx\" (UniqueName: \"kubernetes.io/projected/e7dd5576-6e15-4e54-a782-1d2e8a505b5c-kube-api-access-58dgx\") pod \"e7dd5576-6e15-4e54-a782-1d2e8a505b5c\" (UID: \"e7dd5576-6e15-4e54-a782-1d2e8a505b5c\") " Jul 21 21:12:48.266524 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:12:48.266404 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/e7dd5576-6e15-4e54-a782-1d2e8a505b5c-tls-cert\") pod \"e7dd5576-6e15-4e54-a782-1d2e8a505b5c\" (UID: \"e7dd5576-6e15-4e54-a782-1d2e8a505b5c\") " Jul 21 21:12:48.268497 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:12:48.268473 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e7dd5576-6e15-4e54-a782-1d2e8a505b5c-kube-api-access-58dgx" (OuterVolumeSpecName: "kube-api-access-58dgx") pod "e7dd5576-6e15-4e54-a782-1d2e8a505b5c" (UID: "e7dd5576-6e15-4e54-a782-1d2e8a505b5c"). InnerVolumeSpecName "kube-api-access-58dgx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 21 21:12:48.278168 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:12:48.278142 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e7dd5576-6e15-4e54-a782-1d2e8a505b5c-tls-cert" (OuterVolumeSpecName: "tls-cert") pod "e7dd5576-6e15-4e54-a782-1d2e8a505b5c" (UID: "e7dd5576-6e15-4e54-a782-1d2e8a505b5c"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 21 21:12:48.367157 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:12:48.367123 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-58dgx\" (UniqueName: \"kubernetes.io/projected/e7dd5576-6e15-4e54-a782-1d2e8a505b5c-kube-api-access-58dgx\") on node \"ip-10-0-140-97.ec2.internal\" DevicePath \"\"" Jul 21 21:12:48.367157 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:12:48.367151 2575 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/e7dd5576-6e15-4e54-a782-1d2e8a505b5c-tls-cert\") on node \"ip-10-0-140-97.ec2.internal\" DevicePath \"\"" Jul 21 21:12:48.953805 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:12:48.953766 2575 generic.go:356] "Generic (PLEG): container finished" podID="e7dd5576-6e15-4e54-a782-1d2e8a505b5c" containerID="fcd3dc0066df7cc40e9b0d20d838a0d95d7cebd38c680076a7e4f8e2e88e1e07" exitCode=0 Jul 21 21:12:48.954238 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:12:48.953819 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6fdcfcc664-r8bxd" Jul 21 21:12:48.954238 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:12:48.953855 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6fdcfcc664-r8bxd" event={"ID":"e7dd5576-6e15-4e54-a782-1d2e8a505b5c","Type":"ContainerDied","Data":"fcd3dc0066df7cc40e9b0d20d838a0d95d7cebd38c680076a7e4f8e2e88e1e07"} Jul 21 21:12:48.954238 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:12:48.953901 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6fdcfcc664-r8bxd" event={"ID":"e7dd5576-6e15-4e54-a782-1d2e8a505b5c","Type":"ContainerDied","Data":"48c99eb2b7d0fac6ce0c3bcc06333a09beecf121ceb7f614ff2e9f3175ae5100"} Jul 21 21:12:48.954238 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:12:48.953925 2575 scope.go:117] "RemoveContainer" containerID="fcd3dc0066df7cc40e9b0d20d838a0d95d7cebd38c680076a7e4f8e2e88e1e07" Jul 21 21:12:48.965998 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:12:48.965978 2575 scope.go:117] "RemoveContainer" containerID="fcd3dc0066df7cc40e9b0d20d838a0d95d7cebd38c680076a7e4f8e2e88e1e07" Jul 21 21:12:48.966351 ip-10-0-140-97 kubenswrapper[2575]: E0721 21:12:48.966318 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fcd3dc0066df7cc40e9b0d20d838a0d95d7cebd38c680076a7e4f8e2e88e1e07\": container with ID starting with fcd3dc0066df7cc40e9b0d20d838a0d95d7cebd38c680076a7e4f8e2e88e1e07 not found: ID does not exist" containerID="fcd3dc0066df7cc40e9b0d20d838a0d95d7cebd38c680076a7e4f8e2e88e1e07" Jul 21 21:12:48.966404 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:12:48.966357 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fcd3dc0066df7cc40e9b0d20d838a0d95d7cebd38c680076a7e4f8e2e88e1e07"} err="failed to get container status \"fcd3dc0066df7cc40e9b0d20d838a0d95d7cebd38c680076a7e4f8e2e88e1e07\": rpc error: code = NotFound desc = could not find container \"fcd3dc0066df7cc40e9b0d20d838a0d95d7cebd38c680076a7e4f8e2e88e1e07\": container with ID starting with fcd3dc0066df7cc40e9b0d20d838a0d95d7cebd38c680076a7e4f8e2e88e1e07 not found: ID does not exist" Jul 21 21:12:48.971810 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:12:48.971786 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6fdcfcc664-r8bxd"] Jul 21 21:12:48.976803 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:12:48.976780 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-6fdcfcc664-r8bxd"] Jul 21 21:12:50.741319 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:12:50.741284 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e7dd5576-6e15-4e54-a782-1d2e8a505b5c" path="/var/lib/kubelet/pods/e7dd5576-6e15-4e54-a782-1d2e8a505b5c/volumes" Jul 21 21:15:00.158812 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:15:00.158774 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29744475-s4gdw"] Jul 21 21:15:00.159305 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:15:00.159216 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e7dd5576-6e15-4e54-a782-1d2e8a505b5c" containerName="authorino" Jul 21 21:15:00.159305 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:15:00.159232 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="e7dd5576-6e15-4e54-a782-1d2e8a505b5c" containerName="authorino" Jul 21 21:15:00.159305 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:15:00.159295 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="e7dd5576-6e15-4e54-a782-1d2e8a505b5c" containerName="authorino" Jul 21 21:15:00.162138 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:15:00.162121 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29744475-s4gdw" Jul 21 21:15:00.164862 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:15:00.164839 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9fxhr\"" Jul 21 21:15:00.171097 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:15:00.171075 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29744475-s4gdw"] Jul 21 21:15:00.173806 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:15:00.173785 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29744475-s4gdw" Jul 21 21:15:00.304979 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:15:00.304951 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29744475-s4gdw"] Jul 21 21:15:00.561230 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:15:00.561147 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29744475-s4gdw" event={"ID":"23c11dc8-340e-413d-9cfa-19db16bb4708","Type":"ContainerStarted","Data":"3ec69f551eaa88100f1e2602b7ced5e53c840572d08b006423105928f8120283"} Jul 21 21:15:04.584921 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:15:04.584879 2575 generic.go:356] "Generic (PLEG): container finished" podID="23c11dc8-340e-413d-9cfa-19db16bb4708" containerID="17ce0bbd90f4d7f93ddcb7d13727fc111f8bf7a61e5b197a6b6990a698aa34d5" exitCode=0 Jul 21 21:15:04.585336 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:15:04.584960 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29744475-s4gdw" event={"ID":"23c11dc8-340e-413d-9cfa-19db16bb4708","Type":"ContainerDied","Data":"17ce0bbd90f4d7f93ddcb7d13727fc111f8bf7a61e5b197a6b6990a698aa34d5"} Jul 21 21:15:05.730558 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:15:05.730513 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29744475-s4gdw" Jul 21 21:15:06.595658 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:15:06.595629 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29744475-s4gdw" Jul 21 21:15:06.595824 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:15:06.595667 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29744475-s4gdw" event={"ID":"23c11dc8-340e-413d-9cfa-19db16bb4708","Type":"ContainerDied","Data":"3ec69f551eaa88100f1e2602b7ced5e53c840572d08b006423105928f8120283"} Jul 21 21:15:06.595824 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:15:06.595712 2575 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3ec69f551eaa88100f1e2602b7ced5e53c840572d08b006423105928f8120283" Jul 21 21:19:41.860772 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:19:41.860730 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-6gqzm"] Jul 21 21:19:41.861355 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:19:41.860949 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-6gqzm" podUID="8a7d2814-c13b-4fc4-bd0d-9739d5bab9d4" containerName="manager" containerID="cri-o://62352d1abfb749c96b1e280536782f5bda0be32638ab3a78e1ec4af8f7ae2e01" gracePeriod=10 Jul 21 21:19:44.306591 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:19:44.306565 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-6gqzm" Jul 21 21:19:44.379116 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:19:44.379086 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/8a7d2814-c13b-4fc4-bd0d-9739d5bab9d4-extensions-socket-volume\") pod \"8a7d2814-c13b-4fc4-bd0d-9739d5bab9d4\" (UID: \"8a7d2814-c13b-4fc4-bd0d-9739d5bab9d4\") " Jul 21 21:19:44.379294 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:19:44.379208 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wbc6m\" (UniqueName: \"kubernetes.io/projected/8a7d2814-c13b-4fc4-bd0d-9739d5bab9d4-kube-api-access-wbc6m\") pod \"8a7d2814-c13b-4fc4-bd0d-9739d5bab9d4\" (UID: \"8a7d2814-c13b-4fc4-bd0d-9739d5bab9d4\") " Jul 21 21:19:44.379484 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:19:44.379461 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8a7d2814-c13b-4fc4-bd0d-9739d5bab9d4-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "8a7d2814-c13b-4fc4-bd0d-9739d5bab9d4" (UID: "8a7d2814-c13b-4fc4-bd0d-9739d5bab9d4"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 21:19:44.381210 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:19:44.381172 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8a7d2814-c13b-4fc4-bd0d-9739d5bab9d4-kube-api-access-wbc6m" (OuterVolumeSpecName: "kube-api-access-wbc6m") pod "8a7d2814-c13b-4fc4-bd0d-9739d5bab9d4" (UID: "8a7d2814-c13b-4fc4-bd0d-9739d5bab9d4"). InnerVolumeSpecName "kube-api-access-wbc6m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 21 21:19:44.480588 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:19:44.480502 2575 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/8a7d2814-c13b-4fc4-bd0d-9739d5bab9d4-extensions-socket-volume\") on node \"ip-10-0-140-97.ec2.internal\" DevicePath \"\"" Jul 21 21:19:44.480588 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:19:44.480555 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wbc6m\" (UniqueName: \"kubernetes.io/projected/8a7d2814-c13b-4fc4-bd0d-9739d5bab9d4-kube-api-access-wbc6m\") on node \"ip-10-0-140-97.ec2.internal\" DevicePath \"\"" Jul 21 21:19:44.929669 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:19:44.929635 2575 generic.go:356] "Generic (PLEG): container finished" podID="8a7d2814-c13b-4fc4-bd0d-9739d5bab9d4" containerID="62352d1abfb749c96b1e280536782f5bda0be32638ab3a78e1ec4af8f7ae2e01" exitCode=0 Jul 21 21:19:44.929835 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:19:44.929679 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-6gqzm" Jul 21 21:19:44.929835 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:19:44.929707 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-6gqzm" event={"ID":"8a7d2814-c13b-4fc4-bd0d-9739d5bab9d4","Type":"ContainerDied","Data":"62352d1abfb749c96b1e280536782f5bda0be32638ab3a78e1ec4af8f7ae2e01"} Jul 21 21:19:44.929835 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:19:44.929748 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-6gqzm" event={"ID":"8a7d2814-c13b-4fc4-bd0d-9739d5bab9d4","Type":"ContainerDied","Data":"f482372395c397141d268b341772b5e521db00c72c0c1e3d3f5e8e6e954f01c7"} Jul 21 21:19:44.929835 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:19:44.929776 2575 scope.go:117] "RemoveContainer" containerID="62352d1abfb749c96b1e280536782f5bda0be32638ab3a78e1ec4af8f7ae2e01" Jul 21 21:19:44.941115 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:19:44.941097 2575 scope.go:117] "RemoveContainer" containerID="62352d1abfb749c96b1e280536782f5bda0be32638ab3a78e1ec4af8f7ae2e01" Jul 21 21:19:44.941383 ip-10-0-140-97 kubenswrapper[2575]: E0721 21:19:44.941366 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"62352d1abfb749c96b1e280536782f5bda0be32638ab3a78e1ec4af8f7ae2e01\": container with ID starting with 62352d1abfb749c96b1e280536782f5bda0be32638ab3a78e1ec4af8f7ae2e01 not found: ID does not exist" containerID="62352d1abfb749c96b1e280536782f5bda0be32638ab3a78e1ec4af8f7ae2e01" Jul 21 21:19:44.941442 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:19:44.941394 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"62352d1abfb749c96b1e280536782f5bda0be32638ab3a78e1ec4af8f7ae2e01"} err="failed to get container status \"62352d1abfb749c96b1e280536782f5bda0be32638ab3a78e1ec4af8f7ae2e01\": rpc error: code = NotFound desc = could not find container \"62352d1abfb749c96b1e280536782f5bda0be32638ab3a78e1ec4af8f7ae2e01\": container with ID starting with 62352d1abfb749c96b1e280536782f5bda0be32638ab3a78e1ec4af8f7ae2e01 not found: ID does not exist" Jul 21 21:19:44.949440 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:19:44.949414 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-6gqzm"] Jul 21 21:19:44.953564 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:19:44.953518 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-6gqzm"] Jul 21 21:19:46.745886 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:19:46.745842 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8a7d2814-c13b-4fc4-bd0d-9739d5bab9d4" path="/var/lib/kubelet/pods/8a7d2814-c13b-4fc4-bd0d-9739d5bab9d4/volumes" Jul 21 21:20:43.077761 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:20:43.077728 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-zl6lh"] Jul 21 21:20:43.078139 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:20:43.078066 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="23c11dc8-340e-413d-9cfa-19db16bb4708" containerName="cleanup" Jul 21 21:20:43.078139 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:20:43.078076 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="23c11dc8-340e-413d-9cfa-19db16bb4708" containerName="cleanup" Jul 21 21:20:43.078139 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:20:43.078085 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8a7d2814-c13b-4fc4-bd0d-9739d5bab9d4" containerName="manager" Jul 21 21:20:43.078139 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:20:43.078091 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="8a7d2814-c13b-4fc4-bd0d-9739d5bab9d4" containerName="manager" Jul 21 21:20:43.078265 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:20:43.078152 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="8a7d2814-c13b-4fc4-bd0d-9739d5bab9d4" containerName="manager" Jul 21 21:20:43.078265 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:20:43.078161 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="23c11dc8-340e-413d-9cfa-19db16bb4708" containerName="cleanup" Jul 21 21:20:43.081324 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:20:43.081305 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-zl6lh" Jul 21 21:20:43.083928 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:20:43.083907 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-dpnx5\"" Jul 21 21:20:43.094593 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:20:43.094574 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-zl6lh"] Jul 21 21:20:43.201752 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:20:43.201720 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/2e172821-8e5a-4f3d-87cc-8742d612b230-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-zl6lh\" (UID: \"2e172821-8e5a-4f3d-87cc-8742d612b230\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-zl6lh" Jul 21 21:20:43.201902 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:20:43.201771 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dd675\" (UniqueName: \"kubernetes.io/projected/2e172821-8e5a-4f3d-87cc-8742d612b230-kube-api-access-dd675\") pod \"kuadrant-operator-controller-manager-86878c769f-zl6lh\" (UID: \"2e172821-8e5a-4f3d-87cc-8742d612b230\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-zl6lh" Jul 21 21:20:43.302404 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:20:43.302368 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/2e172821-8e5a-4f3d-87cc-8742d612b230-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-zl6lh\" (UID: \"2e172821-8e5a-4f3d-87cc-8742d612b230\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-zl6lh" Jul 21 21:20:43.302579 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:20:43.302419 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dd675\" (UniqueName: \"kubernetes.io/projected/2e172821-8e5a-4f3d-87cc-8742d612b230-kube-api-access-dd675\") pod \"kuadrant-operator-controller-manager-86878c769f-zl6lh\" (UID: \"2e172821-8e5a-4f3d-87cc-8742d612b230\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-zl6lh" Jul 21 21:20:43.302770 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:20:43.302743 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/2e172821-8e5a-4f3d-87cc-8742d612b230-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-zl6lh\" (UID: \"2e172821-8e5a-4f3d-87cc-8742d612b230\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-zl6lh" Jul 21 21:20:43.312499 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:20:43.312473 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dd675\" (UniqueName: \"kubernetes.io/projected/2e172821-8e5a-4f3d-87cc-8742d612b230-kube-api-access-dd675\") pod \"kuadrant-operator-controller-manager-86878c769f-zl6lh\" (UID: \"2e172821-8e5a-4f3d-87cc-8742d612b230\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-zl6lh" Jul 21 21:20:43.391635 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:20:43.391607 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-zl6lh" Jul 21 21:20:43.531555 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:20:43.531519 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-zl6lh"] Jul 21 21:20:43.533888 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:20:43.533859 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2e172821_8e5a_4f3d_87cc_8742d612b230.slice/crio-bd4bbfc472ae8b65bc81318c8007132baf42c800b602278a8b72a0b072856236 WatchSource:0}: Error finding container bd4bbfc472ae8b65bc81318c8007132baf42c800b602278a8b72a0b072856236: Status 404 returned error can't find the container with id bd4bbfc472ae8b65bc81318c8007132baf42c800b602278a8b72a0b072856236 Jul 21 21:20:44.207688 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:20:44.207649 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-zl6lh" event={"ID":"2e172821-8e5a-4f3d-87cc-8742d612b230","Type":"ContainerStarted","Data":"3de406234a698baff47297d38d912a74c12fb012188ed69b6e6bf50d5b845799"} Jul 21 21:20:44.208082 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:20:44.207711 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-zl6lh" event={"ID":"2e172821-8e5a-4f3d-87cc-8742d612b230","Type":"ContainerStarted","Data":"bd4bbfc472ae8b65bc81318c8007132baf42c800b602278a8b72a0b072856236"} Jul 21 21:20:44.208082 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:20:44.207782 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-zl6lh" Jul 21 21:20:44.230776 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:20:44.230723 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-zl6lh" podStartSLOduration=1.23070562 podStartE2EDuration="1.23070562s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 21:20:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 21:20:44.229678475 +0000 UTC m=+1202.164428153" watchObservedRunningTime="2026-07-21 21:20:44.23070562 +0000 UTC m=+1202.165455285" Jul 21 21:20:55.214874 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:20:55.214841 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-zl6lh" Jul 21 21:23:20.971181 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:23:20.971151 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-a-d178517c-kserve-696b8cd8c5-nvgjn"] Jul 21 21:23:20.974583 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:23:20.974565 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-d178517c-kserve-696b8cd8c5-nvgjn" Jul 21 21:23:20.977379 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:23:20.977357 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-a-d178517c-kserve-self-signed-certs\"" Jul 21 21:23:20.996175 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:23:20.996148 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-d178517c-kserve-696b8cd8c5-nvgjn"] Jul 21 21:23:21.109618 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:23:21.109585 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/86ddf7fc-a390-4615-97f9-aace0146930e-tls-certs\") pod \"e2e-same-id-a-d178517c-kserve-696b8cd8c5-nvgjn\" (UID: \"86ddf7fc-a390-4615-97f9-aace0146930e\") " pod="llm/e2e-same-id-a-d178517c-kserve-696b8cd8c5-nvgjn" Jul 21 21:23:21.109775 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:23:21.109680 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/86ddf7fc-a390-4615-97f9-aace0146930e-tmp-dir\") pod \"e2e-same-id-a-d178517c-kserve-696b8cd8c5-nvgjn\" (UID: \"86ddf7fc-a390-4615-97f9-aace0146930e\") " pod="llm/e2e-same-id-a-d178517c-kserve-696b8cd8c5-nvgjn" Jul 21 21:23:21.109775 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:23:21.109732 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/86ddf7fc-a390-4615-97f9-aace0146930e-home\") pod \"e2e-same-id-a-d178517c-kserve-696b8cd8c5-nvgjn\" (UID: \"86ddf7fc-a390-4615-97f9-aace0146930e\") " pod="llm/e2e-same-id-a-d178517c-kserve-696b8cd8c5-nvgjn" Jul 21 21:23:21.109775 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:23:21.109765 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/86ddf7fc-a390-4615-97f9-aace0146930e-dshm\") pod \"e2e-same-id-a-d178517c-kserve-696b8cd8c5-nvgjn\" (UID: \"86ddf7fc-a390-4615-97f9-aace0146930e\") " pod="llm/e2e-same-id-a-d178517c-kserve-696b8cd8c5-nvgjn" Jul 21 21:23:21.109939 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:23:21.109784 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m4vz7\" (UniqueName: \"kubernetes.io/projected/86ddf7fc-a390-4615-97f9-aace0146930e-kube-api-access-m4vz7\") pod \"e2e-same-id-a-d178517c-kserve-696b8cd8c5-nvgjn\" (UID: \"86ddf7fc-a390-4615-97f9-aace0146930e\") " pod="llm/e2e-same-id-a-d178517c-kserve-696b8cd8c5-nvgjn" Jul 21 21:23:21.109939 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:23:21.109804 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/86ddf7fc-a390-4615-97f9-aace0146930e-model-cache\") pod \"e2e-same-id-a-d178517c-kserve-696b8cd8c5-nvgjn\" (UID: \"86ddf7fc-a390-4615-97f9-aace0146930e\") " pod="llm/e2e-same-id-a-d178517c-kserve-696b8cd8c5-nvgjn" Jul 21 21:23:21.211037 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:23:21.211001 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/86ddf7fc-a390-4615-97f9-aace0146930e-tmp-dir\") pod \"e2e-same-id-a-d178517c-kserve-696b8cd8c5-nvgjn\" (UID: \"86ddf7fc-a390-4615-97f9-aace0146930e\") " pod="llm/e2e-same-id-a-d178517c-kserve-696b8cd8c5-nvgjn" Jul 21 21:23:21.211037 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:23:21.211041 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/86ddf7fc-a390-4615-97f9-aace0146930e-home\") pod \"e2e-same-id-a-d178517c-kserve-696b8cd8c5-nvgjn\" (UID: \"86ddf7fc-a390-4615-97f9-aace0146930e\") " pod="llm/e2e-same-id-a-d178517c-kserve-696b8cd8c5-nvgjn" Jul 21 21:23:21.211234 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:23:21.211065 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/86ddf7fc-a390-4615-97f9-aace0146930e-dshm\") pod \"e2e-same-id-a-d178517c-kserve-696b8cd8c5-nvgjn\" (UID: \"86ddf7fc-a390-4615-97f9-aace0146930e\") " pod="llm/e2e-same-id-a-d178517c-kserve-696b8cd8c5-nvgjn" Jul 21 21:23:21.211234 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:23:21.211086 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m4vz7\" (UniqueName: \"kubernetes.io/projected/86ddf7fc-a390-4615-97f9-aace0146930e-kube-api-access-m4vz7\") pod \"e2e-same-id-a-d178517c-kserve-696b8cd8c5-nvgjn\" (UID: \"86ddf7fc-a390-4615-97f9-aace0146930e\") " pod="llm/e2e-same-id-a-d178517c-kserve-696b8cd8c5-nvgjn" Jul 21 21:23:21.211234 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:23:21.211208 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/86ddf7fc-a390-4615-97f9-aace0146930e-model-cache\") pod \"e2e-same-id-a-d178517c-kserve-696b8cd8c5-nvgjn\" (UID: \"86ddf7fc-a390-4615-97f9-aace0146930e\") " pod="llm/e2e-same-id-a-d178517c-kserve-696b8cd8c5-nvgjn" Jul 21 21:23:21.211381 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:23:21.211341 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/86ddf7fc-a390-4615-97f9-aace0146930e-tls-certs\") pod \"e2e-same-id-a-d178517c-kserve-696b8cd8c5-nvgjn\" (UID: \"86ddf7fc-a390-4615-97f9-aace0146930e\") " pod="llm/e2e-same-id-a-d178517c-kserve-696b8cd8c5-nvgjn" Jul 21 21:23:21.211440 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:23:21.211422 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/86ddf7fc-a390-4615-97f9-aace0146930e-tmp-dir\") pod \"e2e-same-id-a-d178517c-kserve-696b8cd8c5-nvgjn\" (UID: \"86ddf7fc-a390-4615-97f9-aace0146930e\") " pod="llm/e2e-same-id-a-d178517c-kserve-696b8cd8c5-nvgjn" Jul 21 21:23:21.211510 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:23:21.211491 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/86ddf7fc-a390-4615-97f9-aace0146930e-home\") pod \"e2e-same-id-a-d178517c-kserve-696b8cd8c5-nvgjn\" (UID: \"86ddf7fc-a390-4615-97f9-aace0146930e\") " pod="llm/e2e-same-id-a-d178517c-kserve-696b8cd8c5-nvgjn" Jul 21 21:23:21.211654 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:23:21.211633 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/86ddf7fc-a390-4615-97f9-aace0146930e-model-cache\") pod \"e2e-same-id-a-d178517c-kserve-696b8cd8c5-nvgjn\" (UID: \"86ddf7fc-a390-4615-97f9-aace0146930e\") " pod="llm/e2e-same-id-a-d178517c-kserve-696b8cd8c5-nvgjn" Jul 21 21:23:21.213226 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:23:21.213204 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/86ddf7fc-a390-4615-97f9-aace0146930e-dshm\") pod \"e2e-same-id-a-d178517c-kserve-696b8cd8c5-nvgjn\" (UID: \"86ddf7fc-a390-4615-97f9-aace0146930e\") " pod="llm/e2e-same-id-a-d178517c-kserve-696b8cd8c5-nvgjn" Jul 21 21:23:21.213688 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:23:21.213673 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/86ddf7fc-a390-4615-97f9-aace0146930e-tls-certs\") pod \"e2e-same-id-a-d178517c-kserve-696b8cd8c5-nvgjn\" (UID: \"86ddf7fc-a390-4615-97f9-aace0146930e\") " pod="llm/e2e-same-id-a-d178517c-kserve-696b8cd8c5-nvgjn" Jul 21 21:23:21.222944 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:23:21.222886 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m4vz7\" (UniqueName: \"kubernetes.io/projected/86ddf7fc-a390-4615-97f9-aace0146930e-kube-api-access-m4vz7\") pod \"e2e-same-id-a-d178517c-kserve-696b8cd8c5-nvgjn\" (UID: \"86ddf7fc-a390-4615-97f9-aace0146930e\") " pod="llm/e2e-same-id-a-d178517c-kserve-696b8cd8c5-nvgjn" Jul 21 21:23:21.283569 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:23:21.283522 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-d178517c-kserve-696b8cd8c5-nvgjn" Jul 21 21:23:21.415565 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:23:21.415522 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-d178517c-kserve-696b8cd8c5-nvgjn"] Jul 21 21:23:21.417246 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:23:21.417219 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod86ddf7fc_a390_4615_97f9_aace0146930e.slice/crio-2c6b69651c241137008c7111eec7692ca22c5345e0117a71c781cd724c79d5a8 WatchSource:0}: Error finding container 2c6b69651c241137008c7111eec7692ca22c5345e0117a71c781cd724c79d5a8: Status 404 returned error can't find the container with id 2c6b69651c241137008c7111eec7692ca22c5345e0117a71c781cd724c79d5a8 Jul 21 21:23:21.952032 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:23:21.952003 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-d178517c-kserve-696b8cd8c5-nvgjn" event={"ID":"86ddf7fc-a390-4615-97f9-aace0146930e","Type":"ContainerStarted","Data":"69dcdcccc209d66afec5982c6dae7177006cfc951cf8aca7c65ad3bfadc64d10"} Jul 21 21:23:21.952210 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:23:21.952042 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-d178517c-kserve-696b8cd8c5-nvgjn" event={"ID":"86ddf7fc-a390-4615-97f9-aace0146930e","Type":"ContainerStarted","Data":"2c6b69651c241137008c7111eec7692ca22c5345e0117a71c781cd724c79d5a8"} Jul 21 21:23:21.970043 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:23:21.970000 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-same-id-a-d178517c-kserve-696b8cd8c5-nvgjn" podStartSLOduration=1.969985767 podStartE2EDuration="1.969985767s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 21:23:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 21:23:21.968896179 +0000 UTC m=+1359.903645847" watchObservedRunningTime="2026-07-21 21:23:21.969985767 +0000 UTC m=+1359.904735471" Jul 21 21:23:31.284142 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:23:31.284047 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-a-d178517c-kserve-696b8cd8c5-nvgjn" Jul 21 21:23:31.284142 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:23:31.284100 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-a-d178517c-kserve-696b8cd8c5-nvgjn" Jul 21 21:23:31.296414 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:23:31.296390 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-a-d178517c-kserve-696b8cd8c5-nvgjn" Jul 21 21:23:32.016573 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:23:32.016542 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-a-d178517c-kserve-696b8cd8c5-nvgjn" Jul 21 21:23:53.428016 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:23:53.427977 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-d178517c-kserve-696b8cd8c5-nvgjn"] Jul 21 21:24:08.449749 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:24:08.449687 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-same-id-a-d178517c-kserve-696b8cd8c5-nvgjn" podUID="86ddf7fc-a390-4615-97f9-aace0146930e" containerName="main" containerID="cri-o://69dcdcccc209d66afec5982c6dae7177006cfc951cf8aca7c65ad3bfadc64d10" gracePeriod=45 Jul 21 21:24:08.702044 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:24:08.701991 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-d178517c-kserve-696b8cd8c5-nvgjn" Jul 21 21:24:08.839573 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:24:08.839546 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/86ddf7fc-a390-4615-97f9-aace0146930e-tmp-dir\") pod \"86ddf7fc-a390-4615-97f9-aace0146930e\" (UID: \"86ddf7fc-a390-4615-97f9-aace0146930e\") " Jul 21 21:24:08.839728 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:24:08.839582 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/86ddf7fc-a390-4615-97f9-aace0146930e-home\") pod \"86ddf7fc-a390-4615-97f9-aace0146930e\" (UID: \"86ddf7fc-a390-4615-97f9-aace0146930e\") " Jul 21 21:24:08.839728 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:24:08.839625 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/86ddf7fc-a390-4615-97f9-aace0146930e-model-cache\") pod \"86ddf7fc-a390-4615-97f9-aace0146930e\" (UID: \"86ddf7fc-a390-4615-97f9-aace0146930e\") " Jul 21 21:24:08.839728 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:24:08.839686 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/86ddf7fc-a390-4615-97f9-aace0146930e-tls-certs\") pod \"86ddf7fc-a390-4615-97f9-aace0146930e\" (UID: \"86ddf7fc-a390-4615-97f9-aace0146930e\") " Jul 21 21:24:08.839728 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:24:08.839700 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/86ddf7fc-a390-4615-97f9-aace0146930e-dshm\") pod \"86ddf7fc-a390-4615-97f9-aace0146930e\" (UID: \"86ddf7fc-a390-4615-97f9-aace0146930e\") " Jul 21 21:24:08.839728 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:24:08.839718 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-m4vz7\" (UniqueName: \"kubernetes.io/projected/86ddf7fc-a390-4615-97f9-aace0146930e-kube-api-access-m4vz7\") pod \"86ddf7fc-a390-4615-97f9-aace0146930e\" (UID: \"86ddf7fc-a390-4615-97f9-aace0146930e\") " Jul 21 21:24:08.839982 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:24:08.839821 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/86ddf7fc-a390-4615-97f9-aace0146930e-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "86ddf7fc-a390-4615-97f9-aace0146930e" (UID: "86ddf7fc-a390-4615-97f9-aace0146930e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 21:24:08.839982 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:24:08.839836 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/86ddf7fc-a390-4615-97f9-aace0146930e-home" (OuterVolumeSpecName: "home") pod "86ddf7fc-a390-4615-97f9-aace0146930e" (UID: "86ddf7fc-a390-4615-97f9-aace0146930e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 21:24:08.840087 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:24:08.840008 2575 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/86ddf7fc-a390-4615-97f9-aace0146930e-tmp-dir\") on node \"ip-10-0-140-97.ec2.internal\" DevicePath \"\"" Jul 21 21:24:08.840087 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:24:08.840026 2575 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/86ddf7fc-a390-4615-97f9-aace0146930e-home\") on node \"ip-10-0-140-97.ec2.internal\" DevicePath \"\"" Jul 21 21:24:08.840087 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:24:08.840015 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/86ddf7fc-a390-4615-97f9-aace0146930e-model-cache" (OuterVolumeSpecName: "model-cache") pod "86ddf7fc-a390-4615-97f9-aace0146930e" (UID: "86ddf7fc-a390-4615-97f9-aace0146930e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 21:24:08.841897 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:24:08.841868 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/86ddf7fc-a390-4615-97f9-aace0146930e-kube-api-access-m4vz7" (OuterVolumeSpecName: "kube-api-access-m4vz7") pod "86ddf7fc-a390-4615-97f9-aace0146930e" (UID: "86ddf7fc-a390-4615-97f9-aace0146930e"). InnerVolumeSpecName "kube-api-access-m4vz7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 21 21:24:08.842008 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:24:08.841910 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/86ddf7fc-a390-4615-97f9-aace0146930e-dshm" (OuterVolumeSpecName: "dshm") pod "86ddf7fc-a390-4615-97f9-aace0146930e" (UID: "86ddf7fc-a390-4615-97f9-aace0146930e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 21:24:08.842008 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:24:08.841926 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/86ddf7fc-a390-4615-97f9-aace0146930e-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "86ddf7fc-a390-4615-97f9-aace0146930e" (UID: "86ddf7fc-a390-4615-97f9-aace0146930e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 21 21:24:08.941237 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:24:08.941208 2575 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/86ddf7fc-a390-4615-97f9-aace0146930e-model-cache\") on node \"ip-10-0-140-97.ec2.internal\" DevicePath \"\"" Jul 21 21:24:08.941237 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:24:08.941237 2575 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/86ddf7fc-a390-4615-97f9-aace0146930e-tls-certs\") on node \"ip-10-0-140-97.ec2.internal\" DevicePath \"\"" Jul 21 21:24:08.941439 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:24:08.941249 2575 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/86ddf7fc-a390-4615-97f9-aace0146930e-dshm\") on node \"ip-10-0-140-97.ec2.internal\" DevicePath \"\"" Jul 21 21:24:08.941439 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:24:08.941262 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m4vz7\" (UniqueName: \"kubernetes.io/projected/86ddf7fc-a390-4615-97f9-aace0146930e-kube-api-access-m4vz7\") on node \"ip-10-0-140-97.ec2.internal\" DevicePath \"\"" Jul 21 21:24:09.164790 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:24:09.164759 2575 generic.go:356] "Generic (PLEG): container finished" podID="86ddf7fc-a390-4615-97f9-aace0146930e" containerID="69dcdcccc209d66afec5982c6dae7177006cfc951cf8aca7c65ad3bfadc64d10" exitCode=0 Jul 21 21:24:09.164950 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:24:09.164801 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-d178517c-kserve-696b8cd8c5-nvgjn" Jul 21 21:24:09.164950 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:24:09.164817 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-d178517c-kserve-696b8cd8c5-nvgjn" event={"ID":"86ddf7fc-a390-4615-97f9-aace0146930e","Type":"ContainerDied","Data":"69dcdcccc209d66afec5982c6dae7177006cfc951cf8aca7c65ad3bfadc64d10"} Jul 21 21:24:09.164950 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:24:09.164849 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-d178517c-kserve-696b8cd8c5-nvgjn" event={"ID":"86ddf7fc-a390-4615-97f9-aace0146930e","Type":"ContainerDied","Data":"2c6b69651c241137008c7111eec7692ca22c5345e0117a71c781cd724c79d5a8"} Jul 21 21:24:09.164950 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:24:09.164865 2575 scope.go:117] "RemoveContainer" containerID="69dcdcccc209d66afec5982c6dae7177006cfc951cf8aca7c65ad3bfadc64d10" Jul 21 21:24:09.173707 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:24:09.173687 2575 scope.go:117] "RemoveContainer" containerID="69dcdcccc209d66afec5982c6dae7177006cfc951cf8aca7c65ad3bfadc64d10" Jul 21 21:24:09.173970 ip-10-0-140-97 kubenswrapper[2575]: E0721 21:24:09.173951 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"69dcdcccc209d66afec5982c6dae7177006cfc951cf8aca7c65ad3bfadc64d10\": container with ID starting with 69dcdcccc209d66afec5982c6dae7177006cfc951cf8aca7c65ad3bfadc64d10 not found: ID does not exist" containerID="69dcdcccc209d66afec5982c6dae7177006cfc951cf8aca7c65ad3bfadc64d10" Jul 21 21:24:09.174018 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:24:09.173977 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"69dcdcccc209d66afec5982c6dae7177006cfc951cf8aca7c65ad3bfadc64d10"} err="failed to get container status \"69dcdcccc209d66afec5982c6dae7177006cfc951cf8aca7c65ad3bfadc64d10\": rpc error: code = NotFound desc = could not find container \"69dcdcccc209d66afec5982c6dae7177006cfc951cf8aca7c65ad3bfadc64d10\": container with ID starting with 69dcdcccc209d66afec5982c6dae7177006cfc951cf8aca7c65ad3bfadc64d10 not found: ID does not exist" Jul 21 21:24:09.187950 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:24:09.187927 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-d178517c-kserve-696b8cd8c5-nvgjn"] Jul 21 21:24:09.190954 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:24:09.190934 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-a-d178517c-kserve-696b8cd8c5-nvgjn"] Jul 21 21:24:10.741218 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:24:10.741179 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="86ddf7fc-a390-4615-97f9-aace0146930e" path="/var/lib/kubelet/pods/86ddf7fc-a390-4615-97f9-aace0146930e/volumes" Jul 21 21:27:38.584664 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:27:38.584583 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-82b9182c-openshift-default-b78844dfc-k5472"] Jul 21 21:27:38.587291 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:27:38.585143 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="86ddf7fc-a390-4615-97f9-aace0146930e" containerName="main" Jul 21 21:27:38.587291 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:27:38.585161 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="86ddf7fc-a390-4615-97f9-aace0146930e" containerName="main" Jul 21 21:27:38.587291 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:27:38.585223 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="86ddf7fc-a390-4615-97f9-aace0146930e" containerName="main" Jul 21 21:27:38.588403 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:27:38.588385 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-82b9182c-openshift-default-b78844dfc-k5472" Jul 21 21:27:38.592874 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:27:38.592858 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-82b9182c-openshift-default-dockercfg-6hjqf\"" Jul 21 21:27:38.619367 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:27:38.619342 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-82b9182c-openshift-default-b78844dfc-k5472"] Jul 21 21:27:38.712055 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:27:38.712015 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ebda7857-1799-4934-8401-eaf8a77ac7e8-workload-socket\") pod \"e2e-ait-82b9182c-openshift-default-b78844dfc-k5472\" (UID: \"ebda7857-1799-4934-8401-eaf8a77ac7e8\") " pod="openshift-ingress/e2e-ait-82b9182c-openshift-default-b78844dfc-k5472" Jul 21 21:27:38.712055 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:27:38.712058 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ebda7857-1799-4934-8401-eaf8a77ac7e8-istio-data\") pod \"e2e-ait-82b9182c-openshift-default-b78844dfc-k5472\" (UID: \"ebda7857-1799-4934-8401-eaf8a77ac7e8\") " pod="openshift-ingress/e2e-ait-82b9182c-openshift-default-b78844dfc-k5472" Jul 21 21:27:38.712294 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:27:38.712082 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ebda7857-1799-4934-8401-eaf8a77ac7e8-istio-token\") pod \"e2e-ait-82b9182c-openshift-default-b78844dfc-k5472\" (UID: \"ebda7857-1799-4934-8401-eaf8a77ac7e8\") " pod="openshift-ingress/e2e-ait-82b9182c-openshift-default-b78844dfc-k5472" Jul 21 21:27:38.712294 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:27:38.712146 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ebda7857-1799-4934-8401-eaf8a77ac7e8-istio-envoy\") pod \"e2e-ait-82b9182c-openshift-default-b78844dfc-k5472\" (UID: \"ebda7857-1799-4934-8401-eaf8a77ac7e8\") " pod="openshift-ingress/e2e-ait-82b9182c-openshift-default-b78844dfc-k5472" Jul 21 21:27:38.712294 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:27:38.712185 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ebda7857-1799-4934-8401-eaf8a77ac7e8-istiod-ca-cert\") pod \"e2e-ait-82b9182c-openshift-default-b78844dfc-k5472\" (UID: \"ebda7857-1799-4934-8401-eaf8a77ac7e8\") " pod="openshift-ingress/e2e-ait-82b9182c-openshift-default-b78844dfc-k5472" Jul 21 21:27:38.712294 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:27:38.712214 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2f66s\" (UniqueName: \"kubernetes.io/projected/ebda7857-1799-4934-8401-eaf8a77ac7e8-kube-api-access-2f66s\") pod \"e2e-ait-82b9182c-openshift-default-b78844dfc-k5472\" (UID: \"ebda7857-1799-4934-8401-eaf8a77ac7e8\") " pod="openshift-ingress/e2e-ait-82b9182c-openshift-default-b78844dfc-k5472" Jul 21 21:27:38.712456 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:27:38.712335 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ebda7857-1799-4934-8401-eaf8a77ac7e8-workload-certs\") pod \"e2e-ait-82b9182c-openshift-default-b78844dfc-k5472\" (UID: \"ebda7857-1799-4934-8401-eaf8a77ac7e8\") " pod="openshift-ingress/e2e-ait-82b9182c-openshift-default-b78844dfc-k5472" Jul 21 21:27:38.712456 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:27:38.712390 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ebda7857-1799-4934-8401-eaf8a77ac7e8-credential-socket\") pod \"e2e-ait-82b9182c-openshift-default-b78844dfc-k5472\" (UID: \"ebda7857-1799-4934-8401-eaf8a77ac7e8\") " pod="openshift-ingress/e2e-ait-82b9182c-openshift-default-b78844dfc-k5472" Jul 21 21:27:38.712456 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:27:38.712406 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ebda7857-1799-4934-8401-eaf8a77ac7e8-istio-podinfo\") pod \"e2e-ait-82b9182c-openshift-default-b78844dfc-k5472\" (UID: \"ebda7857-1799-4934-8401-eaf8a77ac7e8\") " pod="openshift-ingress/e2e-ait-82b9182c-openshift-default-b78844dfc-k5472" Jul 21 21:27:38.813355 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:27:38.813325 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ebda7857-1799-4934-8401-eaf8a77ac7e8-workload-certs\") pod \"e2e-ait-82b9182c-openshift-default-b78844dfc-k5472\" (UID: \"ebda7857-1799-4934-8401-eaf8a77ac7e8\") " pod="openshift-ingress/e2e-ait-82b9182c-openshift-default-b78844dfc-k5472" Jul 21 21:27:38.813508 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:27:38.813391 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ebda7857-1799-4934-8401-eaf8a77ac7e8-credential-socket\") pod \"e2e-ait-82b9182c-openshift-default-b78844dfc-k5472\" (UID: \"ebda7857-1799-4934-8401-eaf8a77ac7e8\") " pod="openshift-ingress/e2e-ait-82b9182c-openshift-default-b78844dfc-k5472" Jul 21 21:27:38.813508 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:27:38.813408 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ebda7857-1799-4934-8401-eaf8a77ac7e8-istio-podinfo\") pod \"e2e-ait-82b9182c-openshift-default-b78844dfc-k5472\" (UID: \"ebda7857-1799-4934-8401-eaf8a77ac7e8\") " pod="openshift-ingress/e2e-ait-82b9182c-openshift-default-b78844dfc-k5472" Jul 21 21:27:38.813641 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:27:38.813594 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ebda7857-1799-4934-8401-eaf8a77ac7e8-workload-socket\") pod \"e2e-ait-82b9182c-openshift-default-b78844dfc-k5472\" (UID: \"ebda7857-1799-4934-8401-eaf8a77ac7e8\") " pod="openshift-ingress/e2e-ait-82b9182c-openshift-default-b78844dfc-k5472" Jul 21 21:27:38.813697 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:27:38.813641 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ebda7857-1799-4934-8401-eaf8a77ac7e8-istio-data\") pod \"e2e-ait-82b9182c-openshift-default-b78844dfc-k5472\" (UID: \"ebda7857-1799-4934-8401-eaf8a77ac7e8\") " pod="openshift-ingress/e2e-ait-82b9182c-openshift-default-b78844dfc-k5472" Jul 21 21:27:38.813697 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:27:38.813684 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ebda7857-1799-4934-8401-eaf8a77ac7e8-istio-token\") pod \"e2e-ait-82b9182c-openshift-default-b78844dfc-k5472\" (UID: \"ebda7857-1799-4934-8401-eaf8a77ac7e8\") " pod="openshift-ingress/e2e-ait-82b9182c-openshift-default-b78844dfc-k5472" Jul 21 21:27:38.813798 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:27:38.813723 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ebda7857-1799-4934-8401-eaf8a77ac7e8-istio-envoy\") pod \"e2e-ait-82b9182c-openshift-default-b78844dfc-k5472\" (UID: \"ebda7857-1799-4934-8401-eaf8a77ac7e8\") " pod="openshift-ingress/e2e-ait-82b9182c-openshift-default-b78844dfc-k5472" Jul 21 21:27:38.813798 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:27:38.813751 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ebda7857-1799-4934-8401-eaf8a77ac7e8-istiod-ca-cert\") pod \"e2e-ait-82b9182c-openshift-default-b78844dfc-k5472\" (UID: \"ebda7857-1799-4934-8401-eaf8a77ac7e8\") " pod="openshift-ingress/e2e-ait-82b9182c-openshift-default-b78844dfc-k5472" Jul 21 21:27:38.813798 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:27:38.813783 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ebda7857-1799-4934-8401-eaf8a77ac7e8-workload-certs\") pod \"e2e-ait-82b9182c-openshift-default-b78844dfc-k5472\" (UID: \"ebda7857-1799-4934-8401-eaf8a77ac7e8\") " pod="openshift-ingress/e2e-ait-82b9182c-openshift-default-b78844dfc-k5472" Jul 21 21:27:38.813798 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:27:38.813791 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2f66s\" (UniqueName: \"kubernetes.io/projected/ebda7857-1799-4934-8401-eaf8a77ac7e8-kube-api-access-2f66s\") pod \"e2e-ait-82b9182c-openshift-default-b78844dfc-k5472\" (UID: \"ebda7857-1799-4934-8401-eaf8a77ac7e8\") " pod="openshift-ingress/e2e-ait-82b9182c-openshift-default-b78844dfc-k5472" Jul 21 21:27:38.813997 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:27:38.813809 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ebda7857-1799-4934-8401-eaf8a77ac7e8-credential-socket\") pod \"e2e-ait-82b9182c-openshift-default-b78844dfc-k5472\" (UID: \"ebda7857-1799-4934-8401-eaf8a77ac7e8\") " pod="openshift-ingress/e2e-ait-82b9182c-openshift-default-b78844dfc-k5472" Jul 21 21:27:38.814116 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:27:38.814093 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ebda7857-1799-4934-8401-eaf8a77ac7e8-workload-socket\") pod \"e2e-ait-82b9182c-openshift-default-b78844dfc-k5472\" (UID: \"ebda7857-1799-4934-8401-eaf8a77ac7e8\") " pod="openshift-ingress/e2e-ait-82b9182c-openshift-default-b78844dfc-k5472" Jul 21 21:27:38.814264 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:27:38.814245 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ebda7857-1799-4934-8401-eaf8a77ac7e8-istio-data\") pod \"e2e-ait-82b9182c-openshift-default-b78844dfc-k5472\" (UID: \"ebda7857-1799-4934-8401-eaf8a77ac7e8\") " pod="openshift-ingress/e2e-ait-82b9182c-openshift-default-b78844dfc-k5472" Jul 21 21:27:38.814716 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:27:38.814694 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ebda7857-1799-4934-8401-eaf8a77ac7e8-istiod-ca-cert\") pod \"e2e-ait-82b9182c-openshift-default-b78844dfc-k5472\" (UID: \"ebda7857-1799-4934-8401-eaf8a77ac7e8\") " pod="openshift-ingress/e2e-ait-82b9182c-openshift-default-b78844dfc-k5472" Jul 21 21:27:38.815899 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:27:38.815882 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ebda7857-1799-4934-8401-eaf8a77ac7e8-istio-podinfo\") pod \"e2e-ait-82b9182c-openshift-default-b78844dfc-k5472\" (UID: \"ebda7857-1799-4934-8401-eaf8a77ac7e8\") " pod="openshift-ingress/e2e-ait-82b9182c-openshift-default-b78844dfc-k5472" Jul 21 21:27:38.815991 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:27:38.815973 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ebda7857-1799-4934-8401-eaf8a77ac7e8-istio-envoy\") pod \"e2e-ait-82b9182c-openshift-default-b78844dfc-k5472\" (UID: \"ebda7857-1799-4934-8401-eaf8a77ac7e8\") " pod="openshift-ingress/e2e-ait-82b9182c-openshift-default-b78844dfc-k5472" Jul 21 21:27:38.821875 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:27:38.821856 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2f66s\" (UniqueName: \"kubernetes.io/projected/ebda7857-1799-4934-8401-eaf8a77ac7e8-kube-api-access-2f66s\") pod \"e2e-ait-82b9182c-openshift-default-b78844dfc-k5472\" (UID: \"ebda7857-1799-4934-8401-eaf8a77ac7e8\") " pod="openshift-ingress/e2e-ait-82b9182c-openshift-default-b78844dfc-k5472" Jul 21 21:27:38.821952 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:27:38.821888 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ebda7857-1799-4934-8401-eaf8a77ac7e8-istio-token\") pod \"e2e-ait-82b9182c-openshift-default-b78844dfc-k5472\" (UID: \"ebda7857-1799-4934-8401-eaf8a77ac7e8\") " pod="openshift-ingress/e2e-ait-82b9182c-openshift-default-b78844dfc-k5472" Jul 21 21:27:38.898124 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:27:38.898099 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-82b9182c-openshift-default-b78844dfc-k5472" Jul 21 21:27:39.035424 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:27:39.035396 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-82b9182c-openshift-default-b78844dfc-k5472"] Jul 21 21:27:39.038101 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:27:39.038072 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podebda7857_1799_4934_8401_eaf8a77ac7e8.slice/crio-ec77b337fc0f701f410004347152523fb411b6c28c4d0a716db326056339c289 WatchSource:0}: Error finding container ec77b337fc0f701f410004347152523fb411b6c28c4d0a716db326056339c289: Status 404 returned error can't find the container with id ec77b337fc0f701f410004347152523fb411b6c28c4d0a716db326056339c289 Jul 21 21:27:39.040697 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:27:39.040659 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 21 21:27:39.040781 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:27:39.040742 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 21 21:27:39.040823 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:27:39.040784 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 21 21:27:39.151514 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:27:39.151421 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-82b9182c-openshift-default-b78844dfc-k5472" event={"ID":"ebda7857-1799-4934-8401-eaf8a77ac7e8","Type":"ContainerStarted","Data":"ec081142a724205459d35b4a28cdba8c66a8030d1117d447e38cf32bbd97f37b"} Jul 21 21:27:39.151514 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:27:39.151484 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-82b9182c-openshift-default-b78844dfc-k5472" event={"ID":"ebda7857-1799-4934-8401-eaf8a77ac7e8","Type":"ContainerStarted","Data":"ec77b337fc0f701f410004347152523fb411b6c28c4d0a716db326056339c289"} Jul 21 21:27:39.174151 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:27:39.174051 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-82b9182c-openshift-default-b78844dfc-k5472" podStartSLOduration=1.174034634 podStartE2EDuration="1.174034634s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 21:27:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 21:27:39.173736035 +0000 UTC m=+1617.108485702" watchObservedRunningTime="2026-07-21 21:27:39.174034634 +0000 UTC m=+1617.108784300" Jul 21 21:27:39.899119 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:27:39.899085 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-82b9182c-openshift-default-b78844dfc-k5472" Jul 21 21:27:40.903615 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:27:40.903588 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-82b9182c-openshift-default-b78844dfc-k5472" Jul 21 21:27:41.161721 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:27:41.161650 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-82b9182c-openshift-default-b78844dfc-k5472" Jul 21 21:27:41.162711 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:27:41.162692 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-82b9182c-openshift-default-b78844dfc-k5472" Jul 21 21:27:55.227240 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:27:55.227205 2575 generic.go:356] "Generic (PLEG): container finished" podID="ebda7857-1799-4934-8401-eaf8a77ac7e8" containerID="ec081142a724205459d35b4a28cdba8c66a8030d1117d447e38cf32bbd97f37b" exitCode=0 Jul 21 21:27:55.227648 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:27:55.227278 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-82b9182c-openshift-default-b78844dfc-k5472" event={"ID":"ebda7857-1799-4934-8401-eaf8a77ac7e8","Type":"ContainerDied","Data":"ec081142a724205459d35b4a28cdba8c66a8030d1117d447e38cf32bbd97f37b"} Jul 21 21:27:55.227700 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:27:55.227684 2575 scope.go:117] "RemoveContainer" containerID="ec081142a724205459d35b4a28cdba8c66a8030d1117d447e38cf32bbd97f37b" Jul 21 21:27:55.228682 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:27:55.228655 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 21 21:27:55.228737 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:27:55.228721 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 21 21:27:55.228771 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:27:55.228752 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 21 21:27:56.162144 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:27:56.162111 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-82b9182c-openshift-default-b78844dfc-k5472" Jul 21 21:27:56.234551 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:27:56.234505 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-82b9182c-openshift-default-b78844dfc-k5472" event={"ID":"ebda7857-1799-4934-8401-eaf8a77ac7e8","Type":"ContainerStarted","Data":"29f39edb3be70ca428873c8938789076e45052748e80fffd5daeb8a4ef07b3ba"} Jul 21 21:27:56.898873 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:27:56.898835 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-82b9182c-openshift-default-b78844dfc-k5472" Jul 21 21:27:56.900025 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:27:56.899998 2575 patch_prober.go:28] interesting pod/e2e-ait-82b9182c-openshift-default-b78844dfc-k5472 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" start-of-body= Jul 21 21:27:56.900167 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:27:56.900053 2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-82b9182c-openshift-default-b78844dfc-k5472" podUID="ebda7857-1799-4934-8401-eaf8a77ac7e8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" Jul 21 21:27:57.899437 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:27:57.899404 2575 patch_prober.go:28] interesting pod/e2e-ait-82b9182c-openshift-default-b78844dfc-k5472 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" start-of-body= Jul 21 21:27:57.899827 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:27:57.899457 2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-82b9182c-openshift-default-b78844dfc-k5472" podUID="ebda7857-1799-4934-8401-eaf8a77ac7e8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" Jul 21 21:27:58.899356 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:27:58.899319 2575 patch_prober.go:28] interesting pod/e2e-ait-82b9182c-openshift-default-b78844dfc-k5472 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" start-of-body= Jul 21 21:27:58.899585 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:27:58.899381 2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-82b9182c-openshift-default-b78844dfc-k5472" podUID="ebda7857-1799-4934-8401-eaf8a77ac7e8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" Jul 21 21:27:59.898833 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:27:59.898798 2575 patch_prober.go:28] interesting pod/e2e-ait-82b9182c-openshift-default-b78844dfc-k5472 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" start-of-body= Jul 21 21:27:59.899007 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:27:59.898868 2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-82b9182c-openshift-default-b78844dfc-k5472" podUID="ebda7857-1799-4934-8401-eaf8a77ac7e8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" Jul 21 21:28:00.903062 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:28:00.903035 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-82b9182c-openshift-default-b78844dfc-k5472" Jul 21 21:28:00.903450 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:28:00.903333 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-82b9182c-openshift-default-b78844dfc-k5472" Jul 21 21:28:00.904159 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:28:00.904139 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-82b9182c-openshift-default-b78844dfc-k5472" Jul 21 21:28:06.258190 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:28:06.258158 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-82b9182c-pnhwq"] Jul 21 21:28:06.260815 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:28:06.260798 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-82b9182c-pnhwq" Jul 21 21:28:06.263400 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:28:06.263376 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 21 21:28:06.263400 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:28:06.263376 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9fxhr\"" Jul 21 21:28:06.269829 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:28:06.269800 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-82b9182c-pnhwq"] Jul 21 21:28:06.372002 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:28:06.371967 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/caf180b8-bf82-43ed-8f1c-1c4aa74a081b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-82b9182c-pnhwq\" (UID: \"caf180b8-bf82-43ed-8f1c-1c4aa74a081b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-82b9182c-pnhwq" Jul 21 21:28:06.472957 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:28:06.472921 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/caf180b8-bf82-43ed-8f1c-1c4aa74a081b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-82b9182c-pnhwq\" (UID: \"caf180b8-bf82-43ed-8f1c-1c4aa74a081b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-82b9182c-pnhwq" Jul 21 21:28:06.473581 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:28:06.473563 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/caf180b8-bf82-43ed-8f1c-1c4aa74a081b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-82b9182c-pnhwq\" (UID: \"caf180b8-bf82-43ed-8f1c-1c4aa74a081b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-82b9182c-pnhwq" Jul 21 21:28:06.573944 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:28:06.573872 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-82b9182c-pnhwq" Jul 21 21:28:06.703373 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:28:06.703345 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-82b9182c-pnhwq"] Jul 21 21:28:06.705359 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:28:06.705333 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcaf180b8_bf82_43ed_8f1c_1c4aa74a081b.slice/crio-adeed91562c3b9dcea9d6a1c9bd53f41a3878c08f34b3440bb508f6ac4627d74 WatchSource:0}: Error finding container adeed91562c3b9dcea9d6a1c9bd53f41a3878c08f34b3440bb508f6ac4627d74: Status 404 returned error can't find the container with id adeed91562c3b9dcea9d6a1c9bd53f41a3878c08f34b3440bb508f6ac4627d74 Jul 21 21:28:07.297621 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:28:07.297519 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-82b9182c-pnhwq" event={"ID":"caf180b8-bf82-43ed-8f1c-1c4aa74a081b","Type":"ContainerStarted","Data":"5117a60da70177226899288dd54516d9118e8eddf919b20878785996c74010d1"} Jul 21 21:28:07.297621 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:28:07.297594 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-82b9182c-pnhwq" event={"ID":"caf180b8-bf82-43ed-8f1c-1c4aa74a081b","Type":"ContainerStarted","Data":"adeed91562c3b9dcea9d6a1c9bd53f41a3878c08f34b3440bb508f6ac4627d74"} Jul 21 21:28:07.314483 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:28:07.314438 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-82b9182c-pnhwq" podStartSLOduration=1.314424294 podStartE2EDuration="1.314424294s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 21:28:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 21:28:07.314303164 +0000 UTC m=+1645.249052849" watchObservedRunningTime="2026-07-21 21:28:07.314424294 +0000 UTC m=+1645.249173956" Jul 21 21:28:08.304963 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:28:08.304927 2575 generic.go:356] "Generic (PLEG): container finished" podID="caf180b8-bf82-43ed-8f1c-1c4aa74a081b" containerID="5117a60da70177226899288dd54516d9118e8eddf919b20878785996c74010d1" exitCode=0 Jul 21 21:28:08.305363 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:28:08.305014 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-82b9182c-pnhwq" event={"ID":"caf180b8-bf82-43ed-8f1c-1c4aa74a081b","Type":"ContainerDied","Data":"5117a60da70177226899288dd54516d9118e8eddf919b20878785996c74010d1"} Jul 21 21:28:09.445276 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:28:09.445251 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-82b9182c-pnhwq" Jul 21 21:28:09.602165 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:28:09.602066 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/caf180b8-bf82-43ed-8f1c-1c4aa74a081b-maas-api-service-ca\") pod \"caf180b8-bf82-43ed-8f1c-1c4aa74a081b\" (UID: \"caf180b8-bf82-43ed-8f1c-1c4aa74a081b\") " Jul 21 21:28:09.602423 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:28:09.602401 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/caf180b8-bf82-43ed-8f1c-1c4aa74a081b-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "caf180b8-bf82-43ed-8f1c-1c4aa74a081b" (UID: "caf180b8-bf82-43ed-8f1c-1c4aa74a081b"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 21 21:28:09.703334 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:28:09.703296 2575 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/caf180b8-bf82-43ed-8f1c-1c4aa74a081b-maas-api-service-ca\") on node \"ip-10-0-140-97.ec2.internal\" DevicePath \"\"" Jul 21 21:28:10.317139 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:28:10.317103 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-82b9182c-pnhwq" event={"ID":"caf180b8-bf82-43ed-8f1c-1c4aa74a081b","Type":"ContainerDied","Data":"adeed91562c3b9dcea9d6a1c9bd53f41a3878c08f34b3440bb508f6ac4627d74"} Jul 21 21:28:10.317304 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:28:10.317146 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-82b9182c-pnhwq" Jul 21 21:28:10.317304 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:28:10.317156 2575 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="adeed91562c3b9dcea9d6a1c9bd53f41a3878c08f34b3440bb508f6ac4627d74" Jul 21 21:28:21.338242 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:28:21.338199 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-82b9182c-pnhwq"] Jul 21 21:28:21.340977 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:28:21.340944 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-82b9182c-pnhwq"] Jul 21 21:28:21.491173 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:28:21.491137 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-82b9182c-openshift-default-b78844dfc-k5472"] Jul 21 21:28:21.491438 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:28:21.491395 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-82b9182c-openshift-default-b78844dfc-k5472" podUID="ebda7857-1799-4934-8401-eaf8a77ac7e8" containerName="istio-proxy" containerID="cri-o://29f39edb3be70ca428873c8938789076e45052748e80fffd5daeb8a4ef07b3ba" gracePeriod=30 Jul 21 21:28:22.558348 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:28:22.558311 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-migrate-1710f404-gw-openshift-default-7ffdcd7b87-f692z"] Jul 21 21:28:22.558743 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:28:22.558691 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="caf180b8-bf82-43ed-8f1c-1c4aa74a081b" containerName="revoke-keys" Jul 21 21:28:22.558743 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:28:22.558703 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="caf180b8-bf82-43ed-8f1c-1c4aa74a081b" containerName="revoke-keys" Jul 21 21:28:22.558819 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:28:22.558769 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="caf180b8-bf82-43ed-8f1c-1c4aa74a081b" containerName="revoke-keys" Jul 21 21:28:22.561037 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:28:22.561013 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-1710f404-gw-openshift-default-7ffdcd7b87-f692z" Jul 21 21:28:22.564203 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:28:22.564182 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-migrate-1710f404-gw-openshift-default-dockercfg-q6r94\"" Jul 21 21:28:22.576482 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:28:22.576457 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-1710f404-gw-openshift-default-7ffdcd7b87-f692z"] Jul 21 21:28:22.624089 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:28:22.624054 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bfa501fa-77bd-4550-a865-d8a95bd05a2f-workload-socket\") pod \"e2e-migrate-1710f404-gw-openshift-default-7ffdcd7b87-f692z\" (UID: \"bfa501fa-77bd-4550-a865-d8a95bd05a2f\") " pod="openshift-ingress/e2e-migrate-1710f404-gw-openshift-default-7ffdcd7b87-f692z" Jul 21 21:28:22.624269 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:28:22.624099 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bfa501fa-77bd-4550-a865-d8a95bd05a2f-credential-socket\") pod \"e2e-migrate-1710f404-gw-openshift-default-7ffdcd7b87-f692z\" (UID: \"bfa501fa-77bd-4550-a865-d8a95bd05a2f\") " pod="openshift-ingress/e2e-migrate-1710f404-gw-openshift-default-7ffdcd7b87-f692z" Jul 21 21:28:22.624269 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:28:22.624117 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bfa501fa-77bd-4550-a865-d8a95bd05a2f-istiod-ca-cert\") pod \"e2e-migrate-1710f404-gw-openshift-default-7ffdcd7b87-f692z\" (UID: \"bfa501fa-77bd-4550-a865-d8a95bd05a2f\") " pod="openshift-ingress/e2e-migrate-1710f404-gw-openshift-default-7ffdcd7b87-f692z" Jul 21 21:28:22.624269 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:28:22.624138 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bfa501fa-77bd-4550-a865-d8a95bd05a2f-istio-token\") pod \"e2e-migrate-1710f404-gw-openshift-default-7ffdcd7b87-f692z\" (UID: \"bfa501fa-77bd-4550-a865-d8a95bd05a2f\") " pod="openshift-ingress/e2e-migrate-1710f404-gw-openshift-default-7ffdcd7b87-f692z" Jul 21 21:28:22.624269 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:28:22.624222 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x68m9\" (UniqueName: \"kubernetes.io/projected/bfa501fa-77bd-4550-a865-d8a95bd05a2f-kube-api-access-x68m9\") pod \"e2e-migrate-1710f404-gw-openshift-default-7ffdcd7b87-f692z\" (UID: \"bfa501fa-77bd-4550-a865-d8a95bd05a2f\") " pod="openshift-ingress/e2e-migrate-1710f404-gw-openshift-default-7ffdcd7b87-f692z" Jul 21 21:28:22.624477 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:28:22.624271 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bfa501fa-77bd-4550-a865-d8a95bd05a2f-istio-data\") pod \"e2e-migrate-1710f404-gw-openshift-default-7ffdcd7b87-f692z\" (UID: \"bfa501fa-77bd-4550-a865-d8a95bd05a2f\") " pod="openshift-ingress/e2e-migrate-1710f404-gw-openshift-default-7ffdcd7b87-f692z" Jul 21 21:28:22.624477 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:28:22.624357 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bfa501fa-77bd-4550-a865-d8a95bd05a2f-workload-certs\") pod \"e2e-migrate-1710f404-gw-openshift-default-7ffdcd7b87-f692z\" (UID: \"bfa501fa-77bd-4550-a865-d8a95bd05a2f\") " pod="openshift-ingress/e2e-migrate-1710f404-gw-openshift-default-7ffdcd7b87-f692z" Jul 21 21:28:22.624477 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:28:22.624405 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bfa501fa-77bd-4550-a865-d8a95bd05a2f-istio-podinfo\") pod \"e2e-migrate-1710f404-gw-openshift-default-7ffdcd7b87-f692z\" (UID: \"bfa501fa-77bd-4550-a865-d8a95bd05a2f\") " pod="openshift-ingress/e2e-migrate-1710f404-gw-openshift-default-7ffdcd7b87-f692z" Jul 21 21:28:22.624477 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:28:22.624435 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bfa501fa-77bd-4550-a865-d8a95bd05a2f-istio-envoy\") pod \"e2e-migrate-1710f404-gw-openshift-default-7ffdcd7b87-f692z\" (UID: \"bfa501fa-77bd-4550-a865-d8a95bd05a2f\") " pod="openshift-ingress/e2e-migrate-1710f404-gw-openshift-default-7ffdcd7b87-f692z" Jul 21 21:28:22.725727 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:28:22.725685 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bfa501fa-77bd-4550-a865-d8a95bd05a2f-istio-data\") pod \"e2e-migrate-1710f404-gw-openshift-default-7ffdcd7b87-f692z\" (UID: \"bfa501fa-77bd-4550-a865-d8a95bd05a2f\") " pod="openshift-ingress/e2e-migrate-1710f404-gw-openshift-default-7ffdcd7b87-f692z" Jul 21 21:28:22.725917 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:28:22.725744 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bfa501fa-77bd-4550-a865-d8a95bd05a2f-workload-certs\") pod \"e2e-migrate-1710f404-gw-openshift-default-7ffdcd7b87-f692z\" (UID: \"bfa501fa-77bd-4550-a865-d8a95bd05a2f\") " pod="openshift-ingress/e2e-migrate-1710f404-gw-openshift-default-7ffdcd7b87-f692z" Jul 21 21:28:22.725917 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:28:22.725778 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bfa501fa-77bd-4550-a865-d8a95bd05a2f-istio-podinfo\") pod \"e2e-migrate-1710f404-gw-openshift-default-7ffdcd7b87-f692z\" (UID: \"bfa501fa-77bd-4550-a865-d8a95bd05a2f\") " pod="openshift-ingress/e2e-migrate-1710f404-gw-openshift-default-7ffdcd7b87-f692z" Jul 21 21:28:22.725917 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:28:22.725806 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bfa501fa-77bd-4550-a865-d8a95bd05a2f-istio-envoy\") pod \"e2e-migrate-1710f404-gw-openshift-default-7ffdcd7b87-f692z\" (UID: \"bfa501fa-77bd-4550-a865-d8a95bd05a2f\") " pod="openshift-ingress/e2e-migrate-1710f404-gw-openshift-default-7ffdcd7b87-f692z" Jul 21 21:28:22.726073 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:28:22.726017 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bfa501fa-77bd-4550-a865-d8a95bd05a2f-workload-socket\") pod \"e2e-migrate-1710f404-gw-openshift-default-7ffdcd7b87-f692z\" (UID: \"bfa501fa-77bd-4550-a865-d8a95bd05a2f\") " pod="openshift-ingress/e2e-migrate-1710f404-gw-openshift-default-7ffdcd7b87-f692z" Jul 21 21:28:22.726073 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:28:22.726071 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bfa501fa-77bd-4550-a865-d8a95bd05a2f-credential-socket\") pod \"e2e-migrate-1710f404-gw-openshift-default-7ffdcd7b87-f692z\" (UID: \"bfa501fa-77bd-4550-a865-d8a95bd05a2f\") " pod="openshift-ingress/e2e-migrate-1710f404-gw-openshift-default-7ffdcd7b87-f692z" Jul 21 21:28:22.726182 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:28:22.726097 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bfa501fa-77bd-4550-a865-d8a95bd05a2f-istiod-ca-cert\") pod \"e2e-migrate-1710f404-gw-openshift-default-7ffdcd7b87-f692z\" (UID: \"bfa501fa-77bd-4550-a865-d8a95bd05a2f\") " pod="openshift-ingress/e2e-migrate-1710f404-gw-openshift-default-7ffdcd7b87-f692z" Jul 21 21:28:22.726182 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:28:22.726129 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bfa501fa-77bd-4550-a865-d8a95bd05a2f-istio-token\") pod \"e2e-migrate-1710f404-gw-openshift-default-7ffdcd7b87-f692z\" (UID: \"bfa501fa-77bd-4550-a865-d8a95bd05a2f\") " pod="openshift-ingress/e2e-migrate-1710f404-gw-openshift-default-7ffdcd7b87-f692z" Jul 21 21:28:22.726182 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:28:22.726152 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bfa501fa-77bd-4550-a865-d8a95bd05a2f-istio-data\") pod \"e2e-migrate-1710f404-gw-openshift-default-7ffdcd7b87-f692z\" (UID: \"bfa501fa-77bd-4550-a865-d8a95bd05a2f\") " pod="openshift-ingress/e2e-migrate-1710f404-gw-openshift-default-7ffdcd7b87-f692z" Jul 21 21:28:22.726182 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:28:22.726167 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x68m9\" (UniqueName: \"kubernetes.io/projected/bfa501fa-77bd-4550-a865-d8a95bd05a2f-kube-api-access-x68m9\") pod \"e2e-migrate-1710f404-gw-openshift-default-7ffdcd7b87-f692z\" (UID: \"bfa501fa-77bd-4550-a865-d8a95bd05a2f\") " pod="openshift-ingress/e2e-migrate-1710f404-gw-openshift-default-7ffdcd7b87-f692z" Jul 21 21:28:22.726395 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:28:22.726211 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bfa501fa-77bd-4550-a865-d8a95bd05a2f-workload-certs\") pod \"e2e-migrate-1710f404-gw-openshift-default-7ffdcd7b87-f692z\" (UID: \"bfa501fa-77bd-4550-a865-d8a95bd05a2f\") " pod="openshift-ingress/e2e-migrate-1710f404-gw-openshift-default-7ffdcd7b87-f692z" Jul 21 21:28:22.726395 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:28:22.726387 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bfa501fa-77bd-4550-a865-d8a95bd05a2f-credential-socket\") pod \"e2e-migrate-1710f404-gw-openshift-default-7ffdcd7b87-f692z\" (UID: \"bfa501fa-77bd-4550-a865-d8a95bd05a2f\") " pod="openshift-ingress/e2e-migrate-1710f404-gw-openshift-default-7ffdcd7b87-f692z" Jul 21 21:28:22.726504 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:28:22.726409 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bfa501fa-77bd-4550-a865-d8a95bd05a2f-workload-socket\") pod \"e2e-migrate-1710f404-gw-openshift-default-7ffdcd7b87-f692z\" (UID: \"bfa501fa-77bd-4550-a865-d8a95bd05a2f\") " pod="openshift-ingress/e2e-migrate-1710f404-gw-openshift-default-7ffdcd7b87-f692z" Jul 21 21:28:22.726880 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:28:22.726853 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bfa501fa-77bd-4550-a865-d8a95bd05a2f-istiod-ca-cert\") pod \"e2e-migrate-1710f404-gw-openshift-default-7ffdcd7b87-f692z\" (UID: \"bfa501fa-77bd-4550-a865-d8a95bd05a2f\") " pod="openshift-ingress/e2e-migrate-1710f404-gw-openshift-default-7ffdcd7b87-f692z" Jul 21 21:28:22.728080 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:28:22.728052 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bfa501fa-77bd-4550-a865-d8a95bd05a2f-istio-envoy\") pod \"e2e-migrate-1710f404-gw-openshift-default-7ffdcd7b87-f692z\" (UID: \"bfa501fa-77bd-4550-a865-d8a95bd05a2f\") " pod="openshift-ingress/e2e-migrate-1710f404-gw-openshift-default-7ffdcd7b87-f692z" Jul 21 21:28:22.728326 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:28:22.728303 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bfa501fa-77bd-4550-a865-d8a95bd05a2f-istio-podinfo\") pod \"e2e-migrate-1710f404-gw-openshift-default-7ffdcd7b87-f692z\" (UID: \"bfa501fa-77bd-4550-a865-d8a95bd05a2f\") " pod="openshift-ingress/e2e-migrate-1710f404-gw-openshift-default-7ffdcd7b87-f692z" Jul 21 21:28:22.734405 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:28:22.734383 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bfa501fa-77bd-4550-a865-d8a95bd05a2f-istio-token\") pod \"e2e-migrate-1710f404-gw-openshift-default-7ffdcd7b87-f692z\" (UID: \"bfa501fa-77bd-4550-a865-d8a95bd05a2f\") " pod="openshift-ingress/e2e-migrate-1710f404-gw-openshift-default-7ffdcd7b87-f692z" Jul 21 21:28:22.734629 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:28:22.734608 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x68m9\" (UniqueName: \"kubernetes.io/projected/bfa501fa-77bd-4550-a865-d8a95bd05a2f-kube-api-access-x68m9\") pod \"e2e-migrate-1710f404-gw-openshift-default-7ffdcd7b87-f692z\" (UID: \"bfa501fa-77bd-4550-a865-d8a95bd05a2f\") " pod="openshift-ingress/e2e-migrate-1710f404-gw-openshift-default-7ffdcd7b87-f692z" Jul 21 21:28:22.741109 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:28:22.741081 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="caf180b8-bf82-43ed-8f1c-1c4aa74a081b" path="/var/lib/kubelet/pods/caf180b8-bf82-43ed-8f1c-1c4aa74a081b/volumes" Jul 21 21:28:22.871747 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:28:22.871709 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-1710f404-gw-openshift-default-7ffdcd7b87-f692z" Jul 21 21:28:23.017262 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:28:23.017237 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-1710f404-gw-openshift-default-7ffdcd7b87-f692z"] Jul 21 21:28:23.019737 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:28:23.019707 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbfa501fa_77bd_4550_a865_d8a95bd05a2f.slice/crio-96eb230883cad3abf9c1d7527ff8ac57e0034db6fdf54139c15fe717f3c3e5c0 WatchSource:0}: Error finding container 96eb230883cad3abf9c1d7527ff8ac57e0034db6fdf54139c15fe717f3c3e5c0: Status 404 returned error can't find the container with id 96eb230883cad3abf9c1d7527ff8ac57e0034db6fdf54139c15fe717f3c3e5c0 Jul 21 21:28:23.022254 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:28:23.022214 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 21 21:28:23.022338 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:28:23.022288 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 21 21:28:23.022338 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:28:23.022325 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 21 21:28:23.382725 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:28:23.382684 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-1710f404-gw-openshift-default-7ffdcd7b87-f692z" event={"ID":"bfa501fa-77bd-4550-a865-d8a95bd05a2f","Type":"ContainerStarted","Data":"7395d1616531dde01a5609f73dd8de546282b3c500bcc08e88b6a8ad0530bf60"} Jul 21 21:28:23.382890 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:28:23.382748 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-1710f404-gw-openshift-default-7ffdcd7b87-f692z" event={"ID":"bfa501fa-77bd-4550-a865-d8a95bd05a2f","Type":"ContainerStarted","Data":"96eb230883cad3abf9c1d7527ff8ac57e0034db6fdf54139c15fe717f3c3e5c0"} Jul 21 21:28:23.405668 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:28:23.404434 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-migrate-1710f404-gw-openshift-default-7ffdcd7b87-f692z" podStartSLOduration=1.404412679 podStartE2EDuration="1.404412679s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 21:28:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 21:28:23.402447024 +0000 UTC m=+1661.337196718" watchObservedRunningTime="2026-07-21 21:28:23.404412679 +0000 UTC m=+1661.339162346" Jul 21 21:28:23.872815 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:28:23.872788 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-1710f404-gw-openshift-default-7ffdcd7b87-f692z" Jul 21 21:28:24.876959 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:28:24.876933 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-1710f404-gw-openshift-default-7ffdcd7b87-f692z" Jul 21 21:28:25.394315 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:28:25.394284 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-1710f404-gw-openshift-default-7ffdcd7b87-f692z" Jul 21 21:28:25.395166 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:28:25.395146 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-1710f404-gw-openshift-default-7ffdcd7b87-f692z" Jul 21 21:28:26.750298 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:28:26.750234 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-82b9182c-openshift-default-b78844dfc-k5472" Jul 21 21:28:26.868993 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:28:26.868940 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2f66s\" (UniqueName: \"kubernetes.io/projected/ebda7857-1799-4934-8401-eaf8a77ac7e8-kube-api-access-2f66s\") pod \"ebda7857-1799-4934-8401-eaf8a77ac7e8\" (UID: \"ebda7857-1799-4934-8401-eaf8a77ac7e8\") " Jul 21 21:28:26.869167 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:28:26.869041 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ebda7857-1799-4934-8401-eaf8a77ac7e8-workload-certs\") pod \"ebda7857-1799-4934-8401-eaf8a77ac7e8\" (UID: \"ebda7857-1799-4934-8401-eaf8a77ac7e8\") " Jul 21 21:28:26.869167 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:28:26.869082 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ebda7857-1799-4934-8401-eaf8a77ac7e8-istio-data\") pod \"ebda7857-1799-4934-8401-eaf8a77ac7e8\" (UID: \"ebda7857-1799-4934-8401-eaf8a77ac7e8\") " Jul 21 21:28:26.869167 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:28:26.869134 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ebda7857-1799-4934-8401-eaf8a77ac7e8-istio-token\") pod \"ebda7857-1799-4934-8401-eaf8a77ac7e8\" (UID: \"ebda7857-1799-4934-8401-eaf8a77ac7e8\") " Jul 21 21:28:26.869167 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:28:26.869154 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ebda7857-1799-4934-8401-eaf8a77ac7e8-istio-envoy\") pod \"ebda7857-1799-4934-8401-eaf8a77ac7e8\" (UID: \"ebda7857-1799-4934-8401-eaf8a77ac7e8\") " Jul 21 21:28:26.869350 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:28:26.869179 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ebda7857-1799-4934-8401-eaf8a77ac7e8-workload-socket\") pod \"ebda7857-1799-4934-8401-eaf8a77ac7e8\" (UID: \"ebda7857-1799-4934-8401-eaf8a77ac7e8\") " Jul 21 21:28:26.869350 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:28:26.869216 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ebda7857-1799-4934-8401-eaf8a77ac7e8-credential-socket\") pod \"ebda7857-1799-4934-8401-eaf8a77ac7e8\" (UID: \"ebda7857-1799-4934-8401-eaf8a77ac7e8\") " Jul 21 21:28:26.869448 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:28:26.869377 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ebda7857-1799-4934-8401-eaf8a77ac7e8-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "ebda7857-1799-4934-8401-eaf8a77ac7e8" (UID: "ebda7857-1799-4934-8401-eaf8a77ac7e8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 21:28:26.869499 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:28:26.869464 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ebda7857-1799-4934-8401-eaf8a77ac7e8-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "ebda7857-1799-4934-8401-eaf8a77ac7e8" (UID: "ebda7857-1799-4934-8401-eaf8a77ac7e8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 21:28:26.869499 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:28:26.869485 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ebda7857-1799-4934-8401-eaf8a77ac7e8-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "ebda7857-1799-4934-8401-eaf8a77ac7e8" (UID: "ebda7857-1799-4934-8401-eaf8a77ac7e8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 21:28:26.869651 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:28:26.869498 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ebda7857-1799-4934-8401-eaf8a77ac7e8-istiod-ca-cert\") pod \"ebda7857-1799-4934-8401-eaf8a77ac7e8\" (UID: \"ebda7857-1799-4934-8401-eaf8a77ac7e8\") " Jul 21 21:28:26.869651 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:28:26.869637 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ebda7857-1799-4934-8401-eaf8a77ac7e8-istio-podinfo\") pod \"ebda7857-1799-4934-8401-eaf8a77ac7e8\" (UID: \"ebda7857-1799-4934-8401-eaf8a77ac7e8\") " Jul 21 21:28:26.869821 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:28:26.869799 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ebda7857-1799-4934-8401-eaf8a77ac7e8-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "ebda7857-1799-4934-8401-eaf8a77ac7e8" (UID: "ebda7857-1799-4934-8401-eaf8a77ac7e8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 21 21:28:26.870026 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:28:26.870010 2575 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ebda7857-1799-4934-8401-eaf8a77ac7e8-workload-certs\") on node \"ip-10-0-140-97.ec2.internal\" DevicePath \"\"" Jul 21 21:28:26.870109 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:28:26.870029 2575 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ebda7857-1799-4934-8401-eaf8a77ac7e8-workload-socket\") on node \"ip-10-0-140-97.ec2.internal\" DevicePath \"\"" Jul 21 21:28:26.870109 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:28:26.870041 2575 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ebda7857-1799-4934-8401-eaf8a77ac7e8-credential-socket\") on node \"ip-10-0-140-97.ec2.internal\" DevicePath \"\"" Jul 21 21:28:26.870109 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:28:26.870053 2575 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ebda7857-1799-4934-8401-eaf8a77ac7e8-istiod-ca-cert\") on node \"ip-10-0-140-97.ec2.internal\" DevicePath \"\"" Jul 21 21:28:26.870306 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:28:26.870147 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ebda7857-1799-4934-8401-eaf8a77ac7e8-istio-data" (OuterVolumeSpecName: "istio-data") pod "ebda7857-1799-4934-8401-eaf8a77ac7e8" (UID: "ebda7857-1799-4934-8401-eaf8a77ac7e8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 21:28:26.871722 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:28:26.871680 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ebda7857-1799-4934-8401-eaf8a77ac7e8-istio-token" (OuterVolumeSpecName: "istio-token") pod "ebda7857-1799-4934-8401-eaf8a77ac7e8" (UID: "ebda7857-1799-4934-8401-eaf8a77ac7e8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 21 21:28:26.871818 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:28:26.871721 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ebda7857-1799-4934-8401-eaf8a77ac7e8-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "ebda7857-1799-4934-8401-eaf8a77ac7e8" (UID: "ebda7857-1799-4934-8401-eaf8a77ac7e8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 21 21:28:26.871818 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:28:26.871733 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ebda7857-1799-4934-8401-eaf8a77ac7e8-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "ebda7857-1799-4934-8401-eaf8a77ac7e8" (UID: "ebda7857-1799-4934-8401-eaf8a77ac7e8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 21:28:26.871818 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:28:26.871752 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ebda7857-1799-4934-8401-eaf8a77ac7e8-kube-api-access-2f66s" (OuterVolumeSpecName: "kube-api-access-2f66s") pod "ebda7857-1799-4934-8401-eaf8a77ac7e8" (UID: "ebda7857-1799-4934-8401-eaf8a77ac7e8"). InnerVolumeSpecName "kube-api-access-2f66s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 21 21:28:26.971510 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:28:26.971469 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2f66s\" (UniqueName: \"kubernetes.io/projected/ebda7857-1799-4934-8401-eaf8a77ac7e8-kube-api-access-2f66s\") on node \"ip-10-0-140-97.ec2.internal\" DevicePath \"\"" Jul 21 21:28:26.971510 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:28:26.971504 2575 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ebda7857-1799-4934-8401-eaf8a77ac7e8-istio-data\") on node \"ip-10-0-140-97.ec2.internal\" DevicePath \"\"" Jul 21 21:28:26.971510 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:28:26.971517 2575 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ebda7857-1799-4934-8401-eaf8a77ac7e8-istio-token\") on node \"ip-10-0-140-97.ec2.internal\" DevicePath \"\"" Jul 21 21:28:26.971775 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:28:26.971557 2575 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ebda7857-1799-4934-8401-eaf8a77ac7e8-istio-envoy\") on node \"ip-10-0-140-97.ec2.internal\" DevicePath \"\"" Jul 21 21:28:26.971775 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:28:26.971569 2575 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ebda7857-1799-4934-8401-eaf8a77ac7e8-istio-podinfo\") on node \"ip-10-0-140-97.ec2.internal\" DevicePath \"\"" Jul 21 21:28:27.406967 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:28:27.406932 2575 generic.go:356] "Generic (PLEG): container finished" podID="ebda7857-1799-4934-8401-eaf8a77ac7e8" containerID="29f39edb3be70ca428873c8938789076e45052748e80fffd5daeb8a4ef07b3ba" exitCode=0 Jul 21 21:28:27.407139 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:28:27.407001 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-82b9182c-openshift-default-b78844dfc-k5472" Jul 21 21:28:27.407139 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:28:27.407012 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-82b9182c-openshift-default-b78844dfc-k5472" event={"ID":"ebda7857-1799-4934-8401-eaf8a77ac7e8","Type":"ContainerDied","Data":"29f39edb3be70ca428873c8938789076e45052748e80fffd5daeb8a4ef07b3ba"} Jul 21 21:28:27.407139 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:28:27.407086 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-82b9182c-openshift-default-b78844dfc-k5472" event={"ID":"ebda7857-1799-4934-8401-eaf8a77ac7e8","Type":"ContainerDied","Data":"ec77b337fc0f701f410004347152523fb411b6c28c4d0a716db326056339c289"} Jul 21 21:28:27.407139 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:28:27.407108 2575 scope.go:117] "RemoveContainer" containerID="29f39edb3be70ca428873c8938789076e45052748e80fffd5daeb8a4ef07b3ba" Jul 21 21:28:27.416146 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:28:27.416130 2575 scope.go:117] "RemoveContainer" containerID="ec081142a724205459d35b4a28cdba8c66a8030d1117d447e38cf32bbd97f37b" Jul 21 21:28:27.423957 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:28:27.423935 2575 scope.go:117] "RemoveContainer" containerID="29f39edb3be70ca428873c8938789076e45052748e80fffd5daeb8a4ef07b3ba" Jul 21 21:28:27.424211 ip-10-0-140-97 kubenswrapper[2575]: E0721 21:28:27.424190 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"29f39edb3be70ca428873c8938789076e45052748e80fffd5daeb8a4ef07b3ba\": container with ID starting with 29f39edb3be70ca428873c8938789076e45052748e80fffd5daeb8a4ef07b3ba not found: ID does not exist" containerID="29f39edb3be70ca428873c8938789076e45052748e80fffd5daeb8a4ef07b3ba" Jul 21 21:28:27.424252 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:28:27.424221 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"29f39edb3be70ca428873c8938789076e45052748e80fffd5daeb8a4ef07b3ba"} err="failed to get container status \"29f39edb3be70ca428873c8938789076e45052748e80fffd5daeb8a4ef07b3ba\": rpc error: code = NotFound desc = could not find container \"29f39edb3be70ca428873c8938789076e45052748e80fffd5daeb8a4ef07b3ba\": container with ID starting with 29f39edb3be70ca428873c8938789076e45052748e80fffd5daeb8a4ef07b3ba not found: ID does not exist" Jul 21 21:28:27.424252 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:28:27.424239 2575 scope.go:117] "RemoveContainer" containerID="ec081142a724205459d35b4a28cdba8c66a8030d1117d447e38cf32bbd97f37b" Jul 21 21:28:27.424484 ip-10-0-140-97 kubenswrapper[2575]: E0721 21:28:27.424470 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ec081142a724205459d35b4a28cdba8c66a8030d1117d447e38cf32bbd97f37b\": container with ID starting with ec081142a724205459d35b4a28cdba8c66a8030d1117d447e38cf32bbd97f37b not found: ID does not exist" containerID="ec081142a724205459d35b4a28cdba8c66a8030d1117d447e38cf32bbd97f37b" Jul 21 21:28:27.424517 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:28:27.424489 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ec081142a724205459d35b4a28cdba8c66a8030d1117d447e38cf32bbd97f37b"} err="failed to get container status \"ec081142a724205459d35b4a28cdba8c66a8030d1117d447e38cf32bbd97f37b\": rpc error: code = NotFound desc = could not find container \"ec081142a724205459d35b4a28cdba8c66a8030d1117d447e38cf32bbd97f37b\": container with ID starting with ec081142a724205459d35b4a28cdba8c66a8030d1117d447e38cf32bbd97f37b not found: ID does not exist" Jul 21 21:28:27.429627 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:28:27.429602 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-82b9182c-openshift-default-b78844dfc-k5472"] Jul 21 21:28:27.434946 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:28:27.434921 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-82b9182c-openshift-default-b78844dfc-k5472"] Jul 21 21:28:28.743438 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:28:28.743380 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ebda7857-1799-4934-8401-eaf8a77ac7e8" path="/var/lib/kubelet/pods/ebda7857-1799-4934-8401-eaf8a77ac7e8/volumes" Jul 21 21:28:35.449585 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:28:35.449552 2575 generic.go:356] "Generic (PLEG): container finished" podID="bfa501fa-77bd-4550-a865-d8a95bd05a2f" containerID="7395d1616531dde01a5609f73dd8de546282b3c500bcc08e88b6a8ad0530bf60" exitCode=0 Jul 21 21:28:35.450083 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:28:35.449630 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-1710f404-gw-openshift-default-7ffdcd7b87-f692z" event={"ID":"bfa501fa-77bd-4550-a865-d8a95bd05a2f","Type":"ContainerDied","Data":"7395d1616531dde01a5609f73dd8de546282b3c500bcc08e88b6a8ad0530bf60"} Jul 21 21:28:35.450167 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:28:35.450149 2575 scope.go:117] "RemoveContainer" containerID="7395d1616531dde01a5609f73dd8de546282b3c500bcc08e88b6a8ad0530bf60" Jul 21 21:28:35.451246 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:28:35.451217 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 21 21:28:35.451307 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:28:35.451287 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 21 21:28:35.451347 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:28:35.451317 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 21 21:28:36.460644 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:28:36.460601 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-1710f404-gw-openshift-default-7ffdcd7b87-f692z" event={"ID":"bfa501fa-77bd-4550-a865-d8a95bd05a2f","Type":"ContainerStarted","Data":"344f751ed48fa3e4cf20224f31e49bbbeddadff43b350141adb044bfae02389f"} Jul 21 21:28:36.872619 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:28:36.872576 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-1710f404-gw-openshift-default-7ffdcd7b87-f692z" Jul 21 21:28:36.873897 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:28:36.873867 2575 patch_prober.go:28] interesting pod/e2e-migrate-1710f404-gw-openshift-default-7ffdcd7b87-f692z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" start-of-body= Jul 21 21:28:36.874036 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:28:36.873928 2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-1710f404-gw-openshift-default-7ffdcd7b87-f692z" podUID="bfa501fa-77bd-4550-a865-d8a95bd05a2f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" Jul 21 21:28:37.872736 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:28:37.872703 2575 patch_prober.go:28] interesting pod/e2e-migrate-1710f404-gw-openshift-default-7ffdcd7b87-f692z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" start-of-body= Jul 21 21:28:37.873130 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:28:37.872759 2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-1710f404-gw-openshift-default-7ffdcd7b87-f692z" podUID="bfa501fa-77bd-4550-a865-d8a95bd05a2f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" Jul 21 21:28:38.872412 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:28:38.872375 2575 patch_prober.go:28] interesting pod/e2e-migrate-1710f404-gw-openshift-default-7ffdcd7b87-f692z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" start-of-body= Jul 21 21:28:38.872612 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:28:38.872430 2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-1710f404-gw-openshift-default-7ffdcd7b87-f692z" podUID="bfa501fa-77bd-4550-a865-d8a95bd05a2f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" Jul 21 21:28:39.872575 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:28:39.872499 2575 patch_prober.go:28] interesting pod/e2e-migrate-1710f404-gw-openshift-default-7ffdcd7b87-f692z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" start-of-body= Jul 21 21:28:39.873081 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:28:39.872604 2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-1710f404-gw-openshift-default-7ffdcd7b87-f692z" podUID="bfa501fa-77bd-4550-a865-d8a95bd05a2f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" Jul 21 21:28:40.394810 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:28:40.394779 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-1710f404-gw-openshift-default-7ffdcd7b87-f692z" Jul 21 21:28:40.876382 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:28:40.876353 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-1710f404-gw-openshift-default-7ffdcd7b87-f692z" Jul 21 21:28:40.877331 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:28:40.877308 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-1710f404-gw-openshift-default-7ffdcd7b87-f692z" Jul 21 21:28:49.941271 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:28:49.941237 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1710f404-k7pks"] Jul 21 21:28:49.941672 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:28:49.941606 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ebda7857-1799-4934-8401-eaf8a77ac7e8" containerName="istio-proxy" Jul 21 21:28:49.941672 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:28:49.941617 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="ebda7857-1799-4934-8401-eaf8a77ac7e8" containerName="istio-proxy" Jul 21 21:28:49.941672 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:28:49.941629 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ebda7857-1799-4934-8401-eaf8a77ac7e8" containerName="istio-proxy" Jul 21 21:28:49.941672 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:28:49.941635 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="ebda7857-1799-4934-8401-eaf8a77ac7e8" containerName="istio-proxy" Jul 21 21:28:49.941841 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:28:49.941698 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="ebda7857-1799-4934-8401-eaf8a77ac7e8" containerName="istio-proxy" Jul 21 21:28:49.941841 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:28:49.941709 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="ebda7857-1799-4934-8401-eaf8a77ac7e8" containerName="istio-proxy" Jul 21 21:28:49.943717 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:28:49.943700 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1710f404-k7pks" Jul 21 21:28:49.946134 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:28:49.946109 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9fxhr\"" Jul 21 21:28:49.946262 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:28:49.946109 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 21 21:28:49.952451 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:28:49.952423 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1710f404-k7pks"] Jul 21 21:28:50.096948 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:28:50.096914 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f5dc8db0-a3f4-468f-8841-b399f0561f5b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-1710f404-k7pks\" (UID: \"f5dc8db0-a3f4-468f-8841-b399f0561f5b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1710f404-k7pks" Jul 21 21:28:50.197889 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:28:50.197811 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f5dc8db0-a3f4-468f-8841-b399f0561f5b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-1710f404-k7pks\" (UID: \"f5dc8db0-a3f4-468f-8841-b399f0561f5b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1710f404-k7pks" Jul 21 21:28:50.198404 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:28:50.198385 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f5dc8db0-a3f4-468f-8841-b399f0561f5b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-1710f404-k7pks\" (UID: \"f5dc8db0-a3f4-468f-8841-b399f0561f5b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1710f404-k7pks" Jul 21 21:28:50.254806 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:28:50.254785 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1710f404-k7pks" Jul 21 21:28:50.383434 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:28:50.383400 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1710f404-k7pks"] Jul 21 21:28:50.385146 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:28:50.385112 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf5dc8db0_a3f4_468f_8841_b399f0561f5b.slice/crio-9479aa99f3c809c074f594a8c626f77d0eb17635402df5a2e4f8537561434875 WatchSource:0}: Error finding container 9479aa99f3c809c074f594a8c626f77d0eb17635402df5a2e4f8537561434875: Status 404 returned error can't find the container with id 9479aa99f3c809c074f594a8c626f77d0eb17635402df5a2e4f8537561434875 Jul 21 21:28:50.528990 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:28:50.528900 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1710f404-k7pks" event={"ID":"f5dc8db0-a3f4-468f-8841-b399f0561f5b","Type":"ContainerStarted","Data":"9479aa99f3c809c074f594a8c626f77d0eb17635402df5a2e4f8537561434875"} Jul 21 21:28:51.535552 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:28:51.535495 2575 generic.go:356] "Generic (PLEG): container finished" podID="f5dc8db0-a3f4-468f-8841-b399f0561f5b" containerID="b21a009c6a7dd31933c7361e0b07f5d1842861b0b893972784e8e0c7782b951b" exitCode=0 Jul 21 21:28:51.535949 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:28:51.535562 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1710f404-k7pks" event={"ID":"f5dc8db0-a3f4-468f-8841-b399f0561f5b","Type":"ContainerDied","Data":"b21a009c6a7dd31933c7361e0b07f5d1842861b0b893972784e8e0c7782b951b"} Jul 21 21:28:52.673413 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:28:52.673388 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1710f404-k7pks" Jul 21 21:28:52.820969 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:28:52.820874 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f5dc8db0-a3f4-468f-8841-b399f0561f5b-maas-api-service-ca\") pod \"f5dc8db0-a3f4-468f-8841-b399f0561f5b\" (UID: \"f5dc8db0-a3f4-468f-8841-b399f0561f5b\") " Jul 21 21:28:52.821260 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:28:52.821236 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f5dc8db0-a3f4-468f-8841-b399f0561f5b-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "f5dc8db0-a3f4-468f-8841-b399f0561f5b" (UID: "f5dc8db0-a3f4-468f-8841-b399f0561f5b"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 21 21:28:52.922583 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:28:52.922510 2575 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f5dc8db0-a3f4-468f-8841-b399f0561f5b-maas-api-service-ca\") on node \"ip-10-0-140-97.ec2.internal\" DevicePath \"\"" Jul 21 21:28:53.545713 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:28:53.545682 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1710f404-k7pks" Jul 21 21:28:53.545874 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:28:53.545684 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1710f404-k7pks" event={"ID":"f5dc8db0-a3f4-468f-8841-b399f0561f5b","Type":"ContainerDied","Data":"9479aa99f3c809c074f594a8c626f77d0eb17635402df5a2e4f8537561434875"} Jul 21 21:28:53.545874 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:28:53.545807 2575 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9479aa99f3c809c074f594a8c626f77d0eb17635402df5a2e4f8537561434875" Jul 21 21:29:05.021843 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:29:05.021748 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1710f404-k7pks"] Jul 21 21:29:05.026461 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:29:05.026426 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1710f404-k7pks"] Jul 21 21:29:05.172121 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:29:05.172086 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-1710f404-gw-openshift-default-7ffdcd7b87-f692z"] Jul 21 21:29:05.172372 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:29:05.172349 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-migrate-1710f404-gw-openshift-default-7ffdcd7b87-f692z" podUID="bfa501fa-77bd-4550-a865-d8a95bd05a2f" containerName="istio-proxy" containerID="cri-o://344f751ed48fa3e4cf20224f31e49bbbeddadff43b350141adb044bfae02389f" gracePeriod=30 Jul 21 21:29:05.981344 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:29:05.981314 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-a4a82f5f-openshift-default-7f7f96dc7c-5wbrz"] Jul 21 21:29:05.981713 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:29:05.981697 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f5dc8db0-a3f4-468f-8841-b399f0561f5b" containerName="revoke-keys" Jul 21 21:29:05.981713 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:29:05.981712 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="f5dc8db0-a3f4-468f-8841-b399f0561f5b" containerName="revoke-keys" Jul 21 21:29:05.981824 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:29:05.981768 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="f5dc8db0-a3f4-468f-8841-b399f0561f5b" containerName="revoke-keys" Jul 21 21:29:05.983912 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:29:05.983895 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-a4a82f5f-openshift-default-7f7f96dc7c-5wbrz" Jul 21 21:29:05.986364 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:29:05.986341 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-a4a82f5f-openshift-default-dockercfg-ll8qk\"" Jul 21 21:29:05.997636 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:29:05.997611 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-a4a82f5f-openshift-default-7f7f96dc7c-5wbrz"] Jul 21 21:29:06.038572 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:29:06.038520 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/443386e3-0ac0-4ac0-a04e-5571c0a6c915-credential-socket\") pod \"e2e-ait-a4a82f5f-openshift-default-7f7f96dc7c-5wbrz\" (UID: \"443386e3-0ac0-4ac0-a04e-5571c0a6c915\") " pod="openshift-ingress/e2e-ait-a4a82f5f-openshift-default-7f7f96dc7c-5wbrz" Jul 21 21:29:06.038993 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:29:06.038584 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/443386e3-0ac0-4ac0-a04e-5571c0a6c915-istio-data\") pod \"e2e-ait-a4a82f5f-openshift-default-7f7f96dc7c-5wbrz\" (UID: \"443386e3-0ac0-4ac0-a04e-5571c0a6c915\") " pod="openshift-ingress/e2e-ait-a4a82f5f-openshift-default-7f7f96dc7c-5wbrz" Jul 21 21:29:06.038993 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:29:06.038616 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/443386e3-0ac0-4ac0-a04e-5571c0a6c915-istio-token\") pod \"e2e-ait-a4a82f5f-openshift-default-7f7f96dc7c-5wbrz\" (UID: \"443386e3-0ac0-4ac0-a04e-5571c0a6c915\") " pod="openshift-ingress/e2e-ait-a4a82f5f-openshift-default-7f7f96dc7c-5wbrz" Jul 21 21:29:06.038993 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:29:06.038645 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/443386e3-0ac0-4ac0-a04e-5571c0a6c915-istio-podinfo\") pod \"e2e-ait-a4a82f5f-openshift-default-7f7f96dc7c-5wbrz\" (UID: \"443386e3-0ac0-4ac0-a04e-5571c0a6c915\") " pod="openshift-ingress/e2e-ait-a4a82f5f-openshift-default-7f7f96dc7c-5wbrz" Jul 21 21:29:06.038993 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:29:06.038690 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kmzkd\" (UniqueName: \"kubernetes.io/projected/443386e3-0ac0-4ac0-a04e-5571c0a6c915-kube-api-access-kmzkd\") pod \"e2e-ait-a4a82f5f-openshift-default-7f7f96dc7c-5wbrz\" (UID: \"443386e3-0ac0-4ac0-a04e-5571c0a6c915\") " pod="openshift-ingress/e2e-ait-a4a82f5f-openshift-default-7f7f96dc7c-5wbrz" Jul 21 21:29:06.038993 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:29:06.038727 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/443386e3-0ac0-4ac0-a04e-5571c0a6c915-istio-envoy\") pod \"e2e-ait-a4a82f5f-openshift-default-7f7f96dc7c-5wbrz\" (UID: \"443386e3-0ac0-4ac0-a04e-5571c0a6c915\") " pod="openshift-ingress/e2e-ait-a4a82f5f-openshift-default-7f7f96dc7c-5wbrz" Jul 21 21:29:06.038993 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:29:06.038766 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/443386e3-0ac0-4ac0-a04e-5571c0a6c915-workload-socket\") pod \"e2e-ait-a4a82f5f-openshift-default-7f7f96dc7c-5wbrz\" (UID: \"443386e3-0ac0-4ac0-a04e-5571c0a6c915\") " pod="openshift-ingress/e2e-ait-a4a82f5f-openshift-default-7f7f96dc7c-5wbrz" Jul 21 21:29:06.038993 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:29:06.038859 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/443386e3-0ac0-4ac0-a04e-5571c0a6c915-istiod-ca-cert\") pod \"e2e-ait-a4a82f5f-openshift-default-7f7f96dc7c-5wbrz\" (UID: \"443386e3-0ac0-4ac0-a04e-5571c0a6c915\") " pod="openshift-ingress/e2e-ait-a4a82f5f-openshift-default-7f7f96dc7c-5wbrz" Jul 21 21:29:06.038993 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:29:06.038908 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/443386e3-0ac0-4ac0-a04e-5571c0a6c915-workload-certs\") pod \"e2e-ait-a4a82f5f-openshift-default-7f7f96dc7c-5wbrz\" (UID: \"443386e3-0ac0-4ac0-a04e-5571c0a6c915\") " pod="openshift-ingress/e2e-ait-a4a82f5f-openshift-default-7f7f96dc7c-5wbrz" Jul 21 21:29:06.140321 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:29:06.140281 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/443386e3-0ac0-4ac0-a04e-5571c0a6c915-istio-envoy\") pod \"e2e-ait-a4a82f5f-openshift-default-7f7f96dc7c-5wbrz\" (UID: \"443386e3-0ac0-4ac0-a04e-5571c0a6c915\") " pod="openshift-ingress/e2e-ait-a4a82f5f-openshift-default-7f7f96dc7c-5wbrz" Jul 21 21:29:06.140483 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:29:06.140339 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/443386e3-0ac0-4ac0-a04e-5571c0a6c915-workload-socket\") pod \"e2e-ait-a4a82f5f-openshift-default-7f7f96dc7c-5wbrz\" (UID: \"443386e3-0ac0-4ac0-a04e-5571c0a6c915\") " pod="openshift-ingress/e2e-ait-a4a82f5f-openshift-default-7f7f96dc7c-5wbrz" Jul 21 21:29:06.140483 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:29:06.140390 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/443386e3-0ac0-4ac0-a04e-5571c0a6c915-istiod-ca-cert\") pod \"e2e-ait-a4a82f5f-openshift-default-7f7f96dc7c-5wbrz\" (UID: \"443386e3-0ac0-4ac0-a04e-5571c0a6c915\") " pod="openshift-ingress/e2e-ait-a4a82f5f-openshift-default-7f7f96dc7c-5wbrz" Jul 21 21:29:06.140483 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:29:06.140425 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/443386e3-0ac0-4ac0-a04e-5571c0a6c915-workload-certs\") pod \"e2e-ait-a4a82f5f-openshift-default-7f7f96dc7c-5wbrz\" (UID: \"443386e3-0ac0-4ac0-a04e-5571c0a6c915\") " pod="openshift-ingress/e2e-ait-a4a82f5f-openshift-default-7f7f96dc7c-5wbrz" Jul 21 21:29:06.140483 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:29:06.140478 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/443386e3-0ac0-4ac0-a04e-5571c0a6c915-credential-socket\") pod \"e2e-ait-a4a82f5f-openshift-default-7f7f96dc7c-5wbrz\" (UID: \"443386e3-0ac0-4ac0-a04e-5571c0a6c915\") " pod="openshift-ingress/e2e-ait-a4a82f5f-openshift-default-7f7f96dc7c-5wbrz" Jul 21 21:29:06.140736 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:29:06.140501 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/443386e3-0ac0-4ac0-a04e-5571c0a6c915-istio-data\") pod \"e2e-ait-a4a82f5f-openshift-default-7f7f96dc7c-5wbrz\" (UID: \"443386e3-0ac0-4ac0-a04e-5571c0a6c915\") " pod="openshift-ingress/e2e-ait-a4a82f5f-openshift-default-7f7f96dc7c-5wbrz" Jul 21 21:29:06.140736 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:29:06.140649 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/443386e3-0ac0-4ac0-a04e-5571c0a6c915-istio-token\") pod \"e2e-ait-a4a82f5f-openshift-default-7f7f96dc7c-5wbrz\" (UID: \"443386e3-0ac0-4ac0-a04e-5571c0a6c915\") " pod="openshift-ingress/e2e-ait-a4a82f5f-openshift-default-7f7f96dc7c-5wbrz" Jul 21 21:29:06.140736 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:29:06.140722 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/443386e3-0ac0-4ac0-a04e-5571c0a6c915-istio-podinfo\") pod \"e2e-ait-a4a82f5f-openshift-default-7f7f96dc7c-5wbrz\" (UID: \"443386e3-0ac0-4ac0-a04e-5571c0a6c915\") " pod="openshift-ingress/e2e-ait-a4a82f5f-openshift-default-7f7f96dc7c-5wbrz" Jul 21 21:29:06.140896 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:29:06.140754 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kmzkd\" (UniqueName: \"kubernetes.io/projected/443386e3-0ac0-4ac0-a04e-5571c0a6c915-kube-api-access-kmzkd\") pod \"e2e-ait-a4a82f5f-openshift-default-7f7f96dc7c-5wbrz\" (UID: \"443386e3-0ac0-4ac0-a04e-5571c0a6c915\") " pod="openshift-ingress/e2e-ait-a4a82f5f-openshift-default-7f7f96dc7c-5wbrz" Jul 21 21:29:06.141048 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:29:06.140975 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/443386e3-0ac0-4ac0-a04e-5571c0a6c915-istio-data\") pod \"e2e-ait-a4a82f5f-openshift-default-7f7f96dc7c-5wbrz\" (UID: \"443386e3-0ac0-4ac0-a04e-5571c0a6c915\") " pod="openshift-ingress/e2e-ait-a4a82f5f-openshift-default-7f7f96dc7c-5wbrz" Jul 21 21:29:06.141048 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:29:06.140978 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/443386e3-0ac0-4ac0-a04e-5571c0a6c915-credential-socket\") pod \"e2e-ait-a4a82f5f-openshift-default-7f7f96dc7c-5wbrz\" (UID: \"443386e3-0ac0-4ac0-a04e-5571c0a6c915\") " pod="openshift-ingress/e2e-ait-a4a82f5f-openshift-default-7f7f96dc7c-5wbrz" Jul 21 21:29:06.141229 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:29:06.141209 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/443386e3-0ac0-4ac0-a04e-5571c0a6c915-workload-socket\") pod \"e2e-ait-a4a82f5f-openshift-default-7f7f96dc7c-5wbrz\" (UID: \"443386e3-0ac0-4ac0-a04e-5571c0a6c915\") " pod="openshift-ingress/e2e-ait-a4a82f5f-openshift-default-7f7f96dc7c-5wbrz" Jul 21 21:29:06.141294 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:29:06.141241 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/443386e3-0ac0-4ac0-a04e-5571c0a6c915-istiod-ca-cert\") pod \"e2e-ait-a4a82f5f-openshift-default-7f7f96dc7c-5wbrz\" (UID: \"443386e3-0ac0-4ac0-a04e-5571c0a6c915\") " pod="openshift-ingress/e2e-ait-a4a82f5f-openshift-default-7f7f96dc7c-5wbrz" Jul 21 21:29:06.141294 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:29:06.141262 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/443386e3-0ac0-4ac0-a04e-5571c0a6c915-workload-certs\") pod \"e2e-ait-a4a82f5f-openshift-default-7f7f96dc7c-5wbrz\" (UID: \"443386e3-0ac0-4ac0-a04e-5571c0a6c915\") " pod="openshift-ingress/e2e-ait-a4a82f5f-openshift-default-7f7f96dc7c-5wbrz" Jul 21 21:29:06.142691 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:29:06.142672 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/443386e3-0ac0-4ac0-a04e-5571c0a6c915-istio-envoy\") pod \"e2e-ait-a4a82f5f-openshift-default-7f7f96dc7c-5wbrz\" (UID: \"443386e3-0ac0-4ac0-a04e-5571c0a6c915\") " pod="openshift-ingress/e2e-ait-a4a82f5f-openshift-default-7f7f96dc7c-5wbrz" Jul 21 21:29:06.143429 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:29:06.143409 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/443386e3-0ac0-4ac0-a04e-5571c0a6c915-istio-podinfo\") pod \"e2e-ait-a4a82f5f-openshift-default-7f7f96dc7c-5wbrz\" (UID: \"443386e3-0ac0-4ac0-a04e-5571c0a6c915\") " pod="openshift-ingress/e2e-ait-a4a82f5f-openshift-default-7f7f96dc7c-5wbrz" Jul 21 21:29:06.148926 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:29:06.148901 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kmzkd\" (UniqueName: \"kubernetes.io/projected/443386e3-0ac0-4ac0-a04e-5571c0a6c915-kube-api-access-kmzkd\") pod \"e2e-ait-a4a82f5f-openshift-default-7f7f96dc7c-5wbrz\" (UID: \"443386e3-0ac0-4ac0-a04e-5571c0a6c915\") " pod="openshift-ingress/e2e-ait-a4a82f5f-openshift-default-7f7f96dc7c-5wbrz" Jul 21 21:29:06.149020 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:29:06.148985 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/443386e3-0ac0-4ac0-a04e-5571c0a6c915-istio-token\") pod \"e2e-ait-a4a82f5f-openshift-default-7f7f96dc7c-5wbrz\" (UID: \"443386e3-0ac0-4ac0-a04e-5571c0a6c915\") " pod="openshift-ingress/e2e-ait-a4a82f5f-openshift-default-7f7f96dc7c-5wbrz" Jul 21 21:29:06.294930 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:29:06.294844 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-a4a82f5f-openshift-default-7f7f96dc7c-5wbrz" Jul 21 21:29:06.429307 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:29:06.429282 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-a4a82f5f-openshift-default-7f7f96dc7c-5wbrz"] Jul 21 21:29:06.431111 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:29:06.431082 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod443386e3_0ac0_4ac0_a04e_5571c0a6c915.slice/crio-cad545c25971bb9280a39dcb10d197f685d110fde84d7d10ad5cc5e697d0a50e WatchSource:0}: Error finding container cad545c25971bb9280a39dcb10d197f685d110fde84d7d10ad5cc5e697d0a50e: Status 404 returned error can't find the container with id cad545c25971bb9280a39dcb10d197f685d110fde84d7d10ad5cc5e697d0a50e Jul 21 21:29:06.433166 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:29:06.433135 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 21 21:29:06.433273 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:29:06.433214 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 21 21:29:06.433273 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:29:06.433262 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 21 21:29:06.609575 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:29:06.609514 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-a4a82f5f-openshift-default-7f7f96dc7c-5wbrz" event={"ID":"443386e3-0ac0-4ac0-a04e-5571c0a6c915","Type":"ContainerStarted","Data":"665bfb4c6b783c31377867ac2fa332513387a570d353ab51c630fa8d61fd112a"} Jul 21 21:29:06.609732 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:29:06.609597 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-a4a82f5f-openshift-default-7f7f96dc7c-5wbrz" event={"ID":"443386e3-0ac0-4ac0-a04e-5571c0a6c915","Type":"ContainerStarted","Data":"cad545c25971bb9280a39dcb10d197f685d110fde84d7d10ad5cc5e697d0a50e"} Jul 21 21:29:06.628474 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:29:06.628420 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-a4a82f5f-openshift-default-7f7f96dc7c-5wbrz" podStartSLOduration=1.6284049390000002 podStartE2EDuration="1.628404939s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 21:29:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 21:29:06.62786611 +0000 UTC m=+1704.562615776" watchObservedRunningTime="2026-07-21 21:29:06.628404939 +0000 UTC m=+1704.563154605" Jul 21 21:29:06.742321 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:29:06.742279 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f5dc8db0-a3f4-468f-8841-b399f0561f5b" path="/var/lib/kubelet/pods/f5dc8db0-a3f4-468f-8841-b399f0561f5b/volumes" Jul 21 21:29:07.296044 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:29:07.296009 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-a4a82f5f-openshift-default-7f7f96dc7c-5wbrz" Jul 21 21:29:07.300964 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:29:07.300930 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-a4a82f5f-openshift-default-7f7f96dc7c-5wbrz" Jul 21 21:29:07.614750 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:29:07.614721 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-a4a82f5f-openshift-default-7f7f96dc7c-5wbrz" Jul 21 21:29:07.615735 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:29:07.615719 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-a4a82f5f-openshift-default-7f7f96dc7c-5wbrz" Jul 21 21:29:10.423044 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:29:10.423018 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-1710f404-gw-openshift-default-7ffdcd7b87-f692z" Jul 21 21:29:10.487329 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:29:10.487254 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bfa501fa-77bd-4550-a865-d8a95bd05a2f-istio-token\") pod \"bfa501fa-77bd-4550-a865-d8a95bd05a2f\" (UID: \"bfa501fa-77bd-4550-a865-d8a95bd05a2f\") " Jul 21 21:29:10.487329 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:29:10.487316 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bfa501fa-77bd-4550-a865-d8a95bd05a2f-credential-socket\") pod \"bfa501fa-77bd-4550-a865-d8a95bd05a2f\" (UID: \"bfa501fa-77bd-4550-a865-d8a95bd05a2f\") " Jul 21 21:29:10.487541 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:29:10.487337 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-x68m9\" (UniqueName: \"kubernetes.io/projected/bfa501fa-77bd-4550-a865-d8a95bd05a2f-kube-api-access-x68m9\") pod \"bfa501fa-77bd-4550-a865-d8a95bd05a2f\" (UID: \"bfa501fa-77bd-4550-a865-d8a95bd05a2f\") " Jul 21 21:29:10.487541 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:29:10.487380 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bfa501fa-77bd-4550-a865-d8a95bd05a2f-istiod-ca-cert\") pod \"bfa501fa-77bd-4550-a865-d8a95bd05a2f\" (UID: \"bfa501fa-77bd-4550-a865-d8a95bd05a2f\") " Jul 21 21:29:10.487541 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:29:10.487399 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bfa501fa-77bd-4550-a865-d8a95bd05a2f-workload-socket\") pod \"bfa501fa-77bd-4550-a865-d8a95bd05a2f\" (UID: \"bfa501fa-77bd-4550-a865-d8a95bd05a2f\") " Jul 21 21:29:10.487541 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:29:10.487426 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bfa501fa-77bd-4550-a865-d8a95bd05a2f-istio-data\") pod \"bfa501fa-77bd-4550-a865-d8a95bd05a2f\" (UID: \"bfa501fa-77bd-4550-a865-d8a95bd05a2f\") " Jul 21 21:29:10.487745 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:29:10.487607 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bfa501fa-77bd-4550-a865-d8a95bd05a2f-workload-certs\") pod \"bfa501fa-77bd-4550-a865-d8a95bd05a2f\" (UID: \"bfa501fa-77bd-4550-a865-d8a95bd05a2f\") " Jul 21 21:29:10.487745 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:29:10.487639 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bfa501fa-77bd-4550-a865-d8a95bd05a2f-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "bfa501fa-77bd-4550-a865-d8a95bd05a2f" (UID: "bfa501fa-77bd-4550-a865-d8a95bd05a2f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 21:29:10.487745 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:29:10.487683 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bfa501fa-77bd-4550-a865-d8a95bd05a2f-istio-podinfo\") pod \"bfa501fa-77bd-4550-a865-d8a95bd05a2f\" (UID: \"bfa501fa-77bd-4550-a865-d8a95bd05a2f\") " Jul 21 21:29:10.488209 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:29:10.487745 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bfa501fa-77bd-4550-a865-d8a95bd05a2f-istio-envoy\") pod \"bfa501fa-77bd-4550-a865-d8a95bd05a2f\" (UID: \"bfa501fa-77bd-4550-a865-d8a95bd05a2f\") " Jul 21 21:29:10.488209 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:29:10.487751 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bfa501fa-77bd-4550-a865-d8a95bd05a2f-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "bfa501fa-77bd-4550-a865-d8a95bd05a2f" (UID: "bfa501fa-77bd-4550-a865-d8a95bd05a2f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 21:29:10.488209 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:29:10.487914 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bfa501fa-77bd-4550-a865-d8a95bd05a2f-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "bfa501fa-77bd-4550-a865-d8a95bd05a2f" (UID: "bfa501fa-77bd-4550-a865-d8a95bd05a2f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 21:29:10.488209 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:29:10.487960 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bfa501fa-77bd-4550-a865-d8a95bd05a2f-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "bfa501fa-77bd-4550-a865-d8a95bd05a2f" (UID: "bfa501fa-77bd-4550-a865-d8a95bd05a2f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 21 21:29:10.488209 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:29:10.488148 2575 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bfa501fa-77bd-4550-a865-d8a95bd05a2f-credential-socket\") on node \"ip-10-0-140-97.ec2.internal\" DevicePath \"\"" Jul 21 21:29:10.488209 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:29:10.488170 2575 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bfa501fa-77bd-4550-a865-d8a95bd05a2f-istiod-ca-cert\") on node \"ip-10-0-140-97.ec2.internal\" DevicePath \"\"" Jul 21 21:29:10.488209 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:29:10.488185 2575 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bfa501fa-77bd-4550-a865-d8a95bd05a2f-workload-socket\") on node \"ip-10-0-140-97.ec2.internal\" DevicePath \"\"" Jul 21 21:29:10.488209 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:29:10.488197 2575 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bfa501fa-77bd-4550-a865-d8a95bd05a2f-workload-certs\") on node \"ip-10-0-140-97.ec2.internal\" DevicePath \"\"" Jul 21 21:29:10.488926 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:29:10.488901 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bfa501fa-77bd-4550-a865-d8a95bd05a2f-istio-data" (OuterVolumeSpecName: "istio-data") pod "bfa501fa-77bd-4550-a865-d8a95bd05a2f" (UID: "bfa501fa-77bd-4550-a865-d8a95bd05a2f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 21:29:10.489628 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:29:10.489598 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bfa501fa-77bd-4550-a865-d8a95bd05a2f-kube-api-access-x68m9" (OuterVolumeSpecName: "kube-api-access-x68m9") pod "bfa501fa-77bd-4550-a865-d8a95bd05a2f" (UID: "bfa501fa-77bd-4550-a865-d8a95bd05a2f"). InnerVolumeSpecName "kube-api-access-x68m9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 21 21:29:10.489827 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:29:10.489802 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/bfa501fa-77bd-4550-a865-d8a95bd05a2f-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "bfa501fa-77bd-4550-a865-d8a95bd05a2f" (UID: "bfa501fa-77bd-4550-a865-d8a95bd05a2f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 21 21:29:10.489933 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:29:10.489915 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bfa501fa-77bd-4550-a865-d8a95bd05a2f-istio-token" (OuterVolumeSpecName: "istio-token") pod "bfa501fa-77bd-4550-a865-d8a95bd05a2f" (UID: "bfa501fa-77bd-4550-a865-d8a95bd05a2f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 21 21:29:10.490008 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:29:10.489994 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bfa501fa-77bd-4550-a865-d8a95bd05a2f-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "bfa501fa-77bd-4550-a865-d8a95bd05a2f" (UID: "bfa501fa-77bd-4550-a865-d8a95bd05a2f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 21:29:10.589553 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:29:10.589500 2575 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bfa501fa-77bd-4550-a865-d8a95bd05a2f-istio-data\") on node \"ip-10-0-140-97.ec2.internal\" DevicePath \"\"" Jul 21 21:29:10.589553 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:29:10.589525 2575 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bfa501fa-77bd-4550-a865-d8a95bd05a2f-istio-podinfo\") on node \"ip-10-0-140-97.ec2.internal\" DevicePath \"\"" Jul 21 21:29:10.589553 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:29:10.589557 2575 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bfa501fa-77bd-4550-a865-d8a95bd05a2f-istio-envoy\") on node \"ip-10-0-140-97.ec2.internal\" DevicePath \"\"" Jul 21 21:29:10.589553 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:29:10.589565 2575 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bfa501fa-77bd-4550-a865-d8a95bd05a2f-istio-token\") on node \"ip-10-0-140-97.ec2.internal\" DevicePath \"\"" Jul 21 21:29:10.589799 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:29:10.589575 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x68m9\" (UniqueName: \"kubernetes.io/projected/bfa501fa-77bd-4550-a865-d8a95bd05a2f-kube-api-access-x68m9\") on node \"ip-10-0-140-97.ec2.internal\" DevicePath \"\"" Jul 21 21:29:10.630394 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:29:10.630366 2575 generic.go:356] "Generic (PLEG): container finished" podID="bfa501fa-77bd-4550-a865-d8a95bd05a2f" containerID="344f751ed48fa3e4cf20224f31e49bbbeddadff43b350141adb044bfae02389f" exitCode=0 Jul 21 21:29:10.630561 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:29:10.630416 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-1710f404-gw-openshift-default-7ffdcd7b87-f692z" Jul 21 21:29:10.630561 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:29:10.630458 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-1710f404-gw-openshift-default-7ffdcd7b87-f692z" event={"ID":"bfa501fa-77bd-4550-a865-d8a95bd05a2f","Type":"ContainerDied","Data":"344f751ed48fa3e4cf20224f31e49bbbeddadff43b350141adb044bfae02389f"} Jul 21 21:29:10.630650 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:29:10.630562 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-1710f404-gw-openshift-default-7ffdcd7b87-f692z" event={"ID":"bfa501fa-77bd-4550-a865-d8a95bd05a2f","Type":"ContainerDied","Data":"96eb230883cad3abf9c1d7527ff8ac57e0034db6fdf54139c15fe717f3c3e5c0"} Jul 21 21:29:10.630650 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:29:10.630588 2575 scope.go:117] "RemoveContainer" containerID="344f751ed48fa3e4cf20224f31e49bbbeddadff43b350141adb044bfae02389f" Jul 21 21:29:10.642116 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:29:10.641281 2575 scope.go:117] "RemoveContainer" containerID="7395d1616531dde01a5609f73dd8de546282b3c500bcc08e88b6a8ad0530bf60" Jul 21 21:29:10.651820 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:29:10.651619 2575 scope.go:117] "RemoveContainer" containerID="344f751ed48fa3e4cf20224f31e49bbbeddadff43b350141adb044bfae02389f" Jul 21 21:29:10.651926 ip-10-0-140-97 kubenswrapper[2575]: E0721 21:29:10.651902 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"344f751ed48fa3e4cf20224f31e49bbbeddadff43b350141adb044bfae02389f\": container with ID starting with 344f751ed48fa3e4cf20224f31e49bbbeddadff43b350141adb044bfae02389f not found: ID does not exist" containerID="344f751ed48fa3e4cf20224f31e49bbbeddadff43b350141adb044bfae02389f" Jul 21 21:29:10.652017 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:29:10.651933 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"344f751ed48fa3e4cf20224f31e49bbbeddadff43b350141adb044bfae02389f"} err="failed to get container status \"344f751ed48fa3e4cf20224f31e49bbbeddadff43b350141adb044bfae02389f\": rpc error: code = NotFound desc = could not find container \"344f751ed48fa3e4cf20224f31e49bbbeddadff43b350141adb044bfae02389f\": container with ID starting with 344f751ed48fa3e4cf20224f31e49bbbeddadff43b350141adb044bfae02389f not found: ID does not exist" Jul 21 21:29:10.652017 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:29:10.651952 2575 scope.go:117] "RemoveContainer" containerID="7395d1616531dde01a5609f73dd8de546282b3c500bcc08e88b6a8ad0530bf60" Jul 21 21:29:10.652207 ip-10-0-140-97 kubenswrapper[2575]: E0721 21:29:10.652189 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7395d1616531dde01a5609f73dd8de546282b3c500bcc08e88b6a8ad0530bf60\": container with ID starting with 7395d1616531dde01a5609f73dd8de546282b3c500bcc08e88b6a8ad0530bf60 not found: ID does not exist" containerID="7395d1616531dde01a5609f73dd8de546282b3c500bcc08e88b6a8ad0530bf60" Jul 21 21:29:10.652256 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:29:10.652216 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7395d1616531dde01a5609f73dd8de546282b3c500bcc08e88b6a8ad0530bf60"} err="failed to get container status \"7395d1616531dde01a5609f73dd8de546282b3c500bcc08e88b6a8ad0530bf60\": rpc error: code = NotFound desc = could not find container \"7395d1616531dde01a5609f73dd8de546282b3c500bcc08e88b6a8ad0530bf60\": container with ID starting with 7395d1616531dde01a5609f73dd8de546282b3c500bcc08e88b6a8ad0530bf60 not found: ID does not exist" Jul 21 21:29:10.652466 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:29:10.652449 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-1710f404-gw-openshift-default-7ffdcd7b87-f692z"] Jul 21 21:29:10.656503 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:29:10.656483 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-migrate-1710f404-gw-openshift-default-7ffdcd7b87-f692z"] Jul 21 21:29:10.741685 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:29:10.741596 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bfa501fa-77bd-4550-a865-d8a95bd05a2f" path="/var/lib/kubelet/pods/bfa501fa-77bd-4550-a865-d8a95bd05a2f/volumes" Jul 21 21:29:22.615869 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:29:22.615841 2575 patch_prober.go:28] interesting pod/e2e-ait-a4a82f5f-openshift-default-7f7f96dc7c-5wbrz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" start-of-body= Jul 21 21:29:22.616208 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:29:22.615899 2575 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-a4a82f5f-openshift-default-7f7f96dc7c-5wbrz" podUID="443386e3-0ac0-4ac0-a04e-5571c0a6c915" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" Jul 21 21:29:22.701658 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:29:22.701626 2575 generic.go:356] "Generic (PLEG): container finished" podID="443386e3-0ac0-4ac0-a04e-5571c0a6c915" containerID="665bfb4c6b783c31377867ac2fa332513387a570d353ab51c630fa8d61fd112a" exitCode=0 Jul 21 21:29:22.701778 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:29:22.701696 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-a4a82f5f-openshift-default-7f7f96dc7c-5wbrz" event={"ID":"443386e3-0ac0-4ac0-a04e-5571c0a6c915","Type":"ContainerDied","Data":"665bfb4c6b783c31377867ac2fa332513387a570d353ab51c630fa8d61fd112a"} Jul 21 21:29:22.702087 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:29:22.702072 2575 scope.go:117] "RemoveContainer" containerID="665bfb4c6b783c31377867ac2fa332513387a570d353ab51c630fa8d61fd112a" Jul 21 21:29:22.703141 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:29:22.703110 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 21 21:29:22.703214 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:29:22.703188 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 21 21:29:22.703255 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:29:22.703222 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 21 21:29:23.714283 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:29:23.714245 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-a4a82f5f-openshift-default-7f7f96dc7c-5wbrz" event={"ID":"443386e3-0ac0-4ac0-a04e-5571c0a6c915","Type":"ContainerStarted","Data":"a1541f9dc2883837ec181b9b84f48fc96da7a020831442b5ace00dc799214a38"} Jul 21 21:29:24.295487 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:29:24.295450 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-a4a82f5f-openshift-default-7f7f96dc7c-5wbrz" Jul 21 21:29:24.295686 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:29:24.295670 2575 patch_prober.go:28] interesting pod/e2e-ait-a4a82f5f-openshift-default-7f7f96dc7c-5wbrz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" start-of-body= Jul 21 21:29:24.295739 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:29:24.295716 2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-a4a82f5f-openshift-default-7f7f96dc7c-5wbrz" podUID="443386e3-0ac0-4ac0-a04e-5571c0a6c915" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" Jul 21 21:29:25.295192 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:29:25.295156 2575 patch_prober.go:28] interesting pod/e2e-ait-a4a82f5f-openshift-default-7f7f96dc7c-5wbrz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" start-of-body= Jul 21 21:29:25.295602 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:29:25.295223 2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-a4a82f5f-openshift-default-7f7f96dc7c-5wbrz" podUID="443386e3-0ac0-4ac0-a04e-5571c0a6c915" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" Jul 21 21:29:26.295514 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:29:26.295476 2575 patch_prober.go:28] interesting pod/e2e-ait-a4a82f5f-openshift-default-7f7f96dc7c-5wbrz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" start-of-body= Jul 21 21:29:26.295928 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:29:26.295572 2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-a4a82f5f-openshift-default-7f7f96dc7c-5wbrz" podUID="443386e3-0ac0-4ac0-a04e-5571c0a6c915" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" Jul 21 21:29:27.296253 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:29:27.296216 2575 patch_prober.go:28] interesting pod/e2e-ait-a4a82f5f-openshift-default-7f7f96dc7c-5wbrz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" start-of-body= Jul 21 21:29:27.296723 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:29:27.296275 2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-a4a82f5f-openshift-default-7f7f96dc7c-5wbrz" podUID="443386e3-0ac0-4ac0-a04e-5571c0a6c915" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" Jul 21 21:29:28.298811 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:29:28.298783 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-a4a82f5f-openshift-default-7f7f96dc7c-5wbrz" Jul 21 21:29:28.299223 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:29:28.299128 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-a4a82f5f-openshift-default-7f7f96dc7c-5wbrz" Jul 21 21:29:28.299908 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:29:28.299888 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-a4a82f5f-openshift-default-7f7f96dc7c-5wbrz" Jul 21 21:29:33.879653 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:29:33.879619 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a4a82f5f-z874d"] Jul 21 21:29:33.880052 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:29:33.879975 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bfa501fa-77bd-4550-a865-d8a95bd05a2f" containerName="istio-proxy" Jul 21 21:29:33.880052 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:29:33.879986 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="bfa501fa-77bd-4550-a865-d8a95bd05a2f" containerName="istio-proxy" Jul 21 21:29:33.880052 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:29:33.880001 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bfa501fa-77bd-4550-a865-d8a95bd05a2f" containerName="istio-proxy" Jul 21 21:29:33.880052 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:29:33.880007 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="bfa501fa-77bd-4550-a865-d8a95bd05a2f" containerName="istio-proxy" Jul 21 21:29:33.880257 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:29:33.880063 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="bfa501fa-77bd-4550-a865-d8a95bd05a2f" containerName="istio-proxy" Jul 21 21:29:33.882288 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:29:33.882272 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a4a82f5f-z874d" Jul 21 21:29:33.884982 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:29:33.884948 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9fxhr\"" Jul 21 21:29:33.885103 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:29:33.884970 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 21 21:29:33.889876 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:29:33.889849 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a4a82f5f-z874d"] Jul 21 21:29:33.920665 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:29:33.920634 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a62245c6-2728-44af-b13f-6510c64ca86d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-a4a82f5f-z874d\" (UID: \"a62245c6-2728-44af-b13f-6510c64ca86d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a4a82f5f-z874d" Jul 21 21:29:34.021916 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:29:34.021885 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a62245c6-2728-44af-b13f-6510c64ca86d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-a4a82f5f-z874d\" (UID: \"a62245c6-2728-44af-b13f-6510c64ca86d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a4a82f5f-z874d" Jul 21 21:29:34.022489 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:29:34.022468 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a62245c6-2728-44af-b13f-6510c64ca86d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-a4a82f5f-z874d\" (UID: \"a62245c6-2728-44af-b13f-6510c64ca86d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a4a82f5f-z874d" Jul 21 21:29:34.194408 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:29:34.194328 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a4a82f5f-z874d" Jul 21 21:29:34.323070 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:29:34.323037 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a4a82f5f-z874d"] Jul 21 21:29:34.326763 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:29:34.326733 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda62245c6_2728_44af_b13f_6510c64ca86d.slice/crio-21949264b74abba6f777bbd0f21de405be0bbcbd41130ee9733cb62466920300 WatchSource:0}: Error finding container 21949264b74abba6f777bbd0f21de405be0bbcbd41130ee9733cb62466920300: Status 404 returned error can't find the container with id 21949264b74abba6f777bbd0f21de405be0bbcbd41130ee9733cb62466920300 Jul 21 21:29:34.771463 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:29:34.771431 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a4a82f5f-z874d" event={"ID":"a62245c6-2728-44af-b13f-6510c64ca86d","Type":"ContainerStarted","Data":"1c1ba03bd8684301edec141a10685a2c46748d7c54feac68e388aab57d144ff2"} Jul 21 21:29:34.771651 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:29:34.771486 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a4a82f5f-z874d" event={"ID":"a62245c6-2728-44af-b13f-6510c64ca86d","Type":"ContainerStarted","Data":"21949264b74abba6f777bbd0f21de405be0bbcbd41130ee9733cb62466920300"} Jul 21 21:29:34.787916 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:29:34.787871 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a4a82f5f-z874d" podStartSLOduration=1.787852802 podStartE2EDuration="1.787852802s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 21:29:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 21:29:34.786289086 +0000 UTC m=+1732.721038765" watchObservedRunningTime="2026-07-21 21:29:34.787852802 +0000 UTC m=+1732.722602468" Jul 21 21:29:35.778176 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:29:35.778143 2575 generic.go:356] "Generic (PLEG): container finished" podID="a62245c6-2728-44af-b13f-6510c64ca86d" containerID="1c1ba03bd8684301edec141a10685a2c46748d7c54feac68e388aab57d144ff2" exitCode=0 Jul 21 21:29:35.778577 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:29:35.778214 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a4a82f5f-z874d" event={"ID":"a62245c6-2728-44af-b13f-6510c64ca86d","Type":"ContainerDied","Data":"1c1ba03bd8684301edec141a10685a2c46748d7c54feac68e388aab57d144ff2"} Jul 21 21:29:36.913963 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:29:36.913936 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a4a82f5f-z874d" Jul 21 21:29:36.948074 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:29:36.948049 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a62245c6-2728-44af-b13f-6510c64ca86d-maas-api-service-ca\") pod \"a62245c6-2728-44af-b13f-6510c64ca86d\" (UID: \"a62245c6-2728-44af-b13f-6510c64ca86d\") " Jul 21 21:29:36.948399 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:29:36.948373 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a62245c6-2728-44af-b13f-6510c64ca86d-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "a62245c6-2728-44af-b13f-6510c64ca86d" (UID: "a62245c6-2728-44af-b13f-6510c64ca86d"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 21 21:29:37.049620 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:29:37.049549 2575 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a62245c6-2728-44af-b13f-6510c64ca86d-maas-api-service-ca\") on node \"ip-10-0-140-97.ec2.internal\" DevicePath \"\"" Jul 21 21:29:37.788504 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:29:37.788477 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a4a82f5f-z874d" Jul 21 21:29:37.788704 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:29:37.788539 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a4a82f5f-z874d" event={"ID":"a62245c6-2728-44af-b13f-6510c64ca86d","Type":"ContainerDied","Data":"21949264b74abba6f777bbd0f21de405be0bbcbd41130ee9733cb62466920300"} Jul 21 21:29:37.788704 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:29:37.788590 2575 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="21949264b74abba6f777bbd0f21de405be0bbcbd41130ee9733cb62466920300" Jul 21 21:29:48.954698 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:29:48.954660 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a4a82f5f-z874d"] Jul 21 21:29:48.956399 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:29:48.956376 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a4a82f5f-z874d"] Jul 21 21:29:50.245211 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:29:50.245173 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-a4a82f5f-openshift-default-7f7f96dc7c-5wbrz"] Jul 21 21:29:50.245689 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:29:50.245392 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-a4a82f5f-openshift-default-7f7f96dc7c-5wbrz" podUID="443386e3-0ac0-4ac0-a04e-5571c0a6c915" containerName="istio-proxy" containerID="cri-o://a1541f9dc2883837ec181b9b84f48fc96da7a020831442b5ace00dc799214a38" gracePeriod=30 Jul 21 21:29:50.741249 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:29:50.741220 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a62245c6-2728-44af-b13f-6510c64ca86d" path="/var/lib/kubelet/pods/a62245c6-2728-44af-b13f-6510c64ca86d/volumes" Jul 21 21:29:51.078748 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:29:51.078670 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-derive-e49e6eb7-openshift-default-5b4cc56b7-nfcp5"] Jul 21 21:29:51.079070 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:29:51.079056 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a62245c6-2728-44af-b13f-6510c64ca86d" containerName="revoke-keys" Jul 21 21:29:51.079124 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:29:51.079071 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="a62245c6-2728-44af-b13f-6510c64ca86d" containerName="revoke-keys" Jul 21 21:29:51.079162 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:29:51.079137 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="a62245c6-2728-44af-b13f-6510c64ca86d" containerName="revoke-keys" Jul 21 21:29:51.079162 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:29:51.079150 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="bfa501fa-77bd-4550-a865-d8a95bd05a2f" containerName="istio-proxy" Jul 21 21:29:51.082146 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:29:51.082127 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-e49e6eb7-openshift-default-5b4cc56b7-nfcp5" Jul 21 21:29:51.084676 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:29:51.084655 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-derive-e49e6eb7-openshift-default-dockercfg-tkgjn\"" Jul 21 21:29:51.099063 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:29:51.099038 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-e49e6eb7-openshift-default-5b4cc56b7-nfcp5"] Jul 21 21:29:51.175616 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:29:51.175582 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dc4ff709-fd03-44a0-9610-2549523ca75d-istio-token\") pod \"e2e-derive-e49e6eb7-openshift-default-5b4cc56b7-nfcp5\" (UID: \"dc4ff709-fd03-44a0-9610-2549523ca75d\") " pod="openshift-ingress/e2e-derive-e49e6eb7-openshift-default-5b4cc56b7-nfcp5" Jul 21 21:29:51.175616 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:29:51.175621 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-49qtm\" (UniqueName: \"kubernetes.io/projected/dc4ff709-fd03-44a0-9610-2549523ca75d-kube-api-access-49qtm\") pod \"e2e-derive-e49e6eb7-openshift-default-5b4cc56b7-nfcp5\" (UID: \"dc4ff709-fd03-44a0-9610-2549523ca75d\") " pod="openshift-ingress/e2e-derive-e49e6eb7-openshift-default-5b4cc56b7-nfcp5" Jul 21 21:29:51.175874 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:29:51.175651 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dc4ff709-fd03-44a0-9610-2549523ca75d-istiod-ca-cert\") pod \"e2e-derive-e49e6eb7-openshift-default-5b4cc56b7-nfcp5\" (UID: \"dc4ff709-fd03-44a0-9610-2549523ca75d\") " pod="openshift-ingress/e2e-derive-e49e6eb7-openshift-default-5b4cc56b7-nfcp5" Jul 21 21:29:51.175874 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:29:51.175744 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dc4ff709-fd03-44a0-9610-2549523ca75d-istio-envoy\") pod \"e2e-derive-e49e6eb7-openshift-default-5b4cc56b7-nfcp5\" (UID: \"dc4ff709-fd03-44a0-9610-2549523ca75d\") " pod="openshift-ingress/e2e-derive-e49e6eb7-openshift-default-5b4cc56b7-nfcp5" Jul 21 21:29:51.175874 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:29:51.175798 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc4ff709-fd03-44a0-9610-2549523ca75d-credential-socket\") pod \"e2e-derive-e49e6eb7-openshift-default-5b4cc56b7-nfcp5\" (UID: \"dc4ff709-fd03-44a0-9610-2549523ca75d\") " pod="openshift-ingress/e2e-derive-e49e6eb7-openshift-default-5b4cc56b7-nfcp5" Jul 21 21:29:51.176042 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:29:51.175894 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dc4ff709-fd03-44a0-9610-2549523ca75d-workload-certs\") pod \"e2e-derive-e49e6eb7-openshift-default-5b4cc56b7-nfcp5\" (UID: \"dc4ff709-fd03-44a0-9610-2549523ca75d\") " pod="openshift-ingress/e2e-derive-e49e6eb7-openshift-default-5b4cc56b7-nfcp5" Jul 21 21:29:51.176042 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:29:51.175923 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dc4ff709-fd03-44a0-9610-2549523ca75d-istio-data\") pod \"e2e-derive-e49e6eb7-openshift-default-5b4cc56b7-nfcp5\" (UID: \"dc4ff709-fd03-44a0-9610-2549523ca75d\") " pod="openshift-ingress/e2e-derive-e49e6eb7-openshift-default-5b4cc56b7-nfcp5" Jul 21 21:29:51.176042 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:29:51.175984 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dc4ff709-fd03-44a0-9610-2549523ca75d-istio-podinfo\") pod \"e2e-derive-e49e6eb7-openshift-default-5b4cc56b7-nfcp5\" (UID: \"dc4ff709-fd03-44a0-9610-2549523ca75d\") " pod="openshift-ingress/e2e-derive-e49e6eb7-openshift-default-5b4cc56b7-nfcp5" Jul 21 21:29:51.176202 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:29:51.176048 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc4ff709-fd03-44a0-9610-2549523ca75d-workload-socket\") pod \"e2e-derive-e49e6eb7-openshift-default-5b4cc56b7-nfcp5\" (UID: \"dc4ff709-fd03-44a0-9610-2549523ca75d\") " pod="openshift-ingress/e2e-derive-e49e6eb7-openshift-default-5b4cc56b7-nfcp5" Jul 21 21:29:51.276897 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:29:51.276860 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dc4ff709-fd03-44a0-9610-2549523ca75d-istio-token\") pod \"e2e-derive-e49e6eb7-openshift-default-5b4cc56b7-nfcp5\" (UID: \"dc4ff709-fd03-44a0-9610-2549523ca75d\") " pod="openshift-ingress/e2e-derive-e49e6eb7-openshift-default-5b4cc56b7-nfcp5" Jul 21 21:29:51.276897 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:29:51.276899 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-49qtm\" (UniqueName: \"kubernetes.io/projected/dc4ff709-fd03-44a0-9610-2549523ca75d-kube-api-access-49qtm\") pod \"e2e-derive-e49e6eb7-openshift-default-5b4cc56b7-nfcp5\" (UID: \"dc4ff709-fd03-44a0-9610-2549523ca75d\") " pod="openshift-ingress/e2e-derive-e49e6eb7-openshift-default-5b4cc56b7-nfcp5" Jul 21 21:29:51.277346 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:29:51.276923 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dc4ff709-fd03-44a0-9610-2549523ca75d-istiod-ca-cert\") pod \"e2e-derive-e49e6eb7-openshift-default-5b4cc56b7-nfcp5\" (UID: \"dc4ff709-fd03-44a0-9610-2549523ca75d\") " pod="openshift-ingress/e2e-derive-e49e6eb7-openshift-default-5b4cc56b7-nfcp5" Jul 21 21:29:51.277346 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:29:51.276958 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dc4ff709-fd03-44a0-9610-2549523ca75d-istio-envoy\") pod \"e2e-derive-e49e6eb7-openshift-default-5b4cc56b7-nfcp5\" (UID: \"dc4ff709-fd03-44a0-9610-2549523ca75d\") " pod="openshift-ingress/e2e-derive-e49e6eb7-openshift-default-5b4cc56b7-nfcp5" Jul 21 21:29:51.277346 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:29:51.276986 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc4ff709-fd03-44a0-9610-2549523ca75d-credential-socket\") pod \"e2e-derive-e49e6eb7-openshift-default-5b4cc56b7-nfcp5\" (UID: \"dc4ff709-fd03-44a0-9610-2549523ca75d\") " pod="openshift-ingress/e2e-derive-e49e6eb7-openshift-default-5b4cc56b7-nfcp5" Jul 21 21:29:51.277346 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:29:51.277039 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dc4ff709-fd03-44a0-9610-2549523ca75d-workload-certs\") pod \"e2e-derive-e49e6eb7-openshift-default-5b4cc56b7-nfcp5\" (UID: \"dc4ff709-fd03-44a0-9610-2549523ca75d\") " pod="openshift-ingress/e2e-derive-e49e6eb7-openshift-default-5b4cc56b7-nfcp5" Jul 21 21:29:51.277346 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:29:51.277065 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dc4ff709-fd03-44a0-9610-2549523ca75d-istio-data\") pod \"e2e-derive-e49e6eb7-openshift-default-5b4cc56b7-nfcp5\" (UID: \"dc4ff709-fd03-44a0-9610-2549523ca75d\") " pod="openshift-ingress/e2e-derive-e49e6eb7-openshift-default-5b4cc56b7-nfcp5" Jul 21 21:29:51.277346 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:29:51.277103 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dc4ff709-fd03-44a0-9610-2549523ca75d-istio-podinfo\") pod \"e2e-derive-e49e6eb7-openshift-default-5b4cc56b7-nfcp5\" (UID: \"dc4ff709-fd03-44a0-9610-2549523ca75d\") " pod="openshift-ingress/e2e-derive-e49e6eb7-openshift-default-5b4cc56b7-nfcp5" Jul 21 21:29:51.277346 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:29:51.277156 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc4ff709-fd03-44a0-9610-2549523ca75d-workload-socket\") pod \"e2e-derive-e49e6eb7-openshift-default-5b4cc56b7-nfcp5\" (UID: \"dc4ff709-fd03-44a0-9610-2549523ca75d\") " pod="openshift-ingress/e2e-derive-e49e6eb7-openshift-default-5b4cc56b7-nfcp5" Jul 21 21:29:51.277730 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:29:51.277557 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc4ff709-fd03-44a0-9610-2549523ca75d-credential-socket\") pod \"e2e-derive-e49e6eb7-openshift-default-5b4cc56b7-nfcp5\" (UID: \"dc4ff709-fd03-44a0-9610-2549523ca75d\") " pod="openshift-ingress/e2e-derive-e49e6eb7-openshift-default-5b4cc56b7-nfcp5" Jul 21 21:29:51.277730 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:29:51.277692 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dc4ff709-fd03-44a0-9610-2549523ca75d-workload-certs\") pod \"e2e-derive-e49e6eb7-openshift-default-5b4cc56b7-nfcp5\" (UID: \"dc4ff709-fd03-44a0-9610-2549523ca75d\") " pod="openshift-ingress/e2e-derive-e49e6eb7-openshift-default-5b4cc56b7-nfcp5" Jul 21 21:29:51.277817 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:29:51.277723 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dc4ff709-fd03-44a0-9610-2549523ca75d-istio-data\") pod \"e2e-derive-e49e6eb7-openshift-default-5b4cc56b7-nfcp5\" (UID: \"dc4ff709-fd03-44a0-9610-2549523ca75d\") " pod="openshift-ingress/e2e-derive-e49e6eb7-openshift-default-5b4cc56b7-nfcp5" Jul 21 21:29:51.277817 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:29:51.277769 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc4ff709-fd03-44a0-9610-2549523ca75d-workload-socket\") pod \"e2e-derive-e49e6eb7-openshift-default-5b4cc56b7-nfcp5\" (UID: \"dc4ff709-fd03-44a0-9610-2549523ca75d\") " pod="openshift-ingress/e2e-derive-e49e6eb7-openshift-default-5b4cc56b7-nfcp5" Jul 21 21:29:51.277914 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:29:51.277896 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dc4ff709-fd03-44a0-9610-2549523ca75d-istiod-ca-cert\") pod \"e2e-derive-e49e6eb7-openshift-default-5b4cc56b7-nfcp5\" (UID: \"dc4ff709-fd03-44a0-9610-2549523ca75d\") " pod="openshift-ingress/e2e-derive-e49e6eb7-openshift-default-5b4cc56b7-nfcp5" Jul 21 21:29:51.279476 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:29:51.279457 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dc4ff709-fd03-44a0-9610-2549523ca75d-istio-envoy\") pod \"e2e-derive-e49e6eb7-openshift-default-5b4cc56b7-nfcp5\" (UID: \"dc4ff709-fd03-44a0-9610-2549523ca75d\") " pod="openshift-ingress/e2e-derive-e49e6eb7-openshift-default-5b4cc56b7-nfcp5" Jul 21 21:29:51.279745 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:29:51.279727 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dc4ff709-fd03-44a0-9610-2549523ca75d-istio-podinfo\") pod \"e2e-derive-e49e6eb7-openshift-default-5b4cc56b7-nfcp5\" (UID: \"dc4ff709-fd03-44a0-9610-2549523ca75d\") " pod="openshift-ingress/e2e-derive-e49e6eb7-openshift-default-5b4cc56b7-nfcp5" Jul 21 21:29:51.284818 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:29:51.284793 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-49qtm\" (UniqueName: \"kubernetes.io/projected/dc4ff709-fd03-44a0-9610-2549523ca75d-kube-api-access-49qtm\") pod \"e2e-derive-e49e6eb7-openshift-default-5b4cc56b7-nfcp5\" (UID: \"dc4ff709-fd03-44a0-9610-2549523ca75d\") " pod="openshift-ingress/e2e-derive-e49e6eb7-openshift-default-5b4cc56b7-nfcp5" Jul 21 21:29:51.284992 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:29:51.284972 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dc4ff709-fd03-44a0-9610-2549523ca75d-istio-token\") pod \"e2e-derive-e49e6eb7-openshift-default-5b4cc56b7-nfcp5\" (UID: \"dc4ff709-fd03-44a0-9610-2549523ca75d\") " pod="openshift-ingress/e2e-derive-e49e6eb7-openshift-default-5b4cc56b7-nfcp5" Jul 21 21:29:51.391850 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:29:51.391818 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-e49e6eb7-openshift-default-5b4cc56b7-nfcp5" Jul 21 21:29:51.528473 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:29:51.528446 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-e49e6eb7-openshift-default-5b4cc56b7-nfcp5"] Jul 21 21:29:51.530598 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:29:51.530569 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddc4ff709_fd03_44a0_9610_2549523ca75d.slice/crio-2db1254f56c2c16c33677adf31588d17dc11e815dc85b91c055aa6e35ba5c636 WatchSource:0}: Error finding container 2db1254f56c2c16c33677adf31588d17dc11e815dc85b91c055aa6e35ba5c636: Status 404 returned error can't find the container with id 2db1254f56c2c16c33677adf31588d17dc11e815dc85b91c055aa6e35ba5c636 Jul 21 21:29:51.532931 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:29:51.532886 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 21 21:29:51.533024 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:29:51.532986 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 21 21:29:51.533065 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:29:51.533029 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 21 21:29:51.859709 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:29:51.859673 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-e49e6eb7-openshift-default-5b4cc56b7-nfcp5" event={"ID":"dc4ff709-fd03-44a0-9610-2549523ca75d","Type":"ContainerStarted","Data":"8dc591bbcfd944abc5201db9e5c5d8e2daa2b544e879eb4060b2e78fe3a38ef0"} Jul 21 21:29:51.859904 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:29:51.859733 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-e49e6eb7-openshift-default-5b4cc56b7-nfcp5" event={"ID":"dc4ff709-fd03-44a0-9610-2549523ca75d","Type":"ContainerStarted","Data":"2db1254f56c2c16c33677adf31588d17dc11e815dc85b91c055aa6e35ba5c636"} Jul 21 21:29:51.878382 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:29:51.878336 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-derive-e49e6eb7-openshift-default-5b4cc56b7-nfcp5" podStartSLOduration=0.878296491 podStartE2EDuration="878.296491ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 21:29:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 21:29:51.877671063 +0000 UTC m=+1749.812420729" watchObservedRunningTime="2026-07-21 21:29:51.878296491 +0000 UTC m=+1749.813046159" Jul 21 21:29:52.392913 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:29:52.392881 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-e49e6eb7-openshift-default-5b4cc56b7-nfcp5" Jul 21 21:29:52.397555 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:29:52.397511 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-e49e6eb7-openshift-default-5b4cc56b7-nfcp5" Jul 21 21:29:52.864508 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:29:52.864484 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-e49e6eb7-openshift-default-5b4cc56b7-nfcp5" Jul 21 21:29:52.865307 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:29:52.865288 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-e49e6eb7-openshift-default-5b4cc56b7-nfcp5" Jul 21 21:29:55.499591 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:29:55.499562 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-a4a82f5f-openshift-default-7f7f96dc7c-5wbrz" Jul 21 21:29:55.619489 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:29:55.619464 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/443386e3-0ac0-4ac0-a04e-5571c0a6c915-credential-socket\") pod \"443386e3-0ac0-4ac0-a04e-5571c0a6c915\" (UID: \"443386e3-0ac0-4ac0-a04e-5571c0a6c915\") " Jul 21 21:29:55.619663 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:29:55.619503 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/443386e3-0ac0-4ac0-a04e-5571c0a6c915-workload-certs\") pod \"443386e3-0ac0-4ac0-a04e-5571c0a6c915\" (UID: \"443386e3-0ac0-4ac0-a04e-5571c0a6c915\") " Jul 21 21:29:55.619663 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:29:55.619562 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/443386e3-0ac0-4ac0-a04e-5571c0a6c915-istiod-ca-cert\") pod \"443386e3-0ac0-4ac0-a04e-5571c0a6c915\" (UID: \"443386e3-0ac0-4ac0-a04e-5571c0a6c915\") " Jul 21 21:29:55.619663 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:29:55.619581 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/443386e3-0ac0-4ac0-a04e-5571c0a6c915-istio-token\") pod \"443386e3-0ac0-4ac0-a04e-5571c0a6c915\" (UID: \"443386e3-0ac0-4ac0-a04e-5571c0a6c915\") " Jul 21 21:29:55.619663 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:29:55.619613 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/443386e3-0ac0-4ac0-a04e-5571c0a6c915-istio-podinfo\") pod \"443386e3-0ac0-4ac0-a04e-5571c0a6c915\" (UID: \"443386e3-0ac0-4ac0-a04e-5571c0a6c915\") " Jul 21 21:29:55.619663 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:29:55.619648 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/443386e3-0ac0-4ac0-a04e-5571c0a6c915-istio-envoy\") pod \"443386e3-0ac0-4ac0-a04e-5571c0a6c915\" (UID: \"443386e3-0ac0-4ac0-a04e-5571c0a6c915\") " Jul 21 21:29:55.619909 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:29:55.619693 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kmzkd\" (UniqueName: \"kubernetes.io/projected/443386e3-0ac0-4ac0-a04e-5571c0a6c915-kube-api-access-kmzkd\") pod \"443386e3-0ac0-4ac0-a04e-5571c0a6c915\" (UID: \"443386e3-0ac0-4ac0-a04e-5571c0a6c915\") " Jul 21 21:29:55.619909 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:29:55.619724 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/443386e3-0ac0-4ac0-a04e-5571c0a6c915-istio-data\") pod \"443386e3-0ac0-4ac0-a04e-5571c0a6c915\" (UID: \"443386e3-0ac0-4ac0-a04e-5571c0a6c915\") " Jul 21 21:29:55.619909 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:29:55.619772 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/443386e3-0ac0-4ac0-a04e-5571c0a6c915-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "443386e3-0ac0-4ac0-a04e-5571c0a6c915" (UID: "443386e3-0ac0-4ac0-a04e-5571c0a6c915"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 21:29:55.619909 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:29:55.619826 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/443386e3-0ac0-4ac0-a04e-5571c0a6c915-workload-socket\") pod \"443386e3-0ac0-4ac0-a04e-5571c0a6c915\" (UID: \"443386e3-0ac0-4ac0-a04e-5571c0a6c915\") " Jul 21 21:29:55.619909 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:29:55.619840 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/443386e3-0ac0-4ac0-a04e-5571c0a6c915-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "443386e3-0ac0-4ac0-a04e-5571c0a6c915" (UID: "443386e3-0ac0-4ac0-a04e-5571c0a6c915"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 21:29:55.620163 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:29:55.620053 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/443386e3-0ac0-4ac0-a04e-5571c0a6c915-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "443386e3-0ac0-4ac0-a04e-5571c0a6c915" (UID: "443386e3-0ac0-4ac0-a04e-5571c0a6c915"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 21 21:29:55.620163 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:29:55.620065 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/443386e3-0ac0-4ac0-a04e-5571c0a6c915-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "443386e3-0ac0-4ac0-a04e-5571c0a6c915" (UID: "443386e3-0ac0-4ac0-a04e-5571c0a6c915"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 21:29:55.620163 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:29:55.620148 2575 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/443386e3-0ac0-4ac0-a04e-5571c0a6c915-workload-certs\") on node \"ip-10-0-140-97.ec2.internal\" DevicePath \"\"" Jul 21 21:29:55.620289 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:29:55.620164 2575 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/443386e3-0ac0-4ac0-a04e-5571c0a6c915-istiod-ca-cert\") on node \"ip-10-0-140-97.ec2.internal\" DevicePath \"\"" Jul 21 21:29:55.620289 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:29:55.620177 2575 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/443386e3-0ac0-4ac0-a04e-5571c0a6c915-workload-socket\") on node \"ip-10-0-140-97.ec2.internal\" DevicePath \"\"" Jul 21 21:29:55.620289 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:29:55.620189 2575 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/443386e3-0ac0-4ac0-a04e-5571c0a6c915-credential-socket\") on node \"ip-10-0-140-97.ec2.internal\" DevicePath \"\"" Jul 21 21:29:55.620628 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:29:55.620605 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/443386e3-0ac0-4ac0-a04e-5571c0a6c915-istio-data" (OuterVolumeSpecName: "istio-data") pod "443386e3-0ac0-4ac0-a04e-5571c0a6c915" (UID: "443386e3-0ac0-4ac0-a04e-5571c0a6c915"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 21:29:55.621955 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:29:55.621932 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/443386e3-0ac0-4ac0-a04e-5571c0a6c915-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "443386e3-0ac0-4ac0-a04e-5571c0a6c915" (UID: "443386e3-0ac0-4ac0-a04e-5571c0a6c915"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 21:29:55.622045 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:29:55.621956 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/443386e3-0ac0-4ac0-a04e-5571c0a6c915-kube-api-access-kmzkd" (OuterVolumeSpecName: "kube-api-access-kmzkd") pod "443386e3-0ac0-4ac0-a04e-5571c0a6c915" (UID: "443386e3-0ac0-4ac0-a04e-5571c0a6c915"). InnerVolumeSpecName "kube-api-access-kmzkd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 21 21:29:55.622226 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:29:55.622212 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/443386e3-0ac0-4ac0-a04e-5571c0a6c915-istio-token" (OuterVolumeSpecName: "istio-token") pod "443386e3-0ac0-4ac0-a04e-5571c0a6c915" (UID: "443386e3-0ac0-4ac0-a04e-5571c0a6c915"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 21 21:29:55.622269 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:29:55.622233 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/443386e3-0ac0-4ac0-a04e-5571c0a6c915-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "443386e3-0ac0-4ac0-a04e-5571c0a6c915" (UID: "443386e3-0ac0-4ac0-a04e-5571c0a6c915"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 21 21:29:55.721568 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:29:55.721522 2575 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/443386e3-0ac0-4ac0-a04e-5571c0a6c915-istio-token\") on node \"ip-10-0-140-97.ec2.internal\" DevicePath \"\"" Jul 21 21:29:55.721568 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:29:55.721563 2575 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/443386e3-0ac0-4ac0-a04e-5571c0a6c915-istio-podinfo\") on node \"ip-10-0-140-97.ec2.internal\" DevicePath \"\"" Jul 21 21:29:55.721568 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:29:55.721573 2575 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/443386e3-0ac0-4ac0-a04e-5571c0a6c915-istio-envoy\") on node \"ip-10-0-140-97.ec2.internal\" DevicePath \"\"" Jul 21 21:29:55.721781 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:29:55.721583 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kmzkd\" (UniqueName: \"kubernetes.io/projected/443386e3-0ac0-4ac0-a04e-5571c0a6c915-kube-api-access-kmzkd\") on node \"ip-10-0-140-97.ec2.internal\" DevicePath \"\"" Jul 21 21:29:55.721781 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:29:55.721592 2575 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/443386e3-0ac0-4ac0-a04e-5571c0a6c915-istio-data\") on node \"ip-10-0-140-97.ec2.internal\" DevicePath \"\"" Jul 21 21:29:55.880054 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:29:55.879971 2575 generic.go:356] "Generic (PLEG): container finished" podID="443386e3-0ac0-4ac0-a04e-5571c0a6c915" containerID="a1541f9dc2883837ec181b9b84f48fc96da7a020831442b5ace00dc799214a38" exitCode=0 Jul 21 21:29:55.880054 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:29:55.880020 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-a4a82f5f-openshift-default-7f7f96dc7c-5wbrz" Jul 21 21:29:55.880265 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:29:55.880052 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-a4a82f5f-openshift-default-7f7f96dc7c-5wbrz" event={"ID":"443386e3-0ac0-4ac0-a04e-5571c0a6c915","Type":"ContainerDied","Data":"a1541f9dc2883837ec181b9b84f48fc96da7a020831442b5ace00dc799214a38"} Jul 21 21:29:55.880265 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:29:55.880101 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-a4a82f5f-openshift-default-7f7f96dc7c-5wbrz" event={"ID":"443386e3-0ac0-4ac0-a04e-5571c0a6c915","Type":"ContainerDied","Data":"cad545c25971bb9280a39dcb10d197f685d110fde84d7d10ad5cc5e697d0a50e"} Jul 21 21:29:55.880265 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:29:55.880125 2575 scope.go:117] "RemoveContainer" containerID="a1541f9dc2883837ec181b9b84f48fc96da7a020831442b5ace00dc799214a38" Jul 21 21:29:55.889485 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:29:55.889469 2575 scope.go:117] "RemoveContainer" containerID="665bfb4c6b783c31377867ac2fa332513387a570d353ab51c630fa8d61fd112a" Jul 21 21:29:55.897058 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:29:55.897040 2575 scope.go:117] "RemoveContainer" containerID="a1541f9dc2883837ec181b9b84f48fc96da7a020831442b5ace00dc799214a38" Jul 21 21:29:55.897310 ip-10-0-140-97 kubenswrapper[2575]: E0721 21:29:55.897294 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a1541f9dc2883837ec181b9b84f48fc96da7a020831442b5ace00dc799214a38\": container with ID starting with a1541f9dc2883837ec181b9b84f48fc96da7a020831442b5ace00dc799214a38 not found: ID does not exist" containerID="a1541f9dc2883837ec181b9b84f48fc96da7a020831442b5ace00dc799214a38" Jul 21 21:29:55.897355 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:29:55.897319 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a1541f9dc2883837ec181b9b84f48fc96da7a020831442b5ace00dc799214a38"} err="failed to get container status \"a1541f9dc2883837ec181b9b84f48fc96da7a020831442b5ace00dc799214a38\": rpc error: code = NotFound desc = could not find container \"a1541f9dc2883837ec181b9b84f48fc96da7a020831442b5ace00dc799214a38\": container with ID starting with a1541f9dc2883837ec181b9b84f48fc96da7a020831442b5ace00dc799214a38 not found: ID does not exist" Jul 21 21:29:55.897355 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:29:55.897335 2575 scope.go:117] "RemoveContainer" containerID="665bfb4c6b783c31377867ac2fa332513387a570d353ab51c630fa8d61fd112a" Jul 21 21:29:55.897683 ip-10-0-140-97 kubenswrapper[2575]: E0721 21:29:55.897665 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"665bfb4c6b783c31377867ac2fa332513387a570d353ab51c630fa8d61fd112a\": container with ID starting with 665bfb4c6b783c31377867ac2fa332513387a570d353ab51c630fa8d61fd112a not found: ID does not exist" containerID="665bfb4c6b783c31377867ac2fa332513387a570d353ab51c630fa8d61fd112a" Jul 21 21:29:55.897732 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:29:55.897689 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"665bfb4c6b783c31377867ac2fa332513387a570d353ab51c630fa8d61fd112a"} err="failed to get container status \"665bfb4c6b783c31377867ac2fa332513387a570d353ab51c630fa8d61fd112a\": rpc error: code = NotFound desc = could not find container \"665bfb4c6b783c31377867ac2fa332513387a570d353ab51c630fa8d61fd112a\": container with ID starting with 665bfb4c6b783c31377867ac2fa332513387a570d353ab51c630fa8d61fd112a not found: ID does not exist" Jul 21 21:29:55.902674 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:29:55.902623 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-a4a82f5f-openshift-default-7f7f96dc7c-5wbrz"] Jul 21 21:29:55.907771 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:29:55.907745 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-a4a82f5f-openshift-default-7f7f96dc7c-5wbrz"] Jul 21 21:29:56.741939 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:29:56.741903 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="443386e3-0ac0-4ac0-a04e-5571c0a6c915" path="/var/lib/kubelet/pods/443386e3-0ac0-4ac0-a04e-5571c0a6c915/volumes" Jul 21 21:30:00.134565 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:30:00.134506 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-e49e6eb7-29744490-q8mfk"] Jul 21 21:30:00.134950 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:30:00.134912 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="443386e3-0ac0-4ac0-a04e-5571c0a6c915" containerName="istio-proxy" Jul 21 21:30:00.134950 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:30:00.134924 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="443386e3-0ac0-4ac0-a04e-5571c0a6c915" containerName="istio-proxy" Jul 21 21:30:00.134950 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:30:00.134931 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="443386e3-0ac0-4ac0-a04e-5571c0a6c915" containerName="istio-proxy" Jul 21 21:30:00.134950 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:30:00.134937 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="443386e3-0ac0-4ac0-a04e-5571c0a6c915" containerName="istio-proxy" Jul 21 21:30:00.135097 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:30:00.135000 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="443386e3-0ac0-4ac0-a04e-5571c0a6c915" containerName="istio-proxy" Jul 21 21:30:00.139580 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:30:00.139555 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29744490-jrddn"] Jul 21 21:30:00.140409 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:30:00.140389 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="443386e3-0ac0-4ac0-a04e-5571c0a6c915" containerName="istio-proxy" Jul 21 21:30:00.140655 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:30:00.139811 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-e49e6eb7-29744490-q8mfk" Jul 21 21:30:00.143814 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:30:00.143792 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29744490-jrddn" Jul 21 21:30:00.145845 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:30:00.145824 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9fxhr\"" Jul 21 21:30:00.147921 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:30:00.147903 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29744490-jrddn"] Jul 21 21:30:00.150337 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:30:00.150304 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-e49e6eb7-29744490-q8mfk"] Jul 21 21:30:00.155936 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:30:00.155909 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-e49e6eb7-29744490-q8mfk" Jul 21 21:30:00.194223 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:30:00.194196 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29744490-jrddn" Jul 21 21:30:00.398145 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:30:00.398121 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29744490-jrddn"] Jul 21 21:30:00.400540 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:30:00.400495 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41509374_45f4_438c_946a_3518b35ffae7.slice/crio-0eed5c6a91a1b3fc43ed6953d7bd3645c4e4fb0981de40c805cd851ed5577e1e WatchSource:0}: Error finding container 0eed5c6a91a1b3fc43ed6953d7bd3645c4e4fb0981de40c805cd851ed5577e1e: Status 404 returned error can't find the container with id 0eed5c6a91a1b3fc43ed6953d7bd3645c4e4fb0981de40c805cd851ed5577e1e Jul 21 21:30:00.551810 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:30:00.551096 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-e49e6eb7-29744490-q8mfk"] Jul 21 21:30:00.909504 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:30:00.909412 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29744490-jrddn" event={"ID":"41509374-45f4-438c-946a-3518b35ffae7","Type":"ContainerStarted","Data":"56d25a287e58ce2ac475a2a8230b653f8050942bcb23723c2f0ed07265b59087"} Jul 21 21:30:00.909504 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:30:00.909482 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29744490-jrddn" event={"ID":"41509374-45f4-438c-946a-3518b35ffae7","Type":"ContainerStarted","Data":"0eed5c6a91a1b3fc43ed6953d7bd3645c4e4fb0981de40c805cd851ed5577e1e"} Jul 21 21:30:00.910878 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:30:00.910850 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-e49e6eb7-29744490-q8mfk" event={"ID":"e8c68d0e-6856-4895-a96f-820d2daf1e53","Type":"ContainerStarted","Data":"6bc54ad0bdece3a43b66b50e680367135cbe60a3aeb4cafc243e5ca36a67f5e8"} Jul 21 21:30:00.926457 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:30:00.926413 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29744490-jrddn" podStartSLOduration=0.926400195 podStartE2EDuration="926.400195ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 21:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 21:30:00.925209178 +0000 UTC m=+1758.859958862" watchObservedRunningTime="2026-07-21 21:30:00.926400195 +0000 UTC m=+1758.861149861" Jul 21 21:30:01.918760 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:30:01.918656 2575 generic.go:356] "Generic (PLEG): container finished" podID="41509374-45f4-438c-946a-3518b35ffae7" containerID="56d25a287e58ce2ac475a2a8230b653f8050942bcb23723c2f0ed07265b59087" exitCode=0 Jul 21 21:30:01.918760 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:30:01.918738 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29744490-jrddn" event={"ID":"41509374-45f4-438c-946a-3518b35ffae7","Type":"ContainerDied","Data":"56d25a287e58ce2ac475a2a8230b653f8050942bcb23723c2f0ed07265b59087"} Jul 21 21:30:01.920309 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:30:01.920284 2575 generic.go:356] "Generic (PLEG): container finished" podID="e8c68d0e-6856-4895-a96f-820d2daf1e53" containerID="b204082bc17cff64ca5a7e171ee199f2da07ab8be419606a703cd4fc3ba2f39d" exitCode=7 Jul 21 21:30:01.920449 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:30:01.920317 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-e49e6eb7-29744490-q8mfk" event={"ID":"e8c68d0e-6856-4895-a96f-820d2daf1e53","Type":"ContainerDied","Data":"b204082bc17cff64ca5a7e171ee199f2da07ab8be419606a703cd4fc3ba2f39d"} Jul 21 21:30:01.920546 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:30:01.920509 2575 scope.go:117] "RemoveContainer" containerID="b204082bc17cff64ca5a7e171ee199f2da07ab8be419606a703cd4fc3ba2f39d" Jul 21 21:30:02.927178 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:30:02.927146 2575 generic.go:356] "Generic (PLEG): container finished" podID="e8c68d0e-6856-4895-a96f-820d2daf1e53" containerID="e605ece58077f3c68c569b3fc931f53b30c7fa8343c3561de05d35e640b9241f" exitCode=0 Jul 21 21:30:02.927577 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:30:02.927227 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-e49e6eb7-29744490-q8mfk" event={"ID":"e8c68d0e-6856-4895-a96f-820d2daf1e53","Type":"ContainerDied","Data":"e605ece58077f3c68c569b3fc931f53b30c7fa8343c3561de05d35e640b9241f"} Jul 21 21:30:02.927577 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:30:02.927287 2575 scope.go:117] "RemoveContainer" containerID="b204082bc17cff64ca5a7e171ee199f2da07ab8be419606a703cd4fc3ba2f39d" Jul 21 21:30:03.074484 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:30:03.074457 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29744490-jrddn" Jul 21 21:30:03.934192 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:30:03.934157 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29744490-jrddn" event={"ID":"41509374-45f4-438c-946a-3518b35ffae7","Type":"ContainerDied","Data":"0eed5c6a91a1b3fc43ed6953d7bd3645c4e4fb0981de40c805cd851ed5577e1e"} Jul 21 21:30:03.944844 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:30:03.934227 2575 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0eed5c6a91a1b3fc43ed6953d7bd3645c4e4fb0981de40c805cd851ed5577e1e" Jul 21 21:30:03.944844 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:30:03.934168 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29744490-jrddn" Jul 21 21:30:04.122868 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:30:04.122847 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-e49e6eb7-29744490-q8mfk" Jul 21 21:30:04.942961 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:30:04.942930 2575 generic.go:356] "Generic (PLEG): container finished" podID="dc4ff709-fd03-44a0-9610-2549523ca75d" containerID="8dc591bbcfd944abc5201db9e5c5d8e2daa2b544e879eb4060b2e78fe3a38ef0" exitCode=0 Jul 21 21:30:04.943332 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:30:04.943002 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-e49e6eb7-openshift-default-5b4cc56b7-nfcp5" event={"ID":"dc4ff709-fd03-44a0-9610-2549523ca75d","Type":"ContainerDied","Data":"8dc591bbcfd944abc5201db9e5c5d8e2daa2b544e879eb4060b2e78fe3a38ef0"} Jul 21 21:30:04.943441 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:30:04.943423 2575 scope.go:117] "RemoveContainer" containerID="8dc591bbcfd944abc5201db9e5c5d8e2daa2b544e879eb4060b2e78fe3a38ef0" Jul 21 21:30:04.944301 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:30:04.944278 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-e49e6eb7-29744490-q8mfk" event={"ID":"e8c68d0e-6856-4895-a96f-820d2daf1e53","Type":"ContainerDied","Data":"6bc54ad0bdece3a43b66b50e680367135cbe60a3aeb4cafc243e5ca36a67f5e8"} Jul 21 21:30:04.944398 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:30:04.944320 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-e49e6eb7-29744490-q8mfk" Jul 21 21:30:04.944398 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:30:04.944324 2575 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6bc54ad0bdece3a43b66b50e680367135cbe60a3aeb4cafc243e5ca36a67f5e8" Jul 21 21:30:04.944569 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:30:04.944520 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 21 21:30:04.944658 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:30:04.944627 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 21 21:30:04.944717 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:30:04.944676 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 21 21:30:05.954381 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:30:05.954331 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-e49e6eb7-openshift-default-5b4cc56b7-nfcp5" event={"ID":"dc4ff709-fd03-44a0-9610-2549523ca75d","Type":"ContainerStarted","Data":"c888426f23c24ee821264dfdbb3f9a45f7e3b982073b507b869e884cb18bc0e9"} Jul 21 21:30:06.392110 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:30:06.392068 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-e49e6eb7-openshift-default-5b4cc56b7-nfcp5" Jul 21 21:30:06.393296 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:30:06.393269 2575 patch_prober.go:28] interesting pod/e2e-derive-e49e6eb7-openshift-default-5b4cc56b7-nfcp5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" start-of-body= Jul 21 21:30:06.393416 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:30:06.393329 2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-e49e6eb7-openshift-default-5b4cc56b7-nfcp5" podUID="dc4ff709-fd03-44a0-9610-2549523ca75d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" Jul 21 21:30:07.393001 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:30:07.392958 2575 patch_prober.go:28] interesting pod/e2e-derive-e49e6eb7-openshift-default-5b4cc56b7-nfcp5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" start-of-body= Jul 21 21:30:07.393379 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:30:07.393017 2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-e49e6eb7-openshift-default-5b4cc56b7-nfcp5" podUID="dc4ff709-fd03-44a0-9610-2549523ca75d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" Jul 21 21:30:07.865301 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:30:07.865267 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-e49e6eb7-openshift-default-5b4cc56b7-nfcp5" Jul 21 21:30:08.392148 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:30:08.392110 2575 patch_prober.go:28] interesting pod/e2e-derive-e49e6eb7-openshift-default-5b4cc56b7-nfcp5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" start-of-body= Jul 21 21:30:08.392322 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:30:08.392166 2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-e49e6eb7-openshift-default-5b4cc56b7-nfcp5" podUID="dc4ff709-fd03-44a0-9610-2549523ca75d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" Jul 21 21:30:09.392198 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:30:09.392164 2575 patch_prober.go:28] interesting pod/e2e-derive-e49e6eb7-openshift-default-5b4cc56b7-nfcp5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" start-of-body= Jul 21 21:30:09.392598 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:30:09.392214 2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-e49e6eb7-openshift-default-5b4cc56b7-nfcp5" podUID="dc4ff709-fd03-44a0-9610-2549523ca75d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" Jul 21 21:30:10.396035 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:30:10.396003 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-e49e6eb7-openshift-default-5b4cc56b7-nfcp5" Jul 21 21:30:10.396938 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:30:10.396920 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-e49e6eb7-openshift-default-5b4cc56b7-nfcp5" Jul 21 21:30:18.179190 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:30:18.179156 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-e49e6eb7-rndxv"] Jul 21 21:30:18.179619 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:30:18.179499 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e8c68d0e-6856-4895-a96f-820d2daf1e53" containerName="cleanup" Jul 21 21:30:18.179619 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:30:18.179509 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="e8c68d0e-6856-4895-a96f-820d2daf1e53" containerName="cleanup" Jul 21 21:30:18.179619 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:30:18.179522 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="41509374-45f4-438c-946a-3518b35ffae7" containerName="cleanup" Jul 21 21:30:18.179619 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:30:18.179541 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="41509374-45f4-438c-946a-3518b35ffae7" containerName="cleanup" Jul 21 21:30:18.179619 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:30:18.179549 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e8c68d0e-6856-4895-a96f-820d2daf1e53" containerName="cleanup" Jul 21 21:30:18.179619 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:30:18.179555 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="e8c68d0e-6856-4895-a96f-820d2daf1e53" containerName="cleanup" Jul 21 21:30:18.179619 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:30:18.179609 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="e8c68d0e-6856-4895-a96f-820d2daf1e53" containerName="cleanup" Jul 21 21:30:18.179619 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:30:18.179618 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="41509374-45f4-438c-946a-3518b35ffae7" containerName="cleanup" Jul 21 21:30:18.179619 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:30:18.179625 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="e8c68d0e-6856-4895-a96f-820d2daf1e53" containerName="cleanup" Jul 21 21:30:18.182506 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:30:18.182491 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-e49e6eb7-rndxv" Jul 21 21:30:18.185264 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:30:18.185241 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9fxhr\"" Jul 21 21:30:18.185388 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:30:18.185245 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 21 21:30:18.190707 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:30:18.190683 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-e49e6eb7-rndxv"] Jul 21 21:30:18.233336 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:30:18.233308 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/52275b9b-d65c-423b-9704-ed570b7ad8c4-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-e49e6eb7-rndxv\" (UID: \"52275b9b-d65c-423b-9704-ed570b7ad8c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-e49e6eb7-rndxv" Jul 21 21:30:18.334157 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:30:18.334124 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/52275b9b-d65c-423b-9704-ed570b7ad8c4-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-e49e6eb7-rndxv\" (UID: \"52275b9b-d65c-423b-9704-ed570b7ad8c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-e49e6eb7-rndxv" Jul 21 21:30:18.334749 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:30:18.334731 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/52275b9b-d65c-423b-9704-ed570b7ad8c4-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-e49e6eb7-rndxv\" (UID: \"52275b9b-d65c-423b-9704-ed570b7ad8c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-e49e6eb7-rndxv" Jul 21 21:30:18.493756 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:30:18.493678 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-e49e6eb7-rndxv" Jul 21 21:30:18.620181 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:30:18.620155 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-e49e6eb7-rndxv"] Jul 21 21:30:18.622328 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:30:18.622300 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod52275b9b_d65c_423b_9704_ed570b7ad8c4.slice/crio-3ce94675799e0000268a82cc2965cd259d13fc5fb990f8630bf756205f9fb7bd WatchSource:0}: Error finding container 3ce94675799e0000268a82cc2965cd259d13fc5fb990f8630bf756205f9fb7bd: Status 404 returned error can't find the container with id 3ce94675799e0000268a82cc2965cd259d13fc5fb990f8630bf756205f9fb7bd Jul 21 21:30:19.022277 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:30:19.022183 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-e49e6eb7-rndxv" event={"ID":"52275b9b-d65c-423b-9704-ed570b7ad8c4","Type":"ContainerStarted","Data":"f13f97833a2e9d27dc985ccb0e959bd46a1d77af9ea7f51038c7dc10aff73b28"} Jul 21 21:30:19.022277 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:30:19.022247 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-e49e6eb7-rndxv" event={"ID":"52275b9b-d65c-423b-9704-ed570b7ad8c4","Type":"ContainerStarted","Data":"3ce94675799e0000268a82cc2965cd259d13fc5fb990f8630bf756205f9fb7bd"} Jul 21 21:30:19.039653 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:30:19.039601 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-e49e6eb7-rndxv" podStartSLOduration=1.039587919 podStartE2EDuration="1.039587919s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 21:30:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 21:30:19.03827267 +0000 UTC m=+1776.973022345" watchObservedRunningTime="2026-07-21 21:30:19.039587919 +0000 UTC m=+1776.974337583" Jul 21 21:30:20.028972 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:30:20.028929 2575 generic.go:356] "Generic (PLEG): container finished" podID="52275b9b-d65c-423b-9704-ed570b7ad8c4" containerID="f13f97833a2e9d27dc985ccb0e959bd46a1d77af9ea7f51038c7dc10aff73b28" exitCode=0 Jul 21 21:30:20.029336 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:30:20.029030 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-e49e6eb7-rndxv" event={"ID":"52275b9b-d65c-423b-9704-ed570b7ad8c4","Type":"ContainerDied","Data":"f13f97833a2e9d27dc985ccb0e959bd46a1d77af9ea7f51038c7dc10aff73b28"} Jul 21 21:30:21.177561 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:30:21.177523 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-e49e6eb7-rndxv" Jul 21 21:30:21.262712 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:30:21.262669 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/52275b9b-d65c-423b-9704-ed570b7ad8c4-maas-api-service-ca\") pod \"52275b9b-d65c-423b-9704-ed570b7ad8c4\" (UID: \"52275b9b-d65c-423b-9704-ed570b7ad8c4\") " Jul 21 21:30:21.263056 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:30:21.263037 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/52275b9b-d65c-423b-9704-ed570b7ad8c4-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "52275b9b-d65c-423b-9704-ed570b7ad8c4" (UID: "52275b9b-d65c-423b-9704-ed570b7ad8c4"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 21 21:30:21.369576 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:30:21.364147 2575 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/52275b9b-d65c-423b-9704-ed570b7ad8c4-maas-api-service-ca\") on node \"ip-10-0-140-97.ec2.internal\" DevicePath \"\"" Jul 21 21:30:22.040632 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:30:22.040593 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-e49e6eb7-rndxv" Jul 21 21:30:22.040817 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:30:22.040605 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-e49e6eb7-rndxv" event={"ID":"52275b9b-d65c-423b-9704-ed570b7ad8c4","Type":"ContainerDied","Data":"3ce94675799e0000268a82cc2965cd259d13fc5fb990f8630bf756205f9fb7bd"} Jul 21 21:30:22.040817 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:30:22.040728 2575 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3ce94675799e0000268a82cc2965cd259d13fc5fb990f8630bf756205f9fb7bd" Jul 21 21:30:29.126130 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:30:29.126098 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-e49e6eb7-29744490-q8mfk"] Jul 21 21:30:29.129031 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:30:29.128988 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-e49e6eb7-29744490-q8mfk"] Jul 21 21:30:30.740503 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:30:30.740467 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e8c68d0e-6856-4895-a96f-820d2daf1e53" path="/var/lib/kubelet/pods/e8c68d0e-6856-4895-a96f-820d2daf1e53/volumes" Jul 21 21:30:33.269144 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:30:33.269102 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-e49e6eb7-rndxv"] Jul 21 21:30:33.270969 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:30:33.270947 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-e49e6eb7-rndxv"] Jul 21 21:30:33.419893 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:30:33.419862 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-e49e6eb7-openshift-default-5b4cc56b7-nfcp5"] Jul 21 21:30:33.420114 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:30:33.420069 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-derive-e49e6eb7-openshift-default-5b4cc56b7-nfcp5" podUID="dc4ff709-fd03-44a0-9610-2549523ca75d" containerName="istio-proxy" containerID="cri-o://c888426f23c24ee821264dfdbb3f9a45f7e3b982073b507b869e884cb18bc0e9" gracePeriod=30 Jul 21 21:30:34.741949 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:30:34.741875 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="52275b9b-d65c-423b-9704-ed570b7ad8c4" path="/var/lib/kubelet/pods/52275b9b-d65c-423b-9704-ed570b7ad8c4/volumes" Jul 21 21:30:38.672140 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:30:38.672117 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-e49e6eb7-openshift-default-5b4cc56b7-nfcp5" Jul 21 21:30:38.820192 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:30:38.820116 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc4ff709-fd03-44a0-9610-2549523ca75d-credential-socket\") pod \"dc4ff709-fd03-44a0-9610-2549523ca75d\" (UID: \"dc4ff709-fd03-44a0-9610-2549523ca75d\") " Jul 21 21:30:38.820192 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:30:38.820172 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dc4ff709-fd03-44a0-9610-2549523ca75d-istio-podinfo\") pod \"dc4ff709-fd03-44a0-9610-2549523ca75d\" (UID: \"dc4ff709-fd03-44a0-9610-2549523ca75d\") " Jul 21 21:30:38.820399 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:30:38.820207 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc4ff709-fd03-44a0-9610-2549523ca75d-workload-socket\") pod \"dc4ff709-fd03-44a0-9610-2549523ca75d\" (UID: \"dc4ff709-fd03-44a0-9610-2549523ca75d\") " Jul 21 21:30:38.820399 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:30:38.820233 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dc4ff709-fd03-44a0-9610-2549523ca75d-istiod-ca-cert\") pod \"dc4ff709-fd03-44a0-9610-2549523ca75d\" (UID: \"dc4ff709-fd03-44a0-9610-2549523ca75d\") " Jul 21 21:30:38.820399 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:30:38.820268 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dc4ff709-fd03-44a0-9610-2549523ca75d-istio-envoy\") pod \"dc4ff709-fd03-44a0-9610-2549523ca75d\" (UID: \"dc4ff709-fd03-44a0-9610-2549523ca75d\") " Jul 21 21:30:38.820399 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:30:38.820312 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dc4ff709-fd03-44a0-9610-2549523ca75d-workload-certs\") pod \"dc4ff709-fd03-44a0-9610-2549523ca75d\" (UID: \"dc4ff709-fd03-44a0-9610-2549523ca75d\") " Jul 21 21:30:38.820399 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:30:38.820326 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dc4ff709-fd03-44a0-9610-2549523ca75d-istio-data\") pod \"dc4ff709-fd03-44a0-9610-2549523ca75d\" (UID: \"dc4ff709-fd03-44a0-9610-2549523ca75d\") " Jul 21 21:30:38.820399 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:30:38.820345 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-49qtm\" (UniqueName: \"kubernetes.io/projected/dc4ff709-fd03-44a0-9610-2549523ca75d-kube-api-access-49qtm\") pod \"dc4ff709-fd03-44a0-9610-2549523ca75d\" (UID: \"dc4ff709-fd03-44a0-9610-2549523ca75d\") " Jul 21 21:30:38.820399 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:30:38.820361 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dc4ff709-fd03-44a0-9610-2549523ca75d-istio-token\") pod \"dc4ff709-fd03-44a0-9610-2549523ca75d\" (UID: \"dc4ff709-fd03-44a0-9610-2549523ca75d\") " Jul 21 21:30:38.820399 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:30:38.820367 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dc4ff709-fd03-44a0-9610-2549523ca75d-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "dc4ff709-fd03-44a0-9610-2549523ca75d" (UID: "dc4ff709-fd03-44a0-9610-2549523ca75d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 21:30:38.820825 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:30:38.820509 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dc4ff709-fd03-44a0-9610-2549523ca75d-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "dc4ff709-fd03-44a0-9610-2549523ca75d" (UID: "dc4ff709-fd03-44a0-9610-2549523ca75d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 21:30:38.820825 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:30:38.820659 2575 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc4ff709-fd03-44a0-9610-2549523ca75d-credential-socket\") on node \"ip-10-0-140-97.ec2.internal\" DevicePath \"\"" Jul 21 21:30:38.820825 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:30:38.820678 2575 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc4ff709-fd03-44a0-9610-2549523ca75d-workload-socket\") on node \"ip-10-0-140-97.ec2.internal\" DevicePath \"\"" Jul 21 21:30:38.820825 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:30:38.820704 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dc4ff709-fd03-44a0-9610-2549523ca75d-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "dc4ff709-fd03-44a0-9610-2549523ca75d" (UID: "dc4ff709-fd03-44a0-9610-2549523ca75d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 21 21:30:38.821074 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:30:38.820874 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dc4ff709-fd03-44a0-9610-2549523ca75d-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "dc4ff709-fd03-44a0-9610-2549523ca75d" (UID: "dc4ff709-fd03-44a0-9610-2549523ca75d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 21:30:38.821373 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:30:38.821350 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dc4ff709-fd03-44a0-9610-2549523ca75d-istio-data" (OuterVolumeSpecName: "istio-data") pod "dc4ff709-fd03-44a0-9610-2549523ca75d" (UID: "dc4ff709-fd03-44a0-9610-2549523ca75d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 21:30:38.822717 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:30:38.822692 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dc4ff709-fd03-44a0-9610-2549523ca75d-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "dc4ff709-fd03-44a0-9610-2549523ca75d" (UID: "dc4ff709-fd03-44a0-9610-2549523ca75d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 21:30:38.822813 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:30:38.822694 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dc4ff709-fd03-44a0-9610-2549523ca75d-kube-api-access-49qtm" (OuterVolumeSpecName: "kube-api-access-49qtm") pod "dc4ff709-fd03-44a0-9610-2549523ca75d" (UID: "dc4ff709-fd03-44a0-9610-2549523ca75d"). InnerVolumeSpecName "kube-api-access-49qtm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 21 21:30:38.822957 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:30:38.822930 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dc4ff709-fd03-44a0-9610-2549523ca75d-istio-token" (OuterVolumeSpecName: "istio-token") pod "dc4ff709-fd03-44a0-9610-2549523ca75d" (UID: "dc4ff709-fd03-44a0-9610-2549523ca75d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 21 21:30:38.822957 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:30:38.822928 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/dc4ff709-fd03-44a0-9610-2549523ca75d-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "dc4ff709-fd03-44a0-9610-2549523ca75d" (UID: "dc4ff709-fd03-44a0-9610-2549523ca75d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 21 21:30:38.921783 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:30:38.921757 2575 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dc4ff709-fd03-44a0-9610-2549523ca75d-istio-envoy\") on node \"ip-10-0-140-97.ec2.internal\" DevicePath \"\"" Jul 21 21:30:38.921783 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:30:38.921780 2575 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dc4ff709-fd03-44a0-9610-2549523ca75d-workload-certs\") on node \"ip-10-0-140-97.ec2.internal\" DevicePath \"\"" Jul 21 21:30:38.921783 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:30:38.921788 2575 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dc4ff709-fd03-44a0-9610-2549523ca75d-istio-data\") on node \"ip-10-0-140-97.ec2.internal\" DevicePath \"\"" Jul 21 21:30:38.921973 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:30:38.921797 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-49qtm\" (UniqueName: \"kubernetes.io/projected/dc4ff709-fd03-44a0-9610-2549523ca75d-kube-api-access-49qtm\") on node \"ip-10-0-140-97.ec2.internal\" DevicePath \"\"" Jul 21 21:30:38.921973 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:30:38.921806 2575 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dc4ff709-fd03-44a0-9610-2549523ca75d-istio-token\") on node \"ip-10-0-140-97.ec2.internal\" DevicePath \"\"" Jul 21 21:30:38.921973 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:30:38.921814 2575 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dc4ff709-fd03-44a0-9610-2549523ca75d-istio-podinfo\") on node \"ip-10-0-140-97.ec2.internal\" DevicePath \"\"" Jul 21 21:30:38.921973 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:30:38.921822 2575 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dc4ff709-fd03-44a0-9610-2549523ca75d-istiod-ca-cert\") on node \"ip-10-0-140-97.ec2.internal\" DevicePath \"\"" Jul 21 21:30:39.131046 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:30:39.131010 2575 generic.go:356] "Generic (PLEG): container finished" podID="dc4ff709-fd03-44a0-9610-2549523ca75d" containerID="c888426f23c24ee821264dfdbb3f9a45f7e3b982073b507b869e884cb18bc0e9" exitCode=0 Jul 21 21:30:39.131199 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:30:39.131049 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-e49e6eb7-openshift-default-5b4cc56b7-nfcp5" Jul 21 21:30:39.131199 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:30:39.131050 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-e49e6eb7-openshift-default-5b4cc56b7-nfcp5" event={"ID":"dc4ff709-fd03-44a0-9610-2549523ca75d","Type":"ContainerDied","Data":"c888426f23c24ee821264dfdbb3f9a45f7e3b982073b507b869e884cb18bc0e9"} Jul 21 21:30:39.131199 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:30:39.131176 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-e49e6eb7-openshift-default-5b4cc56b7-nfcp5" event={"ID":"dc4ff709-fd03-44a0-9610-2549523ca75d","Type":"ContainerDied","Data":"2db1254f56c2c16c33677adf31588d17dc11e815dc85b91c055aa6e35ba5c636"} Jul 21 21:30:39.131296 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:30:39.131204 2575 scope.go:117] "RemoveContainer" containerID="c888426f23c24ee821264dfdbb3f9a45f7e3b982073b507b869e884cb18bc0e9" Jul 21 21:30:39.140979 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:30:39.140955 2575 scope.go:117] "RemoveContainer" containerID="8dc591bbcfd944abc5201db9e5c5d8e2daa2b544e879eb4060b2e78fe3a38ef0" Jul 21 21:30:39.148888 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:30:39.148866 2575 scope.go:117] "RemoveContainer" containerID="c888426f23c24ee821264dfdbb3f9a45f7e3b982073b507b869e884cb18bc0e9" Jul 21 21:30:39.149159 ip-10-0-140-97 kubenswrapper[2575]: E0721 21:30:39.149140 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c888426f23c24ee821264dfdbb3f9a45f7e3b982073b507b869e884cb18bc0e9\": container with ID starting with c888426f23c24ee821264dfdbb3f9a45f7e3b982073b507b869e884cb18bc0e9 not found: ID does not exist" containerID="c888426f23c24ee821264dfdbb3f9a45f7e3b982073b507b869e884cb18bc0e9" Jul 21 21:30:39.149210 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:30:39.149167 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c888426f23c24ee821264dfdbb3f9a45f7e3b982073b507b869e884cb18bc0e9"} err="failed to get container status \"c888426f23c24ee821264dfdbb3f9a45f7e3b982073b507b869e884cb18bc0e9\": rpc error: code = NotFound desc = could not find container \"c888426f23c24ee821264dfdbb3f9a45f7e3b982073b507b869e884cb18bc0e9\": container with ID starting with c888426f23c24ee821264dfdbb3f9a45f7e3b982073b507b869e884cb18bc0e9 not found: ID does not exist" Jul 21 21:30:39.149210 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:30:39.149183 2575 scope.go:117] "RemoveContainer" containerID="8dc591bbcfd944abc5201db9e5c5d8e2daa2b544e879eb4060b2e78fe3a38ef0" Jul 21 21:30:39.149388 ip-10-0-140-97 kubenswrapper[2575]: E0721 21:30:39.149372 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8dc591bbcfd944abc5201db9e5c5d8e2daa2b544e879eb4060b2e78fe3a38ef0\": container with ID starting with 8dc591bbcfd944abc5201db9e5c5d8e2daa2b544e879eb4060b2e78fe3a38ef0 not found: ID does not exist" containerID="8dc591bbcfd944abc5201db9e5c5d8e2daa2b544e879eb4060b2e78fe3a38ef0" Jul 21 21:30:39.149427 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:30:39.149392 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8dc591bbcfd944abc5201db9e5c5d8e2daa2b544e879eb4060b2e78fe3a38ef0"} err="failed to get container status \"8dc591bbcfd944abc5201db9e5c5d8e2daa2b544e879eb4060b2e78fe3a38ef0\": rpc error: code = NotFound desc = could not find container \"8dc591bbcfd944abc5201db9e5c5d8e2daa2b544e879eb4060b2e78fe3a38ef0\": container with ID starting with 8dc591bbcfd944abc5201db9e5c5d8e2daa2b544e879eb4060b2e78fe3a38ef0 not found: ID does not exist" Jul 21 21:30:39.154396 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:30:39.154374 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-e49e6eb7-openshift-default-5b4cc56b7-nfcp5"] Jul 21 21:30:39.158445 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:30:39.158424 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-derive-e49e6eb7-openshift-default-5b4cc56b7-nfcp5"] Jul 21 21:30:40.740648 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:30:40.740612 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dc4ff709-fd03-44a0-9610-2549523ca75d" path="/var/lib/kubelet/pods/dc4ff709-fd03-44a0-9610-2549523ca75d/volumes" Jul 21 21:32:11.895083 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:32:11.895009 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-e31a199a-openshift-default-694bfdf587-bsktr"] Jul 21 21:32:11.895516 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:32:11.895348 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dc4ff709-fd03-44a0-9610-2549523ca75d" containerName="istio-proxy" Jul 21 21:32:11.895516 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:32:11.895359 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="dc4ff709-fd03-44a0-9610-2549523ca75d" containerName="istio-proxy" Jul 21 21:32:11.895516 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:32:11.895375 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dc4ff709-fd03-44a0-9610-2549523ca75d" containerName="istio-proxy" Jul 21 21:32:11.895516 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:32:11.895381 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="dc4ff709-fd03-44a0-9610-2549523ca75d" containerName="istio-proxy" Jul 21 21:32:11.895516 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:32:11.895394 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="52275b9b-d65c-423b-9704-ed570b7ad8c4" containerName="revoke-keys" Jul 21 21:32:11.895516 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:32:11.895400 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="52275b9b-d65c-423b-9704-ed570b7ad8c4" containerName="revoke-keys" Jul 21 21:32:11.895516 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:32:11.895454 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="52275b9b-d65c-423b-9704-ed570b7ad8c4" containerName="revoke-keys" Jul 21 21:32:11.895516 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:32:11.895463 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="dc4ff709-fd03-44a0-9610-2549523ca75d" containerName="istio-proxy" Jul 21 21:32:11.895516 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:32:11.895470 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="dc4ff709-fd03-44a0-9610-2549523ca75d" containerName="istio-proxy" Jul 21 21:32:11.898464 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:32:11.898440 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-e31a199a-openshift-default-694bfdf587-bsktr" Jul 21 21:32:11.900965 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:32:11.900947 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-e31a199a-openshift-default-dockercfg-tw9z9\"" Jul 21 21:32:11.920900 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:32:11.920871 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-e31a199a-openshift-default-694bfdf587-bsktr"] Jul 21 21:32:11.993119 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:32:11.993046 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/be110a31-e791-4de0-889a-a9845df47fa4-istio-token\") pod \"e2e-mt-e31a199a-openshift-default-694bfdf587-bsktr\" (UID: \"be110a31-e791-4de0-889a-a9845df47fa4\") " pod="openshift-ingress/e2e-mt-e31a199a-openshift-default-694bfdf587-bsktr" Jul 21 21:32:11.993318 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:32:11.993137 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/be110a31-e791-4de0-889a-a9845df47fa4-workload-socket\") pod \"e2e-mt-e31a199a-openshift-default-694bfdf587-bsktr\" (UID: \"be110a31-e791-4de0-889a-a9845df47fa4\") " pod="openshift-ingress/e2e-mt-e31a199a-openshift-default-694bfdf587-bsktr" Jul 21 21:32:11.993318 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:32:11.993235 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/be110a31-e791-4de0-889a-a9845df47fa4-credential-socket\") pod \"e2e-mt-e31a199a-openshift-default-694bfdf587-bsktr\" (UID: \"be110a31-e791-4de0-889a-a9845df47fa4\") " pod="openshift-ingress/e2e-mt-e31a199a-openshift-default-694bfdf587-bsktr" Jul 21 21:32:11.993318 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:32:11.993260 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/be110a31-e791-4de0-889a-a9845df47fa4-workload-certs\") pod \"e2e-mt-e31a199a-openshift-default-694bfdf587-bsktr\" (UID: \"be110a31-e791-4de0-889a-a9845df47fa4\") " pod="openshift-ingress/e2e-mt-e31a199a-openshift-default-694bfdf587-bsktr" Jul 21 21:32:11.993318 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:32:11.993282 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/be110a31-e791-4de0-889a-a9845df47fa4-istio-envoy\") pod \"e2e-mt-e31a199a-openshift-default-694bfdf587-bsktr\" (UID: \"be110a31-e791-4de0-889a-a9845df47fa4\") " pod="openshift-ingress/e2e-mt-e31a199a-openshift-default-694bfdf587-bsktr" Jul 21 21:32:11.993318 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:32:11.993304 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/be110a31-e791-4de0-889a-a9845df47fa4-istiod-ca-cert\") pod \"e2e-mt-e31a199a-openshift-default-694bfdf587-bsktr\" (UID: \"be110a31-e791-4de0-889a-a9845df47fa4\") " pod="openshift-ingress/e2e-mt-e31a199a-openshift-default-694bfdf587-bsktr" Jul 21 21:32:11.993818 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:32:11.993352 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/be110a31-e791-4de0-889a-a9845df47fa4-istio-data\") pod \"e2e-mt-e31a199a-openshift-default-694bfdf587-bsktr\" (UID: \"be110a31-e791-4de0-889a-a9845df47fa4\") " pod="openshift-ingress/e2e-mt-e31a199a-openshift-default-694bfdf587-bsktr" Jul 21 21:32:11.993818 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:32:11.993414 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4rg7k\" (UniqueName: \"kubernetes.io/projected/be110a31-e791-4de0-889a-a9845df47fa4-kube-api-access-4rg7k\") pod \"e2e-mt-e31a199a-openshift-default-694bfdf587-bsktr\" (UID: \"be110a31-e791-4de0-889a-a9845df47fa4\") " pod="openshift-ingress/e2e-mt-e31a199a-openshift-default-694bfdf587-bsktr" Jul 21 21:32:11.993818 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:32:11.993557 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/be110a31-e791-4de0-889a-a9845df47fa4-istio-podinfo\") pod \"e2e-mt-e31a199a-openshift-default-694bfdf587-bsktr\" (UID: \"be110a31-e791-4de0-889a-a9845df47fa4\") " pod="openshift-ingress/e2e-mt-e31a199a-openshift-default-694bfdf587-bsktr" Jul 21 21:32:12.094300 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:32:12.094271 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/be110a31-e791-4de0-889a-a9845df47fa4-istio-token\") pod \"e2e-mt-e31a199a-openshift-default-694bfdf587-bsktr\" (UID: \"be110a31-e791-4de0-889a-a9845df47fa4\") " pod="openshift-ingress/e2e-mt-e31a199a-openshift-default-694bfdf587-bsktr" Jul 21 21:32:12.094500 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:32:12.094308 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/be110a31-e791-4de0-889a-a9845df47fa4-workload-socket\") pod \"e2e-mt-e31a199a-openshift-default-694bfdf587-bsktr\" (UID: \"be110a31-e791-4de0-889a-a9845df47fa4\") " pod="openshift-ingress/e2e-mt-e31a199a-openshift-default-694bfdf587-bsktr" Jul 21 21:32:12.094500 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:32:12.094339 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/be110a31-e791-4de0-889a-a9845df47fa4-credential-socket\") pod \"e2e-mt-e31a199a-openshift-default-694bfdf587-bsktr\" (UID: \"be110a31-e791-4de0-889a-a9845df47fa4\") " pod="openshift-ingress/e2e-mt-e31a199a-openshift-default-694bfdf587-bsktr" Jul 21 21:32:12.094500 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:32:12.094457 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/be110a31-e791-4de0-889a-a9845df47fa4-workload-certs\") pod \"e2e-mt-e31a199a-openshift-default-694bfdf587-bsktr\" (UID: \"be110a31-e791-4de0-889a-a9845df47fa4\") " pod="openshift-ingress/e2e-mt-e31a199a-openshift-default-694bfdf587-bsktr" Jul 21 21:32:12.094721 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:32:12.094503 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/be110a31-e791-4de0-889a-a9845df47fa4-istio-envoy\") pod \"e2e-mt-e31a199a-openshift-default-694bfdf587-bsktr\" (UID: \"be110a31-e791-4de0-889a-a9845df47fa4\") " pod="openshift-ingress/e2e-mt-e31a199a-openshift-default-694bfdf587-bsktr" Jul 21 21:32:12.094721 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:32:12.094555 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/be110a31-e791-4de0-889a-a9845df47fa4-istiod-ca-cert\") pod \"e2e-mt-e31a199a-openshift-default-694bfdf587-bsktr\" (UID: \"be110a31-e791-4de0-889a-a9845df47fa4\") " pod="openshift-ingress/e2e-mt-e31a199a-openshift-default-694bfdf587-bsktr" Jul 21 21:32:12.094721 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:32:12.094603 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/be110a31-e791-4de0-889a-a9845df47fa4-istio-data\") pod \"e2e-mt-e31a199a-openshift-default-694bfdf587-bsktr\" (UID: \"be110a31-e791-4de0-889a-a9845df47fa4\") " pod="openshift-ingress/e2e-mt-e31a199a-openshift-default-694bfdf587-bsktr" Jul 21 21:32:12.094721 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:32:12.094668 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4rg7k\" (UniqueName: \"kubernetes.io/projected/be110a31-e791-4de0-889a-a9845df47fa4-kube-api-access-4rg7k\") pod \"e2e-mt-e31a199a-openshift-default-694bfdf587-bsktr\" (UID: \"be110a31-e791-4de0-889a-a9845df47fa4\") " pod="openshift-ingress/e2e-mt-e31a199a-openshift-default-694bfdf587-bsktr" Jul 21 21:32:12.094721 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:32:12.094712 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/be110a31-e791-4de0-889a-a9845df47fa4-workload-socket\") pod \"e2e-mt-e31a199a-openshift-default-694bfdf587-bsktr\" (UID: \"be110a31-e791-4de0-889a-a9845df47fa4\") " pod="openshift-ingress/e2e-mt-e31a199a-openshift-default-694bfdf587-bsktr" Jul 21 21:32:12.094979 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:32:12.094782 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/be110a31-e791-4de0-889a-a9845df47fa4-istio-podinfo\") pod \"e2e-mt-e31a199a-openshift-default-694bfdf587-bsktr\" (UID: \"be110a31-e791-4de0-889a-a9845df47fa4\") " pod="openshift-ingress/e2e-mt-e31a199a-openshift-default-694bfdf587-bsktr" Jul 21 21:32:12.094979 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:32:12.094830 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/be110a31-e791-4de0-889a-a9845df47fa4-workload-certs\") pod \"e2e-mt-e31a199a-openshift-default-694bfdf587-bsktr\" (UID: \"be110a31-e791-4de0-889a-a9845df47fa4\") " pod="openshift-ingress/e2e-mt-e31a199a-openshift-default-694bfdf587-bsktr" Jul 21 21:32:12.095106 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:32:12.094951 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/be110a31-e791-4de0-889a-a9845df47fa4-istio-data\") pod \"e2e-mt-e31a199a-openshift-default-694bfdf587-bsktr\" (UID: \"be110a31-e791-4de0-889a-a9845df47fa4\") " pod="openshift-ingress/e2e-mt-e31a199a-openshift-default-694bfdf587-bsktr" Jul 21 21:32:12.095106 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:32:12.095003 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/be110a31-e791-4de0-889a-a9845df47fa4-credential-socket\") pod \"e2e-mt-e31a199a-openshift-default-694bfdf587-bsktr\" (UID: \"be110a31-e791-4de0-889a-a9845df47fa4\") " pod="openshift-ingress/e2e-mt-e31a199a-openshift-default-694bfdf587-bsktr" Jul 21 21:32:12.095276 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:32:12.095250 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/be110a31-e791-4de0-889a-a9845df47fa4-istiod-ca-cert\") pod \"e2e-mt-e31a199a-openshift-default-694bfdf587-bsktr\" (UID: \"be110a31-e791-4de0-889a-a9845df47fa4\") " pod="openshift-ingress/e2e-mt-e31a199a-openshift-default-694bfdf587-bsktr" Jul 21 21:32:12.096934 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:32:12.096895 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/be110a31-e791-4de0-889a-a9845df47fa4-istio-envoy\") pod \"e2e-mt-e31a199a-openshift-default-694bfdf587-bsktr\" (UID: \"be110a31-e791-4de0-889a-a9845df47fa4\") " pod="openshift-ingress/e2e-mt-e31a199a-openshift-default-694bfdf587-bsktr" Jul 21 21:32:12.097289 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:32:12.097270 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/be110a31-e791-4de0-889a-a9845df47fa4-istio-podinfo\") pod \"e2e-mt-e31a199a-openshift-default-694bfdf587-bsktr\" (UID: \"be110a31-e791-4de0-889a-a9845df47fa4\") " pod="openshift-ingress/e2e-mt-e31a199a-openshift-default-694bfdf587-bsktr" Jul 21 21:32:12.103334 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:32:12.103308 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/be110a31-e791-4de0-889a-a9845df47fa4-istio-token\") pod \"e2e-mt-e31a199a-openshift-default-694bfdf587-bsktr\" (UID: \"be110a31-e791-4de0-889a-a9845df47fa4\") " pod="openshift-ingress/e2e-mt-e31a199a-openshift-default-694bfdf587-bsktr" Jul 21 21:32:12.103453 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:32:12.103426 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4rg7k\" (UniqueName: \"kubernetes.io/projected/be110a31-e791-4de0-889a-a9845df47fa4-kube-api-access-4rg7k\") pod \"e2e-mt-e31a199a-openshift-default-694bfdf587-bsktr\" (UID: \"be110a31-e791-4de0-889a-a9845df47fa4\") " pod="openshift-ingress/e2e-mt-e31a199a-openshift-default-694bfdf587-bsktr" Jul 21 21:32:12.208573 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:32:12.208472 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-e31a199a-openshift-default-694bfdf587-bsktr" Jul 21 21:32:12.352625 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:32:12.352589 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-e31a199a-openshift-default-694bfdf587-bsktr"] Jul 21 21:32:12.354484 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:32:12.354456 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbe110a31_e791_4de0_889a_a9845df47fa4.slice/crio-cce46a07083073706e17093597ff9ef66645bc60a6c4d5474dd8ed01b6e216a6 WatchSource:0}: Error finding container cce46a07083073706e17093597ff9ef66645bc60a6c4d5474dd8ed01b6e216a6: Status 404 returned error can't find the container with id cce46a07083073706e17093597ff9ef66645bc60a6c4d5474dd8ed01b6e216a6 Jul 21 21:32:12.356605 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:32:12.356566 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 21 21:32:12.356728 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:32:12.356657 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 21 21:32:12.356728 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:32:12.356684 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 21 21:32:12.594922 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:32:12.594840 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-e31a199a-openshift-default-694bfdf587-bsktr" event={"ID":"be110a31-e791-4de0-889a-a9845df47fa4","Type":"ContainerStarted","Data":"408ccc78a21c9d65f42793a234cec27d6054b3bb606ce58b46335631a033133c"} Jul 21 21:32:12.594922 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:32:12.594897 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-e31a199a-openshift-default-694bfdf587-bsktr" event={"ID":"be110a31-e791-4de0-889a-a9845df47fa4","Type":"ContainerStarted","Data":"cce46a07083073706e17093597ff9ef66645bc60a6c4d5474dd8ed01b6e216a6"} Jul 21 21:32:12.617362 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:32:12.617320 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-e31a199a-openshift-default-694bfdf587-bsktr" podStartSLOduration=1.617303591 podStartE2EDuration="1.617303591s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 21:32:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 21:32:12.615385696 +0000 UTC m=+1890.550135383" watchObservedRunningTime="2026-07-21 21:32:12.617303591 +0000 UTC m=+1890.552053257" Jul 21 21:32:13.209375 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:32:13.209337 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-e31a199a-openshift-default-694bfdf587-bsktr" Jul 21 21:32:13.214091 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:32:13.214066 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-e31a199a-openshift-default-694bfdf587-bsktr" Jul 21 21:32:13.600622 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:32:13.600523 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-e31a199a-openshift-default-694bfdf587-bsktr" Jul 21 21:32:13.601647 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:32:13.601627 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-e31a199a-openshift-default-694bfdf587-bsktr" Jul 21 21:32:26.671841 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:32:26.671808 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e31a199a-j5d9x"] Jul 21 21:32:26.675795 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:32:26.675774 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e31a199a-j5d9x" Jul 21 21:32:26.678642 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:32:26.678622 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9fxhr\"" Jul 21 21:32:26.678744 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:32:26.678650 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 21 21:32:26.684024 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:32:26.684003 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e31a199a-j5d9x"] Jul 21 21:32:26.738521 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:32:26.738486 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fcdef09c-e25e-4768-ad3b-e288cc3c554a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-e31a199a-j5d9x\" (UID: \"fcdef09c-e25e-4768-ad3b-e288cc3c554a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e31a199a-j5d9x" Jul 21 21:32:26.839883 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:32:26.839832 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fcdef09c-e25e-4768-ad3b-e288cc3c554a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-e31a199a-j5d9x\" (UID: \"fcdef09c-e25e-4768-ad3b-e288cc3c554a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e31a199a-j5d9x" Jul 21 21:32:26.840660 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:32:26.840634 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fcdef09c-e25e-4768-ad3b-e288cc3c554a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-e31a199a-j5d9x\" (UID: \"fcdef09c-e25e-4768-ad3b-e288cc3c554a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e31a199a-j5d9x" Jul 21 21:32:26.989063 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:32:26.988978 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e31a199a-j5d9x" Jul 21 21:32:27.127565 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:32:27.127522 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e31a199a-j5d9x"] Jul 21 21:32:27.129934 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:32:27.129908 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfcdef09c_e25e_4768_ad3b_e288cc3c554a.slice/crio-f1a6265bf5b2cdd24267154947f3920556f3319cdcc55916ecbef17a30adb1d9 WatchSource:0}: Error finding container f1a6265bf5b2cdd24267154947f3920556f3319cdcc55916ecbef17a30adb1d9: Status 404 returned error can't find the container with id f1a6265bf5b2cdd24267154947f3920556f3319cdcc55916ecbef17a30adb1d9 Jul 21 21:32:27.684222 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:32:27.684185 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e31a199a-j5d9x" event={"ID":"fcdef09c-e25e-4768-ad3b-e288cc3c554a","Type":"ContainerStarted","Data":"3dcf28709b8584aa0f45e8a04cba2661c3aed097aa31b5ae0782a6f80019282a"} Jul 21 21:32:27.684715 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:32:27.684244 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e31a199a-j5d9x" event={"ID":"fcdef09c-e25e-4768-ad3b-e288cc3c554a","Type":"ContainerStarted","Data":"f1a6265bf5b2cdd24267154947f3920556f3319cdcc55916ecbef17a30adb1d9"} Jul 21 21:32:27.698994 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:32:27.698950 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e31a199a-j5d9x" podStartSLOduration=1.698931722 podStartE2EDuration="1.698931722s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 21:32:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 21:32:27.698438021 +0000 UTC m=+1905.633187708" watchObservedRunningTime="2026-07-21 21:32:27.698931722 +0000 UTC m=+1905.633681389" Jul 21 21:32:28.601890 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:32:28.601796 2575 patch_prober.go:28] interesting pod/e2e-mt-e31a199a-openshift-default-694bfdf587-bsktr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" start-of-body= Jul 21 21:32:28.601890 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:32:28.601855 2575 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-e31a199a-openshift-default-694bfdf587-bsktr" podUID="be110a31-e791-4de0-889a-a9845df47fa4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" Jul 21 21:32:28.690968 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:32:28.690930 2575 generic.go:356] "Generic (PLEG): container finished" podID="fcdef09c-e25e-4768-ad3b-e288cc3c554a" containerID="3dcf28709b8584aa0f45e8a04cba2661c3aed097aa31b5ae0782a6f80019282a" exitCode=0 Jul 21 21:32:28.691364 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:32:28.691016 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e31a199a-j5d9x" event={"ID":"fcdef09c-e25e-4768-ad3b-e288cc3c554a","Type":"ContainerDied","Data":"3dcf28709b8584aa0f45e8a04cba2661c3aed097aa31b5ae0782a6f80019282a"} Jul 21 21:32:28.692610 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:32:28.692586 2575 generic.go:356] "Generic (PLEG): container finished" podID="be110a31-e791-4de0-889a-a9845df47fa4" containerID="408ccc78a21c9d65f42793a234cec27d6054b3bb606ce58b46335631a033133c" exitCode=0 Jul 21 21:32:28.692719 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:32:28.692623 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-e31a199a-openshift-default-694bfdf587-bsktr" event={"ID":"be110a31-e791-4de0-889a-a9845df47fa4","Type":"ContainerDied","Data":"408ccc78a21c9d65f42793a234cec27d6054b3bb606ce58b46335631a033133c"} Jul 21 21:32:28.692989 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:32:28.692976 2575 scope.go:117] "RemoveContainer" containerID="408ccc78a21c9d65f42793a234cec27d6054b3bb606ce58b46335631a033133c" Jul 21 21:32:28.693978 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:32:28.693945 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 21 21:32:28.694068 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:32:28.694034 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 21 21:32:28.694128 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:32:28.694084 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 21 21:32:29.699800 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:32:29.699767 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-e31a199a-openshift-default-694bfdf587-bsktr" event={"ID":"be110a31-e791-4de0-889a-a9845df47fa4","Type":"ContainerStarted","Data":"f098f09979641d3c560baafb0db12da015bf81170d8732bd3b49e25a71b92961"} Jul 21 21:32:29.869238 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:32:29.869208 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e31a199a-j5d9x" Jul 21 21:32:29.973604 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:32:29.973504 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fcdef09c-e25e-4768-ad3b-e288cc3c554a-maas-api-service-ca\") pod \"fcdef09c-e25e-4768-ad3b-e288cc3c554a\" (UID: \"fcdef09c-e25e-4768-ad3b-e288cc3c554a\") " Jul 21 21:32:29.973830 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:32:29.973809 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fcdef09c-e25e-4768-ad3b-e288cc3c554a-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "fcdef09c-e25e-4768-ad3b-e288cc3c554a" (UID: "fcdef09c-e25e-4768-ad3b-e288cc3c554a"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 21 21:32:30.074724 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:32:30.074685 2575 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fcdef09c-e25e-4768-ad3b-e288cc3c554a-maas-api-service-ca\") on node \"ip-10-0-140-97.ec2.internal\" DevicePath \"\"" Jul 21 21:32:30.209138 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:32:30.209101 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-e31a199a-openshift-default-694bfdf587-bsktr" Jul 21 21:32:30.209405 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:32:30.209382 2575 patch_prober.go:28] interesting pod/e2e-mt-e31a199a-openshift-default-694bfdf587-bsktr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" start-of-body= Jul 21 21:32:30.209451 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:32:30.209430 2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-e31a199a-openshift-default-694bfdf587-bsktr" podUID="be110a31-e791-4de0-889a-a9845df47fa4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" Jul 21 21:32:30.705862 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:32:30.705825 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e31a199a-j5d9x" Jul 21 21:32:30.706281 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:32:30.705856 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e31a199a-j5d9x" event={"ID":"fcdef09c-e25e-4768-ad3b-e288cc3c554a","Type":"ContainerDied","Data":"f1a6265bf5b2cdd24267154947f3920556f3319cdcc55916ecbef17a30adb1d9"} Jul 21 21:32:30.706281 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:32:30.705903 2575 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f1a6265bf5b2cdd24267154947f3920556f3319cdcc55916ecbef17a30adb1d9" Jul 21 21:32:31.209240 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:32:31.209195 2575 patch_prober.go:28] interesting pod/e2e-mt-e31a199a-openshift-default-694bfdf587-bsktr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" start-of-body= Jul 21 21:32:31.209446 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:32:31.209263 2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-e31a199a-openshift-default-694bfdf587-bsktr" podUID="be110a31-e791-4de0-889a-a9845df47fa4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" Jul 21 21:32:32.209346 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:32:32.209311 2575 patch_prober.go:28] interesting pod/e2e-mt-e31a199a-openshift-default-694bfdf587-bsktr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" start-of-body= Jul 21 21:32:32.209749 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:32:32.209362 2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-e31a199a-openshift-default-694bfdf587-bsktr" podUID="be110a31-e791-4de0-889a-a9845df47fa4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" Jul 21 21:32:33.209638 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:32:33.209601 2575 patch_prober.go:28] interesting pod/e2e-mt-e31a199a-openshift-default-694bfdf587-bsktr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" start-of-body= Jul 21 21:32:33.210019 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:32:33.209657 2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-e31a199a-openshift-default-694bfdf587-bsktr" podUID="be110a31-e791-4de0-889a-a9845df47fa4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" Jul 21 21:32:34.213097 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:32:34.213065 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-e31a199a-openshift-default-694bfdf587-bsktr" Jul 21 21:32:34.213458 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:32:34.213394 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-e31a199a-openshift-default-694bfdf587-bsktr" Jul 21 21:32:34.214186 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:32:34.214164 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-e31a199a-openshift-default-694bfdf587-bsktr" Jul 21 21:32:41.787676 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:32:41.787639 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e31a199a-j5d9x"] Jul 21 21:32:41.789361 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:32:41.789339 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e31a199a-j5d9x"] Jul 21 21:32:42.740634 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:32:42.740600 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fcdef09c-e25e-4768-ad3b-e288cc3c554a" path="/var/lib/kubelet/pods/fcdef09c-e25e-4768-ad3b-e288cc3c554a/volumes" Jul 21 21:32:48.731743 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:32:48.731706 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-e31a199a-openshift-default-694bfdf587-bsktr"] Jul 21 21:32:48.732167 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:32:48.731973 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-e31a199a-openshift-default-694bfdf587-bsktr" podUID="be110a31-e791-4de0-889a-a9845df47fa4" containerName="istio-proxy" containerID="cri-o://f098f09979641d3c560baafb0db12da015bf81170d8732bd3b49e25a71b92961" gracePeriod=30 Jul 21 21:32:49.215843 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:32:49.215798 2575 patch_prober.go:28] interesting pod/e2e-mt-e31a199a-openshift-default-694bfdf587-bsktr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 21 21:32:49.216007 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:32:49.215888 2575 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-e31a199a-openshift-default-694bfdf587-bsktr" podUID="be110a31-e791-4de0-889a-a9845df47fa4" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 21 21:32:53.981834 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:32:53.981808 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-e31a199a-openshift-default-694bfdf587-bsktr" Jul 21 21:32:54.093489 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:32:54.093400 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/be110a31-e791-4de0-889a-a9845df47fa4-istio-token\") pod \"be110a31-e791-4de0-889a-a9845df47fa4\" (UID: \"be110a31-e791-4de0-889a-a9845df47fa4\") " Jul 21 21:32:54.093489 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:32:54.093450 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/be110a31-e791-4de0-889a-a9845df47fa4-workload-socket\") pod \"be110a31-e791-4de0-889a-a9845df47fa4\" (UID: \"be110a31-e791-4de0-889a-a9845df47fa4\") " Jul 21 21:32:54.093489 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:32:54.093473 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/be110a31-e791-4de0-889a-a9845df47fa4-istio-data\") pod \"be110a31-e791-4de0-889a-a9845df47fa4\" (UID: \"be110a31-e791-4de0-889a-a9845df47fa4\") " Jul 21 21:32:54.093766 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:32:54.093503 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/be110a31-e791-4de0-889a-a9845df47fa4-istio-envoy\") pod \"be110a31-e791-4de0-889a-a9845df47fa4\" (UID: \"be110a31-e791-4de0-889a-a9845df47fa4\") " Jul 21 21:32:54.093766 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:32:54.093565 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/be110a31-e791-4de0-889a-a9845df47fa4-istio-podinfo\") pod \"be110a31-e791-4de0-889a-a9845df47fa4\" (UID: \"be110a31-e791-4de0-889a-a9845df47fa4\") " Jul 21 21:32:54.093766 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:32:54.093606 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/be110a31-e791-4de0-889a-a9845df47fa4-istiod-ca-cert\") pod \"be110a31-e791-4de0-889a-a9845df47fa4\" (UID: \"be110a31-e791-4de0-889a-a9845df47fa4\") " Jul 21 21:32:54.093766 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:32:54.093624 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4rg7k\" (UniqueName: \"kubernetes.io/projected/be110a31-e791-4de0-889a-a9845df47fa4-kube-api-access-4rg7k\") pod \"be110a31-e791-4de0-889a-a9845df47fa4\" (UID: \"be110a31-e791-4de0-889a-a9845df47fa4\") " Jul 21 21:32:54.093766 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:32:54.093670 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/be110a31-e791-4de0-889a-a9845df47fa4-credential-socket\") pod \"be110a31-e791-4de0-889a-a9845df47fa4\" (UID: \"be110a31-e791-4de0-889a-a9845df47fa4\") " Jul 21 21:32:54.093766 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:32:54.093733 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/be110a31-e791-4de0-889a-a9845df47fa4-workload-certs\") pod \"be110a31-e791-4de0-889a-a9845df47fa4\" (UID: \"be110a31-e791-4de0-889a-a9845df47fa4\") " Jul 21 21:32:54.093766 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:32:54.093753 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be110a31-e791-4de0-889a-a9845df47fa4-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "be110a31-e791-4de0-889a-a9845df47fa4" (UID: "be110a31-e791-4de0-889a-a9845df47fa4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 21:32:54.094122 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:32:54.094013 2575 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/be110a31-e791-4de0-889a-a9845df47fa4-workload-socket\") on node \"ip-10-0-140-97.ec2.internal\" DevicePath \"\"" Jul 21 21:32:54.094192 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:32:54.094110 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/be110a31-e791-4de0-889a-a9845df47fa4-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "be110a31-e791-4de0-889a-a9845df47fa4" (UID: "be110a31-e791-4de0-889a-a9845df47fa4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 21 21:32:54.094266 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:32:54.094249 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be110a31-e791-4de0-889a-a9845df47fa4-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "be110a31-e791-4de0-889a-a9845df47fa4" (UID: "be110a31-e791-4de0-889a-a9845df47fa4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 21:32:54.094358 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:32:54.094330 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be110a31-e791-4de0-889a-a9845df47fa4-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "be110a31-e791-4de0-889a-a9845df47fa4" (UID: "be110a31-e791-4de0-889a-a9845df47fa4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 21:32:54.094677 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:32:54.094649 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be110a31-e791-4de0-889a-a9845df47fa4-istio-data" (OuterVolumeSpecName: "istio-data") pod "be110a31-e791-4de0-889a-a9845df47fa4" (UID: "be110a31-e791-4de0-889a-a9845df47fa4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 21:32:54.095827 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:32:54.095797 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/be110a31-e791-4de0-889a-a9845df47fa4-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "be110a31-e791-4de0-889a-a9845df47fa4" (UID: "be110a31-e791-4de0-889a-a9845df47fa4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 21 21:32:54.095931 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:32:54.095875 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be110a31-e791-4de0-889a-a9845df47fa4-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "be110a31-e791-4de0-889a-a9845df47fa4" (UID: "be110a31-e791-4de0-889a-a9845df47fa4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 21:32:54.096011 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:32:54.095985 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/be110a31-e791-4de0-889a-a9845df47fa4-kube-api-access-4rg7k" (OuterVolumeSpecName: "kube-api-access-4rg7k") pod "be110a31-e791-4de0-889a-a9845df47fa4" (UID: "be110a31-e791-4de0-889a-a9845df47fa4"). InnerVolumeSpecName "kube-api-access-4rg7k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 21 21:32:54.096070 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:32:54.096021 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/be110a31-e791-4de0-889a-a9845df47fa4-istio-token" (OuterVolumeSpecName: "istio-token") pod "be110a31-e791-4de0-889a-a9845df47fa4" (UID: "be110a31-e791-4de0-889a-a9845df47fa4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 21 21:32:54.194962 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:32:54.194930 2575 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/be110a31-e791-4de0-889a-a9845df47fa4-istio-podinfo\") on node \"ip-10-0-140-97.ec2.internal\" DevicePath \"\"" Jul 21 21:32:54.194962 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:32:54.194957 2575 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/be110a31-e791-4de0-889a-a9845df47fa4-istiod-ca-cert\") on node \"ip-10-0-140-97.ec2.internal\" DevicePath \"\"" Jul 21 21:32:54.194962 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:32:54.194965 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4rg7k\" (UniqueName: \"kubernetes.io/projected/be110a31-e791-4de0-889a-a9845df47fa4-kube-api-access-4rg7k\") on node \"ip-10-0-140-97.ec2.internal\" DevicePath \"\"" Jul 21 21:32:54.195190 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:32:54.194975 2575 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/be110a31-e791-4de0-889a-a9845df47fa4-credential-socket\") on node \"ip-10-0-140-97.ec2.internal\" DevicePath \"\"" Jul 21 21:32:54.195190 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:32:54.194984 2575 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/be110a31-e791-4de0-889a-a9845df47fa4-workload-certs\") on node \"ip-10-0-140-97.ec2.internal\" DevicePath \"\"" Jul 21 21:32:54.195190 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:32:54.194992 2575 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/be110a31-e791-4de0-889a-a9845df47fa4-istio-token\") on node \"ip-10-0-140-97.ec2.internal\" DevicePath \"\"" Jul 21 21:32:54.195190 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:32:54.195000 2575 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/be110a31-e791-4de0-889a-a9845df47fa4-istio-data\") on node \"ip-10-0-140-97.ec2.internal\" DevicePath \"\"" Jul 21 21:32:54.195190 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:32:54.195006 2575 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/be110a31-e791-4de0-889a-a9845df47fa4-istio-envoy\") on node \"ip-10-0-140-97.ec2.internal\" DevicePath \"\"" Jul 21 21:32:54.825677 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:32:54.825643 2575 generic.go:356] "Generic (PLEG): container finished" podID="be110a31-e791-4de0-889a-a9845df47fa4" containerID="f098f09979641d3c560baafb0db12da015bf81170d8732bd3b49e25a71b92961" exitCode=0 Jul 21 21:32:54.825863 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:32:54.825699 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-e31a199a-openshift-default-694bfdf587-bsktr" Jul 21 21:32:54.825863 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:32:54.825720 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-e31a199a-openshift-default-694bfdf587-bsktr" event={"ID":"be110a31-e791-4de0-889a-a9845df47fa4","Type":"ContainerDied","Data":"f098f09979641d3c560baafb0db12da015bf81170d8732bd3b49e25a71b92961"} Jul 21 21:32:54.825863 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:32:54.825772 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-e31a199a-openshift-default-694bfdf587-bsktr" event={"ID":"be110a31-e791-4de0-889a-a9845df47fa4","Type":"ContainerDied","Data":"cce46a07083073706e17093597ff9ef66645bc60a6c4d5474dd8ed01b6e216a6"} Jul 21 21:32:54.825863 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:32:54.825792 2575 scope.go:117] "RemoveContainer" containerID="f098f09979641d3c560baafb0db12da015bf81170d8732bd3b49e25a71b92961" Jul 21 21:32:54.834460 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:32:54.834446 2575 scope.go:117] "RemoveContainer" containerID="408ccc78a21c9d65f42793a234cec27d6054b3bb606ce58b46335631a033133c" Jul 21 21:32:54.841949 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:32:54.841931 2575 scope.go:117] "RemoveContainer" containerID="f098f09979641d3c560baafb0db12da015bf81170d8732bd3b49e25a71b92961" Jul 21 21:32:54.842350 ip-10-0-140-97 kubenswrapper[2575]: E0721 21:32:54.842298 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f098f09979641d3c560baafb0db12da015bf81170d8732bd3b49e25a71b92961\": container with ID starting with f098f09979641d3c560baafb0db12da015bf81170d8732bd3b49e25a71b92961 not found: ID does not exist" containerID="f098f09979641d3c560baafb0db12da015bf81170d8732bd3b49e25a71b92961" Jul 21 21:32:54.842473 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:32:54.842351 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f098f09979641d3c560baafb0db12da015bf81170d8732bd3b49e25a71b92961"} err="failed to get container status \"f098f09979641d3c560baafb0db12da015bf81170d8732bd3b49e25a71b92961\": rpc error: code = NotFound desc = could not find container \"f098f09979641d3c560baafb0db12da015bf81170d8732bd3b49e25a71b92961\": container with ID starting with f098f09979641d3c560baafb0db12da015bf81170d8732bd3b49e25a71b92961 not found: ID does not exist" Jul 21 21:32:54.842473 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:32:54.842375 2575 scope.go:117] "RemoveContainer" containerID="408ccc78a21c9d65f42793a234cec27d6054b3bb606ce58b46335631a033133c" Jul 21 21:32:54.843156 ip-10-0-140-97 kubenswrapper[2575]: E0721 21:32:54.843131 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"408ccc78a21c9d65f42793a234cec27d6054b3bb606ce58b46335631a033133c\": container with ID starting with 408ccc78a21c9d65f42793a234cec27d6054b3bb606ce58b46335631a033133c not found: ID does not exist" containerID="408ccc78a21c9d65f42793a234cec27d6054b3bb606ce58b46335631a033133c" Jul 21 21:32:54.843249 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:32:54.843160 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"408ccc78a21c9d65f42793a234cec27d6054b3bb606ce58b46335631a033133c"} err="failed to get container status \"408ccc78a21c9d65f42793a234cec27d6054b3bb606ce58b46335631a033133c\": rpc error: code = NotFound desc = could not find container \"408ccc78a21c9d65f42793a234cec27d6054b3bb606ce58b46335631a033133c\": container with ID starting with 408ccc78a21c9d65f42793a234cec27d6054b3bb606ce58b46335631a033133c not found: ID does not exist" Jul 21 21:32:54.845322 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:32:54.845303 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-e31a199a-openshift-default-694bfdf587-bsktr"] Jul 21 21:32:54.849583 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:32:54.849562 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-e31a199a-openshift-default-694bfdf587-bsktr"] Jul 21 21:32:56.740429 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:32:56.740397 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="be110a31-e791-4de0-889a-a9845df47fa4" path="/var/lib/kubelet/pods/be110a31-e791-4de0-889a-a9845df47fa4/volumes" Jul 21 21:33:08.740853 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:33:08.740820 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-ebe935a0-openshift-default-6bbfc74bf5-dkgll"] Jul 21 21:33:08.741350 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:33:08.741173 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="be110a31-e791-4de0-889a-a9845df47fa4" containerName="istio-proxy" Jul 21 21:33:08.741350 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:33:08.741184 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="be110a31-e791-4de0-889a-a9845df47fa4" containerName="istio-proxy" Jul 21 21:33:08.741350 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:33:08.741193 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="be110a31-e791-4de0-889a-a9845df47fa4" containerName="istio-proxy" Jul 21 21:33:08.741350 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:33:08.741199 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="be110a31-e791-4de0-889a-a9845df47fa4" containerName="istio-proxy" Jul 21 21:33:08.741350 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:33:08.741213 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fcdef09c-e25e-4768-ad3b-e288cc3c554a" containerName="revoke-keys" Jul 21 21:33:08.741350 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:33:08.741218 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="fcdef09c-e25e-4768-ad3b-e288cc3c554a" containerName="revoke-keys" Jul 21 21:33:08.741350 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:33:08.741275 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="be110a31-e791-4de0-889a-a9845df47fa4" containerName="istio-proxy" Jul 21 21:33:08.741350 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:33:08.741282 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="fcdef09c-e25e-4768-ad3b-e288cc3c554a" containerName="revoke-keys" Jul 21 21:33:08.744356 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:33:08.744336 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-ebe935a0-openshift-default-6bbfc74bf5-dkgll" Jul 21 21:33:08.747565 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:33:08.747545 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-ebe935a0-openshift-default-dockercfg-9bn2v\"" Jul 21 21:33:08.763566 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:33:08.763541 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-ebe935a0-openshift-default-6bbfc74bf5-dkgll"] Jul 21 21:33:08.828676 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:33:08.828650 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5618f433-dfcd-4f4f-bb7a-5f2306b5874c-istio-token\") pod \"e2e-mt-ebe935a0-openshift-default-6bbfc74bf5-dkgll\" (UID: \"5618f433-dfcd-4f4f-bb7a-5f2306b5874c\") " pod="openshift-ingress/e2e-mt-ebe935a0-openshift-default-6bbfc74bf5-dkgll" Jul 21 21:33:08.828676 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:33:08.828680 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5618f433-dfcd-4f4f-bb7a-5f2306b5874c-istio-data\") pod \"e2e-mt-ebe935a0-openshift-default-6bbfc74bf5-dkgll\" (UID: \"5618f433-dfcd-4f4f-bb7a-5f2306b5874c\") " pod="openshift-ingress/e2e-mt-ebe935a0-openshift-default-6bbfc74bf5-dkgll" Jul 21 21:33:08.828942 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:33:08.828700 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5618f433-dfcd-4f4f-bb7a-5f2306b5874c-istiod-ca-cert\") pod \"e2e-mt-ebe935a0-openshift-default-6bbfc74bf5-dkgll\" (UID: \"5618f433-dfcd-4f4f-bb7a-5f2306b5874c\") " pod="openshift-ingress/e2e-mt-ebe935a0-openshift-default-6bbfc74bf5-dkgll" Jul 21 21:33:08.828942 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:33:08.828759 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n7zh6\" (UniqueName: \"kubernetes.io/projected/5618f433-dfcd-4f4f-bb7a-5f2306b5874c-kube-api-access-n7zh6\") pod \"e2e-mt-ebe935a0-openshift-default-6bbfc74bf5-dkgll\" (UID: \"5618f433-dfcd-4f4f-bb7a-5f2306b5874c\") " pod="openshift-ingress/e2e-mt-ebe935a0-openshift-default-6bbfc74bf5-dkgll" Jul 21 21:33:08.828942 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:33:08.828850 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5618f433-dfcd-4f4f-bb7a-5f2306b5874c-istio-envoy\") pod \"e2e-mt-ebe935a0-openshift-default-6bbfc74bf5-dkgll\" (UID: \"5618f433-dfcd-4f4f-bb7a-5f2306b5874c\") " pod="openshift-ingress/e2e-mt-ebe935a0-openshift-default-6bbfc74bf5-dkgll" Jul 21 21:33:08.828942 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:33:08.828905 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5618f433-dfcd-4f4f-bb7a-5f2306b5874c-workload-socket\") pod \"e2e-mt-ebe935a0-openshift-default-6bbfc74bf5-dkgll\" (UID: \"5618f433-dfcd-4f4f-bb7a-5f2306b5874c\") " pod="openshift-ingress/e2e-mt-ebe935a0-openshift-default-6bbfc74bf5-dkgll" Jul 21 21:33:08.829198 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:33:08.828964 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5618f433-dfcd-4f4f-bb7a-5f2306b5874c-istio-podinfo\") pod \"e2e-mt-ebe935a0-openshift-default-6bbfc74bf5-dkgll\" (UID: \"5618f433-dfcd-4f4f-bb7a-5f2306b5874c\") " pod="openshift-ingress/e2e-mt-ebe935a0-openshift-default-6bbfc74bf5-dkgll" Jul 21 21:33:08.829198 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:33:08.829052 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5618f433-dfcd-4f4f-bb7a-5f2306b5874c-credential-socket\") pod \"e2e-mt-ebe935a0-openshift-default-6bbfc74bf5-dkgll\" (UID: \"5618f433-dfcd-4f4f-bb7a-5f2306b5874c\") " pod="openshift-ingress/e2e-mt-ebe935a0-openshift-default-6bbfc74bf5-dkgll" Jul 21 21:33:08.831630 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:33:08.829468 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5618f433-dfcd-4f4f-bb7a-5f2306b5874c-workload-certs\") pod \"e2e-mt-ebe935a0-openshift-default-6bbfc74bf5-dkgll\" (UID: \"5618f433-dfcd-4f4f-bb7a-5f2306b5874c\") " pod="openshift-ingress/e2e-mt-ebe935a0-openshift-default-6bbfc74bf5-dkgll" Jul 21 21:33:08.930693 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:33:08.930659 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5618f433-dfcd-4f4f-bb7a-5f2306b5874c-istio-token\") pod \"e2e-mt-ebe935a0-openshift-default-6bbfc74bf5-dkgll\" (UID: \"5618f433-dfcd-4f4f-bb7a-5f2306b5874c\") " pod="openshift-ingress/e2e-mt-ebe935a0-openshift-default-6bbfc74bf5-dkgll" Jul 21 21:33:08.930693 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:33:08.930694 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5618f433-dfcd-4f4f-bb7a-5f2306b5874c-istio-data\") pod \"e2e-mt-ebe935a0-openshift-default-6bbfc74bf5-dkgll\" (UID: \"5618f433-dfcd-4f4f-bb7a-5f2306b5874c\") " pod="openshift-ingress/e2e-mt-ebe935a0-openshift-default-6bbfc74bf5-dkgll" Jul 21 21:33:08.930927 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:33:08.930714 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5618f433-dfcd-4f4f-bb7a-5f2306b5874c-istiod-ca-cert\") pod \"e2e-mt-ebe935a0-openshift-default-6bbfc74bf5-dkgll\" (UID: \"5618f433-dfcd-4f4f-bb7a-5f2306b5874c\") " pod="openshift-ingress/e2e-mt-ebe935a0-openshift-default-6bbfc74bf5-dkgll" Jul 21 21:33:08.930927 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:33:08.930739 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n7zh6\" (UniqueName: \"kubernetes.io/projected/5618f433-dfcd-4f4f-bb7a-5f2306b5874c-kube-api-access-n7zh6\") pod \"e2e-mt-ebe935a0-openshift-default-6bbfc74bf5-dkgll\" (UID: \"5618f433-dfcd-4f4f-bb7a-5f2306b5874c\") " pod="openshift-ingress/e2e-mt-ebe935a0-openshift-default-6bbfc74bf5-dkgll" Jul 21 21:33:08.930927 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:33:08.930766 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5618f433-dfcd-4f4f-bb7a-5f2306b5874c-istio-envoy\") pod \"e2e-mt-ebe935a0-openshift-default-6bbfc74bf5-dkgll\" (UID: \"5618f433-dfcd-4f4f-bb7a-5f2306b5874c\") " pod="openshift-ingress/e2e-mt-ebe935a0-openshift-default-6bbfc74bf5-dkgll" Jul 21 21:33:08.930927 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:33:08.930805 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5618f433-dfcd-4f4f-bb7a-5f2306b5874c-workload-socket\") pod \"e2e-mt-ebe935a0-openshift-default-6bbfc74bf5-dkgll\" (UID: \"5618f433-dfcd-4f4f-bb7a-5f2306b5874c\") " pod="openshift-ingress/e2e-mt-ebe935a0-openshift-default-6bbfc74bf5-dkgll" Jul 21 21:33:08.930927 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:33:08.930830 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5618f433-dfcd-4f4f-bb7a-5f2306b5874c-istio-podinfo\") pod \"e2e-mt-ebe935a0-openshift-default-6bbfc74bf5-dkgll\" (UID: \"5618f433-dfcd-4f4f-bb7a-5f2306b5874c\") " pod="openshift-ingress/e2e-mt-ebe935a0-openshift-default-6bbfc74bf5-dkgll" Jul 21 21:33:08.930927 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:33:08.930855 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5618f433-dfcd-4f4f-bb7a-5f2306b5874c-credential-socket\") pod \"e2e-mt-ebe935a0-openshift-default-6bbfc74bf5-dkgll\" (UID: \"5618f433-dfcd-4f4f-bb7a-5f2306b5874c\") " pod="openshift-ingress/e2e-mt-ebe935a0-openshift-default-6bbfc74bf5-dkgll" Jul 21 21:33:08.930927 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:33:08.930910 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5618f433-dfcd-4f4f-bb7a-5f2306b5874c-workload-certs\") pod \"e2e-mt-ebe935a0-openshift-default-6bbfc74bf5-dkgll\" (UID: \"5618f433-dfcd-4f4f-bb7a-5f2306b5874c\") " pod="openshift-ingress/e2e-mt-ebe935a0-openshift-default-6bbfc74bf5-dkgll" Jul 21 21:33:08.931212 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:33:08.931139 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5618f433-dfcd-4f4f-bb7a-5f2306b5874c-workload-socket\") pod \"e2e-mt-ebe935a0-openshift-default-6bbfc74bf5-dkgll\" (UID: \"5618f433-dfcd-4f4f-bb7a-5f2306b5874c\") " pod="openshift-ingress/e2e-mt-ebe935a0-openshift-default-6bbfc74bf5-dkgll" Jul 21 21:33:08.931248 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:33:08.931206 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5618f433-dfcd-4f4f-bb7a-5f2306b5874c-workload-certs\") pod \"e2e-mt-ebe935a0-openshift-default-6bbfc74bf5-dkgll\" (UID: \"5618f433-dfcd-4f4f-bb7a-5f2306b5874c\") " pod="openshift-ingress/e2e-mt-ebe935a0-openshift-default-6bbfc74bf5-dkgll" Jul 21 21:33:08.931343 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:33:08.931325 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5618f433-dfcd-4f4f-bb7a-5f2306b5874c-credential-socket\") pod \"e2e-mt-ebe935a0-openshift-default-6bbfc74bf5-dkgll\" (UID: \"5618f433-dfcd-4f4f-bb7a-5f2306b5874c\") " pod="openshift-ingress/e2e-mt-ebe935a0-openshift-default-6bbfc74bf5-dkgll" Jul 21 21:33:08.931398 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:33:08.931362 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5618f433-dfcd-4f4f-bb7a-5f2306b5874c-istio-data\") pod \"e2e-mt-ebe935a0-openshift-default-6bbfc74bf5-dkgll\" (UID: \"5618f433-dfcd-4f4f-bb7a-5f2306b5874c\") " pod="openshift-ingress/e2e-mt-ebe935a0-openshift-default-6bbfc74bf5-dkgll" Jul 21 21:33:08.931437 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:33:08.931428 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5618f433-dfcd-4f4f-bb7a-5f2306b5874c-istiod-ca-cert\") pod \"e2e-mt-ebe935a0-openshift-default-6bbfc74bf5-dkgll\" (UID: \"5618f433-dfcd-4f4f-bb7a-5f2306b5874c\") " pod="openshift-ingress/e2e-mt-ebe935a0-openshift-default-6bbfc74bf5-dkgll" Jul 21 21:33:08.933193 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:33:08.933171 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5618f433-dfcd-4f4f-bb7a-5f2306b5874c-istio-envoy\") pod \"e2e-mt-ebe935a0-openshift-default-6bbfc74bf5-dkgll\" (UID: \"5618f433-dfcd-4f4f-bb7a-5f2306b5874c\") " pod="openshift-ingress/e2e-mt-ebe935a0-openshift-default-6bbfc74bf5-dkgll" Jul 21 21:33:08.933327 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:33:08.933311 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5618f433-dfcd-4f4f-bb7a-5f2306b5874c-istio-podinfo\") pod \"e2e-mt-ebe935a0-openshift-default-6bbfc74bf5-dkgll\" (UID: \"5618f433-dfcd-4f4f-bb7a-5f2306b5874c\") " pod="openshift-ingress/e2e-mt-ebe935a0-openshift-default-6bbfc74bf5-dkgll" Jul 21 21:33:08.944270 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:33:08.944246 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5618f433-dfcd-4f4f-bb7a-5f2306b5874c-istio-token\") pod \"e2e-mt-ebe935a0-openshift-default-6bbfc74bf5-dkgll\" (UID: \"5618f433-dfcd-4f4f-bb7a-5f2306b5874c\") " pod="openshift-ingress/e2e-mt-ebe935a0-openshift-default-6bbfc74bf5-dkgll" Jul 21 21:33:08.944366 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:33:08.944344 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n7zh6\" (UniqueName: \"kubernetes.io/projected/5618f433-dfcd-4f4f-bb7a-5f2306b5874c-kube-api-access-n7zh6\") pod \"e2e-mt-ebe935a0-openshift-default-6bbfc74bf5-dkgll\" (UID: \"5618f433-dfcd-4f4f-bb7a-5f2306b5874c\") " pod="openshift-ingress/e2e-mt-ebe935a0-openshift-default-6bbfc74bf5-dkgll" Jul 21 21:33:09.054504 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:33:09.054430 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-ebe935a0-openshift-default-6bbfc74bf5-dkgll" Jul 21 21:33:09.207154 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:33:09.207124 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-ebe935a0-openshift-default-6bbfc74bf5-dkgll"] Jul 21 21:33:09.208249 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:33:09.208226 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5618f433_dfcd_4f4f_bb7a_5f2306b5874c.slice/crio-9701b38f9272c8b619a19f29473a67c81fd9f82edc7bd33871151ce68716ee92 WatchSource:0}: Error finding container 9701b38f9272c8b619a19f29473a67c81fd9f82edc7bd33871151ce68716ee92: Status 404 returned error can't find the container with id 9701b38f9272c8b619a19f29473a67c81fd9f82edc7bd33871151ce68716ee92 Jul 21 21:33:09.210463 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:33:09.210431 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 21 21:33:09.210576 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:33:09.210495 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 21 21:33:09.210576 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:33:09.210522 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 21 21:33:09.905681 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:33:09.905640 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-ebe935a0-openshift-default-6bbfc74bf5-dkgll" event={"ID":"5618f433-dfcd-4f4f-bb7a-5f2306b5874c","Type":"ContainerStarted","Data":"25409a0349ed1c6888b74c6c282e1f22201bd22a6dfbd0730724844f7b69931b"} Jul 21 21:33:09.906069 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:33:09.905742 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-ebe935a0-openshift-default-6bbfc74bf5-dkgll" event={"ID":"5618f433-dfcd-4f4f-bb7a-5f2306b5874c","Type":"ContainerStarted","Data":"9701b38f9272c8b619a19f29473a67c81fd9f82edc7bd33871151ce68716ee92"} Jul 21 21:33:09.937287 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:33:09.937243 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-ebe935a0-openshift-default-6bbfc74bf5-dkgll" podStartSLOduration=1.937227784 podStartE2EDuration="1.937227784s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 21:33:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 21:33:09.934851886 +0000 UTC m=+1947.869601574" watchObservedRunningTime="2026-07-21 21:33:09.937227784 +0000 UTC m=+1947.871977450" Jul 21 21:33:10.054994 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:33:10.054962 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-ebe935a0-openshift-default-6bbfc74bf5-dkgll" Jul 21 21:33:10.064333 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:33:10.064306 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-ebe935a0-openshift-default-6bbfc74bf5-dkgll" Jul 21 21:33:10.911472 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:33:10.911432 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-ebe935a0-openshift-default-6bbfc74bf5-dkgll" Jul 21 21:33:10.912282 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:33:10.912265 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-ebe935a0-openshift-default-6bbfc74bf5-dkgll" Jul 21 21:33:26.440715 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:33:26.440331 2575 patch_prober.go:28] interesting pod/e2e-mt-ebe935a0-openshift-default-6bbfc74bf5-dkgll container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.53:15021/healthz/ready\": read tcp 10.134.0.2:46206->10.134.0.53:15021: read: connection reset by peer" start-of-body= Jul 21 21:33:26.440715 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:33:26.440399 2575 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-ebe935a0-openshift-default-6bbfc74bf5-dkgll" podUID="5618f433-dfcd-4f4f-bb7a-5f2306b5874c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.53:15021/healthz/ready\": read tcp 10.134.0.2:46206->10.134.0.53:15021: read: connection reset by peer" Jul 21 21:33:26.994999 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:33:26.994965 2575 generic.go:356] "Generic (PLEG): container finished" podID="5618f433-dfcd-4f4f-bb7a-5f2306b5874c" containerID="25409a0349ed1c6888b74c6c282e1f22201bd22a6dfbd0730724844f7b69931b" exitCode=0 Jul 21 21:33:26.995168 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:33:26.995037 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-ebe935a0-openshift-default-6bbfc74bf5-dkgll" event={"ID":"5618f433-dfcd-4f4f-bb7a-5f2306b5874c","Type":"ContainerDied","Data":"25409a0349ed1c6888b74c6c282e1f22201bd22a6dfbd0730724844f7b69931b"} Jul 21 21:33:26.995450 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:33:26.995436 2575 scope.go:117] "RemoveContainer" containerID="25409a0349ed1c6888b74c6c282e1f22201bd22a6dfbd0730724844f7b69931b" Jul 21 21:33:26.996447 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:33:26.996418 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 21 21:33:26.996562 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:33:26.996500 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 21 21:33:26.996632 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:33:26.996573 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 21 21:33:28.002371 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:33:28.002332 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-ebe935a0-openshift-default-6bbfc74bf5-dkgll" event={"ID":"5618f433-dfcd-4f4f-bb7a-5f2306b5874c","Type":"ContainerStarted","Data":"7ed79733206cb7c81715e7e8e8760ecfcdbb3e4439a034a748b49acee2cf2aac"} Jul 21 21:33:28.055241 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:33:28.055198 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-ebe935a0-openshift-default-6bbfc74bf5-dkgll" Jul 21 21:33:28.055637 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:33:28.055606 2575 patch_prober.go:28] interesting pod/e2e-mt-ebe935a0-openshift-default-6bbfc74bf5-dkgll container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" start-of-body= Jul 21 21:33:28.055762 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:33:28.055666 2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-ebe935a0-openshift-default-6bbfc74bf5-dkgll" podUID="5618f433-dfcd-4f4f-bb7a-5f2306b5874c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" Jul 21 21:33:29.055644 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:33:29.055603 2575 patch_prober.go:28] interesting pod/e2e-mt-ebe935a0-openshift-default-6bbfc74bf5-dkgll container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" start-of-body= Jul 21 21:33:29.056216 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:33:29.055669 2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-ebe935a0-openshift-default-6bbfc74bf5-dkgll" podUID="5618f433-dfcd-4f4f-bb7a-5f2306b5874c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" Jul 21 21:33:30.060004 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:33:30.059962 2575 patch_prober.go:28] interesting pod/e2e-mt-ebe935a0-openshift-default-6bbfc74bf5-dkgll container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" start-of-body= Jul 21 21:33:30.060589 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:33:30.060554 2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-ebe935a0-openshift-default-6bbfc74bf5-dkgll" podUID="5618f433-dfcd-4f4f-bb7a-5f2306b5874c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" Jul 21 21:33:31.055291 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:33:31.055239 2575 patch_prober.go:28] interesting pod/e2e-mt-ebe935a0-openshift-default-6bbfc74bf5-dkgll container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" start-of-body= Jul 21 21:33:31.055499 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:33:31.055307 2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-ebe935a0-openshift-default-6bbfc74bf5-dkgll" podUID="5618f433-dfcd-4f4f-bb7a-5f2306b5874c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" Jul 21 21:33:32.058352 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:33:32.058324 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-ebe935a0-openshift-default-6bbfc74bf5-dkgll" Jul 21 21:33:32.058847 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:33:32.058708 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-ebe935a0-openshift-default-6bbfc74bf5-dkgll" Jul 21 21:33:32.059359 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:33:32.059343 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-ebe935a0-openshift-default-6bbfc74bf5-dkgll" Jul 21 21:33:49.979964 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:33:49.979925 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ebe935a0-bg222"] Jul 21 21:33:49.980608 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:33:49.980588 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="be110a31-e791-4de0-889a-a9845df47fa4" containerName="istio-proxy" Jul 21 21:33:49.982611 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:33:49.982591 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ebe935a0-bg222" Jul 21 21:33:49.985229 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:33:49.985199 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 21 21:33:49.985332 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:33:49.985236 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9fxhr\"" Jul 21 21:33:49.990647 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:33:49.990624 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ebe935a0-bg222"] Jul 21 21:33:50.124842 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:33:50.124807 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f0e61151-2498-475d-b877-bfde3498ddad-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-ebe935a0-bg222\" (UID: \"f0e61151-2498-475d-b877-bfde3498ddad\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ebe935a0-bg222" Jul 21 21:33:50.225641 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:33:50.225605 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f0e61151-2498-475d-b877-bfde3498ddad-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-ebe935a0-bg222\" (UID: \"f0e61151-2498-475d-b877-bfde3498ddad\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ebe935a0-bg222" Jul 21 21:33:50.226223 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:33:50.226203 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f0e61151-2498-475d-b877-bfde3498ddad-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-ebe935a0-bg222\" (UID: \"f0e61151-2498-475d-b877-bfde3498ddad\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ebe935a0-bg222" Jul 21 21:33:50.294409 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:33:50.294334 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ebe935a0-bg222" Jul 21 21:33:50.420700 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:33:50.420660 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ebe935a0-bg222"] Jul 21 21:33:50.422454 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:33:50.422426 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf0e61151_2498_475d_b877_bfde3498ddad.slice/crio-e0bc7a8bd76301d1b257a083edc8928b7fe617f01637a6486a0187fa2a6ac74c WatchSource:0}: Error finding container e0bc7a8bd76301d1b257a083edc8928b7fe617f01637a6486a0187fa2a6ac74c: Status 404 returned error can't find the container with id e0bc7a8bd76301d1b257a083edc8928b7fe617f01637a6486a0187fa2a6ac74c Jul 21 21:33:51.128728 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:33:51.128691 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ebe935a0-bg222" event={"ID":"f0e61151-2498-475d-b877-bfde3498ddad","Type":"ContainerStarted","Data":"0ca95462f3794f500fab6414e73ee23af4be57fb468300187a0ac19ee010e348"} Jul 21 21:33:51.129113 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:33:51.128742 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ebe935a0-bg222" event={"ID":"f0e61151-2498-475d-b877-bfde3498ddad","Type":"ContainerStarted","Data":"e0bc7a8bd76301d1b257a083edc8928b7fe617f01637a6486a0187fa2a6ac74c"} Jul 21 21:33:51.148782 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:33:51.148730 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ebe935a0-bg222" podStartSLOduration=2.148715521 podStartE2EDuration="2.148715521s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 21:33:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 21:33:51.147252681 +0000 UTC m=+1989.082002348" watchObservedRunningTime="2026-07-21 21:33:51.148715521 +0000 UTC m=+1989.083465192" Jul 21 21:33:52.135183 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:33:52.135152 2575 generic.go:356] "Generic (PLEG): container finished" podID="f0e61151-2498-475d-b877-bfde3498ddad" containerID="0ca95462f3794f500fab6414e73ee23af4be57fb468300187a0ac19ee010e348" exitCode=0 Jul 21 21:33:52.135605 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:33:52.135206 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ebe935a0-bg222" event={"ID":"f0e61151-2498-475d-b877-bfde3498ddad","Type":"ContainerDied","Data":"0ca95462f3794f500fab6414e73ee23af4be57fb468300187a0ac19ee010e348"} Jul 21 21:33:53.273986 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:33:53.273960 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ebe935a0-bg222" Jul 21 21:33:53.356880 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:33:53.356848 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f0e61151-2498-475d-b877-bfde3498ddad-maas-api-service-ca\") pod \"f0e61151-2498-475d-b877-bfde3498ddad\" (UID: \"f0e61151-2498-475d-b877-bfde3498ddad\") " Jul 21 21:33:53.357181 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:33:53.357157 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f0e61151-2498-475d-b877-bfde3498ddad-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "f0e61151-2498-475d-b877-bfde3498ddad" (UID: "f0e61151-2498-475d-b877-bfde3498ddad"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 21 21:33:53.457642 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:33:53.457570 2575 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f0e61151-2498-475d-b877-bfde3498ddad-maas-api-service-ca\") on node \"ip-10-0-140-97.ec2.internal\" DevicePath \"\"" Jul 21 21:33:54.147796 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:33:54.147769 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ebe935a0-bg222" Jul 21 21:33:54.147965 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:33:54.147768 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ebe935a0-bg222" event={"ID":"f0e61151-2498-475d-b877-bfde3498ddad","Type":"ContainerDied","Data":"e0bc7a8bd76301d1b257a083edc8928b7fe617f01637a6486a0187fa2a6ac74c"} Jul 21 21:33:54.147965 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:33:54.147891 2575 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e0bc7a8bd76301d1b257a083edc8928b7fe617f01637a6486a0187fa2a6ac74c" Jul 21 21:34:10.064694 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:34:10.064654 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ebe935a0-bg222"] Jul 21 21:34:10.072888 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:34:10.072855 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ebe935a0-bg222"] Jul 21 21:34:10.741609 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:34:10.741577 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f0e61151-2498-475d-b877-bfde3498ddad" path="/var/lib/kubelet/pods/f0e61151-2498-475d-b877-bfde3498ddad/volumes" Jul 21 21:34:42.676874 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:34:42.676844 2575 scope.go:117] "RemoveContainer" containerID="5117a60da70177226899288dd54516d9118e8eddf919b20878785996c74010d1" Jul 21 21:35:00.799635 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:35:00.799598 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-ebe935a0-openshift-default-6bbfc74bf5-dkgll"] Jul 21 21:35:00.800122 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:35:00.799850 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-ebe935a0-openshift-default-6bbfc74bf5-dkgll" podUID="5618f433-dfcd-4f4f-bb7a-5f2306b5874c" containerName="istio-proxy" containerID="cri-o://7ed79733206cb7c81715e7e8e8760ecfcdbb3e4439a034a748b49acee2cf2aac" gracePeriod=30 Jul 21 21:35:02.060812 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:35:02.060776 2575 patch_prober.go:28] interesting pod/e2e-mt-ebe935a0-openshift-default-6bbfc74bf5-dkgll container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 21 21:35:02.061211 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:35:02.060885 2575 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-ebe935a0-openshift-default-6bbfc74bf5-dkgll" podUID="5618f433-dfcd-4f4f-bb7a-5f2306b5874c" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 21 21:35:05.852887 ip-10-0-140-97 kubenswrapper[2575]: E0721 21:35:05.852847 2575 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5618f433_dfcd_4f4f_bb7a_5f2306b5874c.slice/crio-conmon-7ed79733206cb7c81715e7e8e8760ecfcdbb3e4439a034a748b49acee2cf2aac.scope\": RecentStats: unable to find data in memory cache]" Jul 21 21:35:06.052041 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:35:06.052017 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-ebe935a0-openshift-default-6bbfc74bf5-dkgll" Jul 21 21:35:06.135853 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:35:06.135821 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5618f433-dfcd-4f4f-bb7a-5f2306b5874c-istio-envoy\") pod \"5618f433-dfcd-4f4f-bb7a-5f2306b5874c\" (UID: \"5618f433-dfcd-4f4f-bb7a-5f2306b5874c\") " Jul 21 21:35:06.136027 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:35:06.135877 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5618f433-dfcd-4f4f-bb7a-5f2306b5874c-istio-data\") pod \"5618f433-dfcd-4f4f-bb7a-5f2306b5874c\" (UID: \"5618f433-dfcd-4f4f-bb7a-5f2306b5874c\") " Jul 21 21:35:06.136027 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:35:06.135955 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5618f433-dfcd-4f4f-bb7a-5f2306b5874c-istio-podinfo\") pod \"5618f433-dfcd-4f4f-bb7a-5f2306b5874c\" (UID: \"5618f433-dfcd-4f4f-bb7a-5f2306b5874c\") " Jul 21 21:35:06.136027 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:35:06.135998 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5618f433-dfcd-4f4f-bb7a-5f2306b5874c-istiod-ca-cert\") pod \"5618f433-dfcd-4f4f-bb7a-5f2306b5874c\" (UID: \"5618f433-dfcd-4f4f-bb7a-5f2306b5874c\") " Jul 21 21:35:06.136195 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:35:06.136060 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5618f433-dfcd-4f4f-bb7a-5f2306b5874c-workload-certs\") pod \"5618f433-dfcd-4f4f-bb7a-5f2306b5874c\" (UID: \"5618f433-dfcd-4f4f-bb7a-5f2306b5874c\") " Jul 21 21:35:06.136195 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:35:06.136086 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5618f433-dfcd-4f4f-bb7a-5f2306b5874c-credential-socket\") pod \"5618f433-dfcd-4f4f-bb7a-5f2306b5874c\" (UID: \"5618f433-dfcd-4f4f-bb7a-5f2306b5874c\") " Jul 21 21:35:06.136195 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:35:06.136124 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-n7zh6\" (UniqueName: \"kubernetes.io/projected/5618f433-dfcd-4f4f-bb7a-5f2306b5874c-kube-api-access-n7zh6\") pod \"5618f433-dfcd-4f4f-bb7a-5f2306b5874c\" (UID: \"5618f433-dfcd-4f4f-bb7a-5f2306b5874c\") " Jul 21 21:35:06.136195 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:35:06.136164 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5618f433-dfcd-4f4f-bb7a-5f2306b5874c-workload-socket\") pod \"5618f433-dfcd-4f4f-bb7a-5f2306b5874c\" (UID: \"5618f433-dfcd-4f4f-bb7a-5f2306b5874c\") " Jul 21 21:35:06.136393 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:35:06.136193 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5618f433-dfcd-4f4f-bb7a-5f2306b5874c-istio-token\") pod \"5618f433-dfcd-4f4f-bb7a-5f2306b5874c\" (UID: \"5618f433-dfcd-4f4f-bb7a-5f2306b5874c\") " Jul 21 21:35:06.136393 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:35:06.136335 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5618f433-dfcd-4f4f-bb7a-5f2306b5874c-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "5618f433-dfcd-4f4f-bb7a-5f2306b5874c" (UID: "5618f433-dfcd-4f4f-bb7a-5f2306b5874c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 21:35:06.136393 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:35:06.136357 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5618f433-dfcd-4f4f-bb7a-5f2306b5874c-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "5618f433-dfcd-4f4f-bb7a-5f2306b5874c" (UID: "5618f433-dfcd-4f4f-bb7a-5f2306b5874c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 21:35:06.136584 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:35:06.136424 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5618f433-dfcd-4f4f-bb7a-5f2306b5874c-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "5618f433-dfcd-4f4f-bb7a-5f2306b5874c" (UID: "5618f433-dfcd-4f4f-bb7a-5f2306b5874c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 21 21:35:06.136649 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:35:06.136584 2575 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5618f433-dfcd-4f4f-bb7a-5f2306b5874c-istiod-ca-cert\") on node \"ip-10-0-140-97.ec2.internal\" DevicePath \"\"" Jul 21 21:35:06.136704 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:35:06.136690 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5618f433-dfcd-4f4f-bb7a-5f2306b5874c-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "5618f433-dfcd-4f4f-bb7a-5f2306b5874c" (UID: "5618f433-dfcd-4f4f-bb7a-5f2306b5874c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 21:35:06.136774 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:35:06.136702 2575 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5618f433-dfcd-4f4f-bb7a-5f2306b5874c-workload-certs\") on node \"ip-10-0-140-97.ec2.internal\" DevicePath \"\"" Jul 21 21:35:06.136774 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:35:06.136718 2575 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5618f433-dfcd-4f4f-bb7a-5f2306b5874c-credential-socket\") on node \"ip-10-0-140-97.ec2.internal\" DevicePath \"\"" Jul 21 21:35:06.136954 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:35:06.136921 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5618f433-dfcd-4f4f-bb7a-5f2306b5874c-istio-data" (OuterVolumeSpecName: "istio-data") pod "5618f433-dfcd-4f4f-bb7a-5f2306b5874c" (UID: "5618f433-dfcd-4f4f-bb7a-5f2306b5874c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 21:35:06.138370 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:35:06.138340 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5618f433-dfcd-4f4f-bb7a-5f2306b5874c-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "5618f433-dfcd-4f4f-bb7a-5f2306b5874c" (UID: "5618f433-dfcd-4f4f-bb7a-5f2306b5874c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 21:35:06.138640 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:35:06.138614 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5618f433-dfcd-4f4f-bb7a-5f2306b5874c-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "5618f433-dfcd-4f4f-bb7a-5f2306b5874c" (UID: "5618f433-dfcd-4f4f-bb7a-5f2306b5874c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 21 21:35:06.138757 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:35:06.138694 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5618f433-dfcd-4f4f-bb7a-5f2306b5874c-kube-api-access-n7zh6" (OuterVolumeSpecName: "kube-api-access-n7zh6") pod "5618f433-dfcd-4f4f-bb7a-5f2306b5874c" (UID: "5618f433-dfcd-4f4f-bb7a-5f2306b5874c"). InnerVolumeSpecName "kube-api-access-n7zh6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 21 21:35:06.138757 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:35:06.138702 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5618f433-dfcd-4f4f-bb7a-5f2306b5874c-istio-token" (OuterVolumeSpecName: "istio-token") pod "5618f433-dfcd-4f4f-bb7a-5f2306b5874c" (UID: "5618f433-dfcd-4f4f-bb7a-5f2306b5874c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 21 21:35:06.237796 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:35:06.237765 2575 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5618f433-dfcd-4f4f-bb7a-5f2306b5874c-workload-socket\") on node \"ip-10-0-140-97.ec2.internal\" DevicePath \"\"" Jul 21 21:35:06.237796 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:35:06.237792 2575 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5618f433-dfcd-4f4f-bb7a-5f2306b5874c-istio-token\") on node \"ip-10-0-140-97.ec2.internal\" DevicePath \"\"" Jul 21 21:35:06.237796 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:35:06.237801 2575 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5618f433-dfcd-4f4f-bb7a-5f2306b5874c-istio-envoy\") on node \"ip-10-0-140-97.ec2.internal\" DevicePath \"\"" Jul 21 21:35:06.238012 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:35:06.237811 2575 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5618f433-dfcd-4f4f-bb7a-5f2306b5874c-istio-data\") on node \"ip-10-0-140-97.ec2.internal\" DevicePath \"\"" Jul 21 21:35:06.238012 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:35:06.237820 2575 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5618f433-dfcd-4f4f-bb7a-5f2306b5874c-istio-podinfo\") on node \"ip-10-0-140-97.ec2.internal\" DevicePath \"\"" Jul 21 21:35:06.238012 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:35:06.237828 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n7zh6\" (UniqueName: \"kubernetes.io/projected/5618f433-dfcd-4f4f-bb7a-5f2306b5874c-kube-api-access-n7zh6\") on node \"ip-10-0-140-97.ec2.internal\" DevicePath \"\"" Jul 21 21:35:06.515961 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:35:06.515876 2575 generic.go:356] "Generic (PLEG): container finished" podID="5618f433-dfcd-4f4f-bb7a-5f2306b5874c" containerID="7ed79733206cb7c81715e7e8e8760ecfcdbb3e4439a034a748b49acee2cf2aac" exitCode=0 Jul 21 21:35:06.515961 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:35:06.515916 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-ebe935a0-openshift-default-6bbfc74bf5-dkgll" Jul 21 21:35:06.516148 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:35:06.515959 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-ebe935a0-openshift-default-6bbfc74bf5-dkgll" event={"ID":"5618f433-dfcd-4f4f-bb7a-5f2306b5874c","Type":"ContainerDied","Data":"7ed79733206cb7c81715e7e8e8760ecfcdbb3e4439a034a748b49acee2cf2aac"} Jul 21 21:35:06.516148 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:35:06.516012 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-ebe935a0-openshift-default-6bbfc74bf5-dkgll" event={"ID":"5618f433-dfcd-4f4f-bb7a-5f2306b5874c","Type":"ContainerDied","Data":"9701b38f9272c8b619a19f29473a67c81fd9f82edc7bd33871151ce68716ee92"} Jul 21 21:35:06.516148 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:35:06.516035 2575 scope.go:117] "RemoveContainer" containerID="7ed79733206cb7c81715e7e8e8760ecfcdbb3e4439a034a748b49acee2cf2aac" Jul 21 21:35:06.525140 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:35:06.525125 2575 scope.go:117] "RemoveContainer" containerID="25409a0349ed1c6888b74c6c282e1f22201bd22a6dfbd0730724844f7b69931b" Jul 21 21:35:06.532622 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:35:06.532605 2575 scope.go:117] "RemoveContainer" containerID="7ed79733206cb7c81715e7e8e8760ecfcdbb3e4439a034a748b49acee2cf2aac" Jul 21 21:35:06.532833 ip-10-0-140-97 kubenswrapper[2575]: E0721 21:35:06.532814 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7ed79733206cb7c81715e7e8e8760ecfcdbb3e4439a034a748b49acee2cf2aac\": container with ID starting with 7ed79733206cb7c81715e7e8e8760ecfcdbb3e4439a034a748b49acee2cf2aac not found: ID does not exist" containerID="7ed79733206cb7c81715e7e8e8760ecfcdbb3e4439a034a748b49acee2cf2aac" Jul 21 21:35:06.532876 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:35:06.532841 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7ed79733206cb7c81715e7e8e8760ecfcdbb3e4439a034a748b49acee2cf2aac"} err="failed to get container status \"7ed79733206cb7c81715e7e8e8760ecfcdbb3e4439a034a748b49acee2cf2aac\": rpc error: code = NotFound desc = could not find container \"7ed79733206cb7c81715e7e8e8760ecfcdbb3e4439a034a748b49acee2cf2aac\": container with ID starting with 7ed79733206cb7c81715e7e8e8760ecfcdbb3e4439a034a748b49acee2cf2aac not found: ID does not exist" Jul 21 21:35:06.532876 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:35:06.532857 2575 scope.go:117] "RemoveContainer" containerID="25409a0349ed1c6888b74c6c282e1f22201bd22a6dfbd0730724844f7b69931b" Jul 21 21:35:06.533045 ip-10-0-140-97 kubenswrapper[2575]: E0721 21:35:06.533030 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"25409a0349ed1c6888b74c6c282e1f22201bd22a6dfbd0730724844f7b69931b\": container with ID starting with 25409a0349ed1c6888b74c6c282e1f22201bd22a6dfbd0730724844f7b69931b not found: ID does not exist" containerID="25409a0349ed1c6888b74c6c282e1f22201bd22a6dfbd0730724844f7b69931b" Jul 21 21:35:06.533085 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:35:06.533049 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"25409a0349ed1c6888b74c6c282e1f22201bd22a6dfbd0730724844f7b69931b"} err="failed to get container status \"25409a0349ed1c6888b74c6c282e1f22201bd22a6dfbd0730724844f7b69931b\": rpc error: code = NotFound desc = could not find container \"25409a0349ed1c6888b74c6c282e1f22201bd22a6dfbd0730724844f7b69931b\": container with ID starting with 25409a0349ed1c6888b74c6c282e1f22201bd22a6dfbd0730724844f7b69931b not found: ID does not exist" Jul 21 21:35:06.541304 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:35:06.541282 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-ebe935a0-openshift-default-6bbfc74bf5-dkgll"] Jul 21 21:35:06.550883 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:35:06.550861 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-ebe935a0-openshift-default-6bbfc74bf5-dkgll"] Jul 21 21:35:06.741403 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:35:06.741363 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5618f433-dfcd-4f4f-bb7a-5f2306b5874c" path="/var/lib/kubelet/pods/5618f433-dfcd-4f4f-bb7a-5f2306b5874c/volumes" Jul 21 21:35:42.696715 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:35:42.696681 2575 scope.go:117] "RemoveContainer" containerID="1c1ba03bd8684301edec141a10685a2c46748d7c54feac68e388aab57d144ff2" Jul 21 21:35:42.706783 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:35:42.705199 2575 scope.go:117] "RemoveContainer" containerID="b21a009c6a7dd31933c7361e0b07f5d1842861b0b893972784e8e0c7782b951b" Jul 21 21:35:59.463989 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:35:59.463958 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-bacc91-openshift-default-6f644c796c-lns9p"] Jul 21 21:35:59.464433 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:35:59.464295 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5618f433-dfcd-4f4f-bb7a-5f2306b5874c" containerName="istio-proxy" Jul 21 21:35:59.464433 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:35:59.464306 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="5618f433-dfcd-4f4f-bb7a-5f2306b5874c" containerName="istio-proxy" Jul 21 21:35:59.464433 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:35:59.464316 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5618f433-dfcd-4f4f-bb7a-5f2306b5874c" containerName="istio-proxy" Jul 21 21:35:59.464433 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:35:59.464321 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="5618f433-dfcd-4f4f-bb7a-5f2306b5874c" containerName="istio-proxy" Jul 21 21:35:59.464433 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:35:59.464329 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f0e61151-2498-475d-b877-bfde3498ddad" containerName="revoke-keys" Jul 21 21:35:59.464433 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:35:59.464334 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="f0e61151-2498-475d-b877-bfde3498ddad" containerName="revoke-keys" Jul 21 21:35:59.464433 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:35:59.464393 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="5618f433-dfcd-4f4f-bb7a-5f2306b5874c" containerName="istio-proxy" Jul 21 21:35:59.464433 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:35:59.464402 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="5618f433-dfcd-4f4f-bb7a-5f2306b5874c" containerName="istio-proxy" Jul 21 21:35:59.464433 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:35:59.464410 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="f0e61151-2498-475d-b877-bfde3498ddad" containerName="revoke-keys" Jul 21 21:35:59.467514 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:35:59.467497 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-bacc91-openshift-default-6f644c796c-lns9p" Jul 21 21:35:59.470247 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:35:59.470225 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-bacc91-openshift-default-dockercfg-sh9gw\"" Jul 21 21:35:59.481266 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:35:59.481243 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-bacc91-openshift-default-6f644c796c-lns9p"] Jul 21 21:35:59.625941 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:35:59.625890 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/65aa8de5-9a34-4c5a-ade1-d0fa36ae1f19-workload-socket\") pod \"e2e-inf-a-bacc91-openshift-default-6f644c796c-lns9p\" (UID: \"65aa8de5-9a34-4c5a-ade1-d0fa36ae1f19\") " pod="openshift-ingress/e2e-inf-a-bacc91-openshift-default-6f644c796c-lns9p" Jul 21 21:35:59.625941 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:35:59.625948 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/65aa8de5-9a34-4c5a-ade1-d0fa36ae1f19-workload-certs\") pod \"e2e-inf-a-bacc91-openshift-default-6f644c796c-lns9p\" (UID: \"65aa8de5-9a34-4c5a-ade1-d0fa36ae1f19\") " pod="openshift-ingress/e2e-inf-a-bacc91-openshift-default-6f644c796c-lns9p" Jul 21 21:35:59.626155 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:35:59.625979 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/65aa8de5-9a34-4c5a-ade1-d0fa36ae1f19-istio-data\") pod \"e2e-inf-a-bacc91-openshift-default-6f644c796c-lns9p\" (UID: \"65aa8de5-9a34-4c5a-ade1-d0fa36ae1f19\") " pod="openshift-ingress/e2e-inf-a-bacc91-openshift-default-6f644c796c-lns9p" Jul 21 21:35:59.626155 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:35:59.626000 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x7t88\" (UniqueName: \"kubernetes.io/projected/65aa8de5-9a34-4c5a-ade1-d0fa36ae1f19-kube-api-access-x7t88\") pod \"e2e-inf-a-bacc91-openshift-default-6f644c796c-lns9p\" (UID: \"65aa8de5-9a34-4c5a-ade1-d0fa36ae1f19\") " pod="openshift-ingress/e2e-inf-a-bacc91-openshift-default-6f644c796c-lns9p" Jul 21 21:35:59.626155 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:35:59.626031 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/65aa8de5-9a34-4c5a-ade1-d0fa36ae1f19-istio-envoy\") pod \"e2e-inf-a-bacc91-openshift-default-6f644c796c-lns9p\" (UID: \"65aa8de5-9a34-4c5a-ade1-d0fa36ae1f19\") " pod="openshift-ingress/e2e-inf-a-bacc91-openshift-default-6f644c796c-lns9p" Jul 21 21:35:59.626155 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:35:59.626071 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/65aa8de5-9a34-4c5a-ade1-d0fa36ae1f19-credential-socket\") pod \"e2e-inf-a-bacc91-openshift-default-6f644c796c-lns9p\" (UID: \"65aa8de5-9a34-4c5a-ade1-d0fa36ae1f19\") " pod="openshift-ingress/e2e-inf-a-bacc91-openshift-default-6f644c796c-lns9p" Jul 21 21:35:59.626155 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:35:59.626087 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/65aa8de5-9a34-4c5a-ade1-d0fa36ae1f19-istiod-ca-cert\") pod \"e2e-inf-a-bacc91-openshift-default-6f644c796c-lns9p\" (UID: \"65aa8de5-9a34-4c5a-ade1-d0fa36ae1f19\") " pod="openshift-ingress/e2e-inf-a-bacc91-openshift-default-6f644c796c-lns9p" Jul 21 21:35:59.626358 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:35:59.626159 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/65aa8de5-9a34-4c5a-ade1-d0fa36ae1f19-istio-token\") pod \"e2e-inf-a-bacc91-openshift-default-6f644c796c-lns9p\" (UID: \"65aa8de5-9a34-4c5a-ade1-d0fa36ae1f19\") " pod="openshift-ingress/e2e-inf-a-bacc91-openshift-default-6f644c796c-lns9p" Jul 21 21:35:59.626358 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:35:59.626197 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/65aa8de5-9a34-4c5a-ade1-d0fa36ae1f19-istio-podinfo\") pod \"e2e-inf-a-bacc91-openshift-default-6f644c796c-lns9p\" (UID: \"65aa8de5-9a34-4c5a-ade1-d0fa36ae1f19\") " pod="openshift-ingress/e2e-inf-a-bacc91-openshift-default-6f644c796c-lns9p" Jul 21 21:35:59.727122 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:35:59.727038 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/65aa8de5-9a34-4c5a-ade1-d0fa36ae1f19-istio-token\") pod \"e2e-inf-a-bacc91-openshift-default-6f644c796c-lns9p\" (UID: \"65aa8de5-9a34-4c5a-ade1-d0fa36ae1f19\") " pod="openshift-ingress/e2e-inf-a-bacc91-openshift-default-6f644c796c-lns9p" Jul 21 21:35:59.727122 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:35:59.727083 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/65aa8de5-9a34-4c5a-ade1-d0fa36ae1f19-istio-podinfo\") pod \"e2e-inf-a-bacc91-openshift-default-6f644c796c-lns9p\" (UID: \"65aa8de5-9a34-4c5a-ade1-d0fa36ae1f19\") " pod="openshift-ingress/e2e-inf-a-bacc91-openshift-default-6f644c796c-lns9p" Jul 21 21:35:59.727311 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:35:59.727150 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/65aa8de5-9a34-4c5a-ade1-d0fa36ae1f19-workload-socket\") pod \"e2e-inf-a-bacc91-openshift-default-6f644c796c-lns9p\" (UID: \"65aa8de5-9a34-4c5a-ade1-d0fa36ae1f19\") " pod="openshift-ingress/e2e-inf-a-bacc91-openshift-default-6f644c796c-lns9p" Jul 21 21:35:59.727311 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:35:59.727183 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/65aa8de5-9a34-4c5a-ade1-d0fa36ae1f19-workload-certs\") pod \"e2e-inf-a-bacc91-openshift-default-6f644c796c-lns9p\" (UID: \"65aa8de5-9a34-4c5a-ade1-d0fa36ae1f19\") " pod="openshift-ingress/e2e-inf-a-bacc91-openshift-default-6f644c796c-lns9p" Jul 21 21:35:59.727311 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:35:59.727217 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/65aa8de5-9a34-4c5a-ade1-d0fa36ae1f19-istio-data\") pod \"e2e-inf-a-bacc91-openshift-default-6f644c796c-lns9p\" (UID: \"65aa8de5-9a34-4c5a-ade1-d0fa36ae1f19\") " pod="openshift-ingress/e2e-inf-a-bacc91-openshift-default-6f644c796c-lns9p" Jul 21 21:35:59.727311 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:35:59.727245 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x7t88\" (UniqueName: \"kubernetes.io/projected/65aa8de5-9a34-4c5a-ade1-d0fa36ae1f19-kube-api-access-x7t88\") pod \"e2e-inf-a-bacc91-openshift-default-6f644c796c-lns9p\" (UID: \"65aa8de5-9a34-4c5a-ade1-d0fa36ae1f19\") " pod="openshift-ingress/e2e-inf-a-bacc91-openshift-default-6f644c796c-lns9p" Jul 21 21:35:59.727311 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:35:59.727278 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/65aa8de5-9a34-4c5a-ade1-d0fa36ae1f19-istio-envoy\") pod \"e2e-inf-a-bacc91-openshift-default-6f644c796c-lns9p\" (UID: \"65aa8de5-9a34-4c5a-ade1-d0fa36ae1f19\") " pod="openshift-ingress/e2e-inf-a-bacc91-openshift-default-6f644c796c-lns9p" Jul 21 21:35:59.727573 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:35:59.727407 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/65aa8de5-9a34-4c5a-ade1-d0fa36ae1f19-credential-socket\") pod \"e2e-inf-a-bacc91-openshift-default-6f644c796c-lns9p\" (UID: \"65aa8de5-9a34-4c5a-ade1-d0fa36ae1f19\") " pod="openshift-ingress/e2e-inf-a-bacc91-openshift-default-6f644c796c-lns9p" Jul 21 21:35:59.727573 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:35:59.727457 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/65aa8de5-9a34-4c5a-ade1-d0fa36ae1f19-istiod-ca-cert\") pod \"e2e-inf-a-bacc91-openshift-default-6f644c796c-lns9p\" (UID: \"65aa8de5-9a34-4c5a-ade1-d0fa36ae1f19\") " pod="openshift-ingress/e2e-inf-a-bacc91-openshift-default-6f644c796c-lns9p" Jul 21 21:35:59.727573 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:35:59.727556 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/65aa8de5-9a34-4c5a-ade1-d0fa36ae1f19-workload-socket\") pod \"e2e-inf-a-bacc91-openshift-default-6f644c796c-lns9p\" (UID: \"65aa8de5-9a34-4c5a-ade1-d0fa36ae1f19\") " pod="openshift-ingress/e2e-inf-a-bacc91-openshift-default-6f644c796c-lns9p" Jul 21 21:35:59.728043 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:35:59.728013 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/65aa8de5-9a34-4c5a-ade1-d0fa36ae1f19-istio-data\") pod \"e2e-inf-a-bacc91-openshift-default-6f644c796c-lns9p\" (UID: \"65aa8de5-9a34-4c5a-ade1-d0fa36ae1f19\") " pod="openshift-ingress/e2e-inf-a-bacc91-openshift-default-6f644c796c-lns9p" Jul 21 21:35:59.728177 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:35:59.728022 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/65aa8de5-9a34-4c5a-ade1-d0fa36ae1f19-workload-certs\") pod \"e2e-inf-a-bacc91-openshift-default-6f644c796c-lns9p\" (UID: \"65aa8de5-9a34-4c5a-ade1-d0fa36ae1f19\") " pod="openshift-ingress/e2e-inf-a-bacc91-openshift-default-6f644c796c-lns9p" Jul 21 21:35:59.728177 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:35:59.728098 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/65aa8de5-9a34-4c5a-ade1-d0fa36ae1f19-credential-socket\") pod \"e2e-inf-a-bacc91-openshift-default-6f644c796c-lns9p\" (UID: \"65aa8de5-9a34-4c5a-ade1-d0fa36ae1f19\") " pod="openshift-ingress/e2e-inf-a-bacc91-openshift-default-6f644c796c-lns9p" Jul 21 21:35:59.728302 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:35:59.728186 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/65aa8de5-9a34-4c5a-ade1-d0fa36ae1f19-istiod-ca-cert\") pod \"e2e-inf-a-bacc91-openshift-default-6f644c796c-lns9p\" (UID: \"65aa8de5-9a34-4c5a-ade1-d0fa36ae1f19\") " pod="openshift-ingress/e2e-inf-a-bacc91-openshift-default-6f644c796c-lns9p" Jul 21 21:35:59.729681 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:35:59.729662 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/65aa8de5-9a34-4c5a-ade1-d0fa36ae1f19-istio-envoy\") pod \"e2e-inf-a-bacc91-openshift-default-6f644c796c-lns9p\" (UID: \"65aa8de5-9a34-4c5a-ade1-d0fa36ae1f19\") " pod="openshift-ingress/e2e-inf-a-bacc91-openshift-default-6f644c796c-lns9p" Jul 21 21:35:59.729777 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:35:59.729705 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/65aa8de5-9a34-4c5a-ade1-d0fa36ae1f19-istio-podinfo\") pod \"e2e-inf-a-bacc91-openshift-default-6f644c796c-lns9p\" (UID: \"65aa8de5-9a34-4c5a-ade1-d0fa36ae1f19\") " pod="openshift-ingress/e2e-inf-a-bacc91-openshift-default-6f644c796c-lns9p" Jul 21 21:35:59.736227 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:35:59.736203 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x7t88\" (UniqueName: \"kubernetes.io/projected/65aa8de5-9a34-4c5a-ade1-d0fa36ae1f19-kube-api-access-x7t88\") pod \"e2e-inf-a-bacc91-openshift-default-6f644c796c-lns9p\" (UID: \"65aa8de5-9a34-4c5a-ade1-d0fa36ae1f19\") " pod="openshift-ingress/e2e-inf-a-bacc91-openshift-default-6f644c796c-lns9p" Jul 21 21:35:59.736323 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:35:59.736225 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/65aa8de5-9a34-4c5a-ade1-d0fa36ae1f19-istio-token\") pod \"e2e-inf-a-bacc91-openshift-default-6f644c796c-lns9p\" (UID: \"65aa8de5-9a34-4c5a-ade1-d0fa36ae1f19\") " pod="openshift-ingress/e2e-inf-a-bacc91-openshift-default-6f644c796c-lns9p" Jul 21 21:35:59.776594 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:35:59.776570 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-bacc91-openshift-default-6f644c796c-lns9p" Jul 21 21:35:59.917906 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:35:59.917876 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-bacc91-openshift-default-6f644c796c-lns9p"] Jul 21 21:35:59.919761 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:35:59.919733 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod65aa8de5_9a34_4c5a_ade1_d0fa36ae1f19.slice/crio-17db0c7002294ba00873a077e1ec0d1eff9b9bbf749222f83941246539dfd3ce WatchSource:0}: Error finding container 17db0c7002294ba00873a077e1ec0d1eff9b9bbf749222f83941246539dfd3ce: Status 404 returned error can't find the container with id 17db0c7002294ba00873a077e1ec0d1eff9b9bbf749222f83941246539dfd3ce Jul 21 21:35:59.921837 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:35:59.921805 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 21 21:35:59.921933 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:35:59.921866 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 21 21:35:59.921933 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:35:59.921894 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 21 21:36:00.784842 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:36:00.784799 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-bacc91-openshift-default-6f644c796c-lns9p" event={"ID":"65aa8de5-9a34-4c5a-ade1-d0fa36ae1f19","Type":"ContainerStarted","Data":"561cc3063f47d5995f56807df316ac0a3dc23026ba22446c3b2f1f8d21c89e88"} Jul 21 21:36:00.785197 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:36:00.784871 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-bacc91-openshift-default-6f644c796c-lns9p" event={"ID":"65aa8de5-9a34-4c5a-ade1-d0fa36ae1f19","Type":"ContainerStarted","Data":"17db0c7002294ba00873a077e1ec0d1eff9b9bbf749222f83941246539dfd3ce"} Jul 21 21:36:00.811961 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:36:00.811917 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-bacc91-openshift-default-6f644c796c-lns9p" podStartSLOduration=1.811901948 podStartE2EDuration="1.811901948s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 21:35:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 21:36:00.809959003 +0000 UTC m=+2118.744708679" watchObservedRunningTime="2026-07-21 21:36:00.811901948 +0000 UTC m=+2118.746651612" Jul 21 21:36:01.777590 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:36:01.777552 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-bacc91-openshift-default-6f644c796c-lns9p" Jul 21 21:36:01.782733 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:36:01.782709 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-bacc91-openshift-default-6f644c796c-lns9p" Jul 21 21:36:01.795915 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:36:01.795888 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-bacc91-openshift-default-6f644c796c-lns9p" Jul 21 21:36:01.796812 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:36:01.796786 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-bacc91-openshift-default-6f644c796c-lns9p" Jul 21 21:36:07.965932 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:36:07.965898 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-9e03c3-openshift-default-749c85bd65-5hqkn"] Jul 21 21:36:07.981414 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:36:07.981380 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-9e03c3-openshift-default-749c85bd65-5hqkn" Jul 21 21:36:07.983379 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:36:07.983353 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-9e03c3-openshift-default-749c85bd65-5hqkn"] Jul 21 21:36:07.984629 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:36:07.984593 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-9e03c3-openshift-default-dockercfg-s4qn9\"" Jul 21 21:36:08.112910 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:36:08.112868 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d9e2c1bc-e15c-4f6e-a12b-4e1ef6acd86e-istio-envoy\") pod \"e2e-inf-b-9e03c3-openshift-default-749c85bd65-5hqkn\" (UID: \"d9e2c1bc-e15c-4f6e-a12b-4e1ef6acd86e\") " pod="openshift-ingress/e2e-inf-b-9e03c3-openshift-default-749c85bd65-5hqkn" Jul 21 21:36:08.113123 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:36:08.112954 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d9e2c1bc-e15c-4f6e-a12b-4e1ef6acd86e-istio-podinfo\") pod \"e2e-inf-b-9e03c3-openshift-default-749c85bd65-5hqkn\" (UID: \"d9e2c1bc-e15c-4f6e-a12b-4e1ef6acd86e\") " pod="openshift-ingress/e2e-inf-b-9e03c3-openshift-default-749c85bd65-5hqkn" Jul 21 21:36:08.113123 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:36:08.113050 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d9e2c1bc-e15c-4f6e-a12b-4e1ef6acd86e-istiod-ca-cert\") pod \"e2e-inf-b-9e03c3-openshift-default-749c85bd65-5hqkn\" (UID: \"d9e2c1bc-e15c-4f6e-a12b-4e1ef6acd86e\") " pod="openshift-ingress/e2e-inf-b-9e03c3-openshift-default-749c85bd65-5hqkn" Jul 21 21:36:08.113287 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:36:08.113124 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5kg5z\" (UniqueName: \"kubernetes.io/projected/d9e2c1bc-e15c-4f6e-a12b-4e1ef6acd86e-kube-api-access-5kg5z\") pod \"e2e-inf-b-9e03c3-openshift-default-749c85bd65-5hqkn\" (UID: \"d9e2c1bc-e15c-4f6e-a12b-4e1ef6acd86e\") " pod="openshift-ingress/e2e-inf-b-9e03c3-openshift-default-749c85bd65-5hqkn" Jul 21 21:36:08.113287 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:36:08.113163 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d9e2c1bc-e15c-4f6e-a12b-4e1ef6acd86e-istio-data\") pod \"e2e-inf-b-9e03c3-openshift-default-749c85bd65-5hqkn\" (UID: \"d9e2c1bc-e15c-4f6e-a12b-4e1ef6acd86e\") " pod="openshift-ingress/e2e-inf-b-9e03c3-openshift-default-749c85bd65-5hqkn" Jul 21 21:36:08.113287 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:36:08.113230 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9e2c1bc-e15c-4f6e-a12b-4e1ef6acd86e-workload-socket\") pod \"e2e-inf-b-9e03c3-openshift-default-749c85bd65-5hqkn\" (UID: \"d9e2c1bc-e15c-4f6e-a12b-4e1ef6acd86e\") " pod="openshift-ingress/e2e-inf-b-9e03c3-openshift-default-749c85bd65-5hqkn" Jul 21 21:36:08.113287 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:36:08.113259 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d9e2c1bc-e15c-4f6e-a12b-4e1ef6acd86e-workload-certs\") pod \"e2e-inf-b-9e03c3-openshift-default-749c85bd65-5hqkn\" (UID: \"d9e2c1bc-e15c-4f6e-a12b-4e1ef6acd86e\") " pod="openshift-ingress/e2e-inf-b-9e03c3-openshift-default-749c85bd65-5hqkn" Jul 21 21:36:08.113287 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:36:08.113288 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d9e2c1bc-e15c-4f6e-a12b-4e1ef6acd86e-istio-token\") pod \"e2e-inf-b-9e03c3-openshift-default-749c85bd65-5hqkn\" (UID: \"d9e2c1bc-e15c-4f6e-a12b-4e1ef6acd86e\") " pod="openshift-ingress/e2e-inf-b-9e03c3-openshift-default-749c85bd65-5hqkn" Jul 21 21:36:08.113482 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:36:08.113334 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9e2c1bc-e15c-4f6e-a12b-4e1ef6acd86e-credential-socket\") pod \"e2e-inf-b-9e03c3-openshift-default-749c85bd65-5hqkn\" (UID: \"d9e2c1bc-e15c-4f6e-a12b-4e1ef6acd86e\") " pod="openshift-ingress/e2e-inf-b-9e03c3-openshift-default-749c85bd65-5hqkn" Jul 21 21:36:08.213933 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:36:08.213902 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d9e2c1bc-e15c-4f6e-a12b-4e1ef6acd86e-istio-token\") pod \"e2e-inf-b-9e03c3-openshift-default-749c85bd65-5hqkn\" (UID: \"d9e2c1bc-e15c-4f6e-a12b-4e1ef6acd86e\") " pod="openshift-ingress/e2e-inf-b-9e03c3-openshift-default-749c85bd65-5hqkn" Jul 21 21:36:08.214113 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:36:08.213950 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9e2c1bc-e15c-4f6e-a12b-4e1ef6acd86e-credential-socket\") pod \"e2e-inf-b-9e03c3-openshift-default-749c85bd65-5hqkn\" (UID: \"d9e2c1bc-e15c-4f6e-a12b-4e1ef6acd86e\") " pod="openshift-ingress/e2e-inf-b-9e03c3-openshift-default-749c85bd65-5hqkn" Jul 21 21:36:08.214113 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:36:08.214026 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d9e2c1bc-e15c-4f6e-a12b-4e1ef6acd86e-istio-envoy\") pod \"e2e-inf-b-9e03c3-openshift-default-749c85bd65-5hqkn\" (UID: \"d9e2c1bc-e15c-4f6e-a12b-4e1ef6acd86e\") " pod="openshift-ingress/e2e-inf-b-9e03c3-openshift-default-749c85bd65-5hqkn" Jul 21 21:36:08.214113 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:36:08.214050 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d9e2c1bc-e15c-4f6e-a12b-4e1ef6acd86e-istio-podinfo\") pod \"e2e-inf-b-9e03c3-openshift-default-749c85bd65-5hqkn\" (UID: \"d9e2c1bc-e15c-4f6e-a12b-4e1ef6acd86e\") " pod="openshift-ingress/e2e-inf-b-9e03c3-openshift-default-749c85bd65-5hqkn" Jul 21 21:36:08.214113 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:36:08.214072 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d9e2c1bc-e15c-4f6e-a12b-4e1ef6acd86e-istiod-ca-cert\") pod \"e2e-inf-b-9e03c3-openshift-default-749c85bd65-5hqkn\" (UID: \"d9e2c1bc-e15c-4f6e-a12b-4e1ef6acd86e\") " pod="openshift-ingress/e2e-inf-b-9e03c3-openshift-default-749c85bd65-5hqkn" Jul 21 21:36:08.214113 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:36:08.214104 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5kg5z\" (UniqueName: \"kubernetes.io/projected/d9e2c1bc-e15c-4f6e-a12b-4e1ef6acd86e-kube-api-access-5kg5z\") pod \"e2e-inf-b-9e03c3-openshift-default-749c85bd65-5hqkn\" (UID: \"d9e2c1bc-e15c-4f6e-a12b-4e1ef6acd86e\") " pod="openshift-ingress/e2e-inf-b-9e03c3-openshift-default-749c85bd65-5hqkn" Jul 21 21:36:08.214374 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:36:08.214143 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d9e2c1bc-e15c-4f6e-a12b-4e1ef6acd86e-istio-data\") pod \"e2e-inf-b-9e03c3-openshift-default-749c85bd65-5hqkn\" (UID: \"d9e2c1bc-e15c-4f6e-a12b-4e1ef6acd86e\") " pod="openshift-ingress/e2e-inf-b-9e03c3-openshift-default-749c85bd65-5hqkn" Jul 21 21:36:08.214374 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:36:08.214175 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9e2c1bc-e15c-4f6e-a12b-4e1ef6acd86e-workload-socket\") pod \"e2e-inf-b-9e03c3-openshift-default-749c85bd65-5hqkn\" (UID: \"d9e2c1bc-e15c-4f6e-a12b-4e1ef6acd86e\") " pod="openshift-ingress/e2e-inf-b-9e03c3-openshift-default-749c85bd65-5hqkn" Jul 21 21:36:08.214374 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:36:08.214198 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d9e2c1bc-e15c-4f6e-a12b-4e1ef6acd86e-workload-certs\") pod \"e2e-inf-b-9e03c3-openshift-default-749c85bd65-5hqkn\" (UID: \"d9e2c1bc-e15c-4f6e-a12b-4e1ef6acd86e\") " pod="openshift-ingress/e2e-inf-b-9e03c3-openshift-default-749c85bd65-5hqkn" Jul 21 21:36:08.214559 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:36:08.214450 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9e2c1bc-e15c-4f6e-a12b-4e1ef6acd86e-credential-socket\") pod \"e2e-inf-b-9e03c3-openshift-default-749c85bd65-5hqkn\" (UID: \"d9e2c1bc-e15c-4f6e-a12b-4e1ef6acd86e\") " pod="openshift-ingress/e2e-inf-b-9e03c3-openshift-default-749c85bd65-5hqkn" Jul 21 21:36:08.214621 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:36:08.214558 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9e2c1bc-e15c-4f6e-a12b-4e1ef6acd86e-workload-socket\") pod \"e2e-inf-b-9e03c3-openshift-default-749c85bd65-5hqkn\" (UID: \"d9e2c1bc-e15c-4f6e-a12b-4e1ef6acd86e\") " pod="openshift-ingress/e2e-inf-b-9e03c3-openshift-default-749c85bd65-5hqkn" Jul 21 21:36:08.214738 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:36:08.214716 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d9e2c1bc-e15c-4f6e-a12b-4e1ef6acd86e-workload-certs\") pod \"e2e-inf-b-9e03c3-openshift-default-749c85bd65-5hqkn\" (UID: \"d9e2c1bc-e15c-4f6e-a12b-4e1ef6acd86e\") " pod="openshift-ingress/e2e-inf-b-9e03c3-openshift-default-749c85bd65-5hqkn" Jul 21 21:36:08.214944 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:36:08.214914 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d9e2c1bc-e15c-4f6e-a12b-4e1ef6acd86e-istio-data\") pod \"e2e-inf-b-9e03c3-openshift-default-749c85bd65-5hqkn\" (UID: \"d9e2c1bc-e15c-4f6e-a12b-4e1ef6acd86e\") " pod="openshift-ingress/e2e-inf-b-9e03c3-openshift-default-749c85bd65-5hqkn" Jul 21 21:36:08.215056 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:36:08.215035 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d9e2c1bc-e15c-4f6e-a12b-4e1ef6acd86e-istiod-ca-cert\") pod \"e2e-inf-b-9e03c3-openshift-default-749c85bd65-5hqkn\" (UID: \"d9e2c1bc-e15c-4f6e-a12b-4e1ef6acd86e\") " pod="openshift-ingress/e2e-inf-b-9e03c3-openshift-default-749c85bd65-5hqkn" Jul 21 21:36:08.216372 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:36:08.216319 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d9e2c1bc-e15c-4f6e-a12b-4e1ef6acd86e-istio-envoy\") pod \"e2e-inf-b-9e03c3-openshift-default-749c85bd65-5hqkn\" (UID: \"d9e2c1bc-e15c-4f6e-a12b-4e1ef6acd86e\") " pod="openshift-ingress/e2e-inf-b-9e03c3-openshift-default-749c85bd65-5hqkn" Jul 21 21:36:08.216630 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:36:08.216614 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d9e2c1bc-e15c-4f6e-a12b-4e1ef6acd86e-istio-podinfo\") pod \"e2e-inf-b-9e03c3-openshift-default-749c85bd65-5hqkn\" (UID: \"d9e2c1bc-e15c-4f6e-a12b-4e1ef6acd86e\") " pod="openshift-ingress/e2e-inf-b-9e03c3-openshift-default-749c85bd65-5hqkn" Jul 21 21:36:08.222139 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:36:08.222113 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5kg5z\" (UniqueName: \"kubernetes.io/projected/d9e2c1bc-e15c-4f6e-a12b-4e1ef6acd86e-kube-api-access-5kg5z\") pod \"e2e-inf-b-9e03c3-openshift-default-749c85bd65-5hqkn\" (UID: \"d9e2c1bc-e15c-4f6e-a12b-4e1ef6acd86e\") " pod="openshift-ingress/e2e-inf-b-9e03c3-openshift-default-749c85bd65-5hqkn" Jul 21 21:36:08.222596 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:36:08.222576 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d9e2c1bc-e15c-4f6e-a12b-4e1ef6acd86e-istio-token\") pod \"e2e-inf-b-9e03c3-openshift-default-749c85bd65-5hqkn\" (UID: \"d9e2c1bc-e15c-4f6e-a12b-4e1ef6acd86e\") " pod="openshift-ingress/e2e-inf-b-9e03c3-openshift-default-749c85bd65-5hqkn" Jul 21 21:36:08.294804 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:36:08.294767 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-9e03c3-openshift-default-749c85bd65-5hqkn" Jul 21 21:36:08.459572 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:36:08.458578 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-9e03c3-openshift-default-749c85bd65-5hqkn"] Jul 21 21:36:08.470221 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:36:08.470193 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd9e2c1bc_e15c_4f6e_a12b_4e1ef6acd86e.slice/crio-b66a0ea0b87ef67e760a48d68cf5710d1820ae619d29a8c22955e0821730b308 WatchSource:0}: Error finding container b66a0ea0b87ef67e760a48d68cf5710d1820ae619d29a8c22955e0821730b308: Status 404 returned error can't find the container with id b66a0ea0b87ef67e760a48d68cf5710d1820ae619d29a8c22955e0821730b308 Jul 21 21:36:08.472403 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:36:08.472369 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 21 21:36:08.472487 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:36:08.472452 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 21 21:36:08.472570 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:36:08.472498 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 21 21:36:08.830443 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:36:08.830341 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-9e03c3-openshift-default-749c85bd65-5hqkn" event={"ID":"d9e2c1bc-e15c-4f6e-a12b-4e1ef6acd86e","Type":"ContainerStarted","Data":"d998f38b69de69dfc8e03ec86c13bce81373cb8689ec473e6d37a518b0e4c139"} Jul 21 21:36:08.830443 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:36:08.830414 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-9e03c3-openshift-default-749c85bd65-5hqkn" event={"ID":"d9e2c1bc-e15c-4f6e-a12b-4e1ef6acd86e","Type":"ContainerStarted","Data":"b66a0ea0b87ef67e760a48d68cf5710d1820ae619d29a8c22955e0821730b308"} Jul 21 21:36:08.853623 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:36:08.853557 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-9e03c3-openshift-default-749c85bd65-5hqkn" podStartSLOduration=1.85351435 podStartE2EDuration="1.85351435s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 21:36:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 21:36:08.850342343 +0000 UTC m=+2126.785092008" watchObservedRunningTime="2026-07-21 21:36:08.85351435 +0000 UTC m=+2126.788264016" Jul 21 21:36:09.295501 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:36:09.295463 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-9e03c3-openshift-default-749c85bd65-5hqkn" Jul 21 21:36:09.300228 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:36:09.300205 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-9e03c3-openshift-default-749c85bd65-5hqkn" Jul 21 21:36:09.835466 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:36:09.835434 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-9e03c3-openshift-default-749c85bd65-5hqkn" Jul 21 21:36:09.836276 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:36:09.836257 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-9e03c3-openshift-default-749c85bd65-5hqkn" Jul 21 21:36:14.863931 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:36:14.863860 2575 generic.go:356] "Generic (PLEG): container finished" podID="65aa8de5-9a34-4c5a-ade1-d0fa36ae1f19" containerID="561cc3063f47d5995f56807df316ac0a3dc23026ba22446c3b2f1f8d21c89e88" exitCode=0 Jul 21 21:36:14.864351 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:36:14.863948 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-bacc91-openshift-default-6f644c796c-lns9p" event={"ID":"65aa8de5-9a34-4c5a-ade1-d0fa36ae1f19","Type":"ContainerDied","Data":"561cc3063f47d5995f56807df316ac0a3dc23026ba22446c3b2f1f8d21c89e88"} Jul 21 21:36:14.864423 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:36:14.864354 2575 scope.go:117] "RemoveContainer" containerID="561cc3063f47d5995f56807df316ac0a3dc23026ba22446c3b2f1f8d21c89e88" Jul 21 21:36:14.865377 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:36:14.865347 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 21 21:36:14.865463 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:36:14.865412 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 21 21:36:14.865463 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:36:14.865440 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 21 21:36:15.871415 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:36:15.871374 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-bacc91-openshift-default-6f644c796c-lns9p" event={"ID":"65aa8de5-9a34-4c5a-ade1-d0fa36ae1f19","Type":"ContainerStarted","Data":"ab6e71afbbbc218e7407612b6e43560bb9b1376c1cf7fbfc8a1fbc1ecb4205cd"} Jul 21 21:36:16.777604 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:36:16.777567 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-bacc91-openshift-default-6f644c796c-lns9p" Jul 21 21:36:16.778649 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:36:16.778619 2575 patch_prober.go:28] interesting pod/e2e-inf-a-bacc91-openshift-default-6f644c796c-lns9p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" start-of-body= Jul 21 21:36:16.778813 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:36:16.778670 2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-bacc91-openshift-default-6f644c796c-lns9p" podUID="65aa8de5-9a34-4c5a-ade1-d0fa36ae1f19" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" Jul 21 21:36:16.796272 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:36:16.796249 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-bacc91-openshift-default-6f644c796c-lns9p" Jul 21 21:36:17.777613 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:36:17.777579 2575 patch_prober.go:28] interesting pod/e2e-inf-a-bacc91-openshift-default-6f644c796c-lns9p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" start-of-body= Jul 21 21:36:17.777984 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:36:17.777631 2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-bacc91-openshift-default-6f644c796c-lns9p" podUID="65aa8de5-9a34-4c5a-ade1-d0fa36ae1f19" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" Jul 21 21:36:18.777334 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:36:18.777294 2575 patch_prober.go:28] interesting pod/e2e-inf-a-bacc91-openshift-default-6f644c796c-lns9p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" start-of-body= Jul 21 21:36:18.777514 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:36:18.777346 2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-bacc91-openshift-default-6f644c796c-lns9p" podUID="65aa8de5-9a34-4c5a-ade1-d0fa36ae1f19" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" Jul 21 21:36:19.781672 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:36:19.781643 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-bacc91-openshift-default-6f644c796c-lns9p" Jul 21 21:36:19.782655 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:36:19.782635 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-bacc91-openshift-default-6f644c796c-lns9p" Jul 21 21:36:25.876722 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:36:25.876672 2575 patch_prober.go:28] interesting pod/e2e-inf-b-9e03c3-openshift-default-749c85bd65-5hqkn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Jul 21 21:36:25.877226 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:36:25.876743 2575 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-9e03c3-openshift-default-749c85bd65-5hqkn" podUID="d9e2c1bc-e15c-4f6e-a12b-4e1ef6acd86e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Jul 21 21:36:26.934313 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:36:26.934276 2575 generic.go:356] "Generic (PLEG): container finished" podID="d9e2c1bc-e15c-4f6e-a12b-4e1ef6acd86e" containerID="d998f38b69de69dfc8e03ec86c13bce81373cb8689ec473e6d37a518b0e4c139" exitCode=0 Jul 21 21:36:26.934771 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:36:26.934350 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-9e03c3-openshift-default-749c85bd65-5hqkn" event={"ID":"d9e2c1bc-e15c-4f6e-a12b-4e1ef6acd86e","Type":"ContainerDied","Data":"d998f38b69de69dfc8e03ec86c13bce81373cb8689ec473e6d37a518b0e4c139"} Jul 21 21:36:26.934771 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:36:26.934754 2575 scope.go:117] "RemoveContainer" containerID="d998f38b69de69dfc8e03ec86c13bce81373cb8689ec473e6d37a518b0e4c139" Jul 21 21:36:26.935958 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:36:26.935914 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 21 21:36:26.936066 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:36:26.936017 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 21 21:36:26.936106 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:36:26.936077 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 21 21:36:27.942300 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:36:27.942269 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-9e03c3-openshift-default-749c85bd65-5hqkn" event={"ID":"d9e2c1bc-e15c-4f6e-a12b-4e1ef6acd86e","Type":"ContainerStarted","Data":"e86c3fb18bdae205740fac3bc1abe49ccca4ebeadf7681b73ada3b03f0232d23"} Jul 21 21:36:28.295105 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:36:28.295022 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-9e03c3-openshift-default-749c85bd65-5hqkn" Jul 21 21:36:28.295249 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:36:28.295221 2575 patch_prober.go:28] interesting pod/e2e-inf-b-9e03c3-openshift-default-749c85bd65-5hqkn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body= Jul 21 21:36:28.295290 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:36:28.295263 2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-9e03c3-openshift-default-749c85bd65-5hqkn" podUID="d9e2c1bc-e15c-4f6e-a12b-4e1ef6acd86e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" Jul 21 21:36:29.295818 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:36:29.295781 2575 patch_prober.go:28] interesting pod/e2e-inf-b-9e03c3-openshift-default-749c85bd65-5hqkn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body= Jul 21 21:36:29.296210 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:36:29.295845 2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-9e03c3-openshift-default-749c85bd65-5hqkn" podUID="d9e2c1bc-e15c-4f6e-a12b-4e1ef6acd86e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" Jul 21 21:36:30.295843 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:36:30.295798 2575 patch_prober.go:28] interesting pod/e2e-inf-b-9e03c3-openshift-default-749c85bd65-5hqkn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body= Jul 21 21:36:30.296383 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:36:30.295863 2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-9e03c3-openshift-default-749c85bd65-5hqkn" podUID="d9e2c1bc-e15c-4f6e-a12b-4e1ef6acd86e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" Jul 21 21:36:31.295666 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:36:31.295624 2575 patch_prober.go:28] interesting pod/e2e-inf-b-9e03c3-openshift-default-749c85bd65-5hqkn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body= Jul 21 21:36:31.295877 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:36:31.295679 2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-9e03c3-openshift-default-749c85bd65-5hqkn" podUID="d9e2c1bc-e15c-4f6e-a12b-4e1ef6acd86e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" Jul 21 21:36:32.295596 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:36:32.295563 2575 patch_prober.go:28] interesting pod/e2e-inf-b-9e03c3-openshift-default-749c85bd65-5hqkn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body= Jul 21 21:36:32.295765 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:36:32.295619 2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-9e03c3-openshift-default-749c85bd65-5hqkn" podUID="d9e2c1bc-e15c-4f6e-a12b-4e1ef6acd86e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" Jul 21 21:36:33.299570 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:36:33.299512 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-9e03c3-openshift-default-749c85bd65-5hqkn" Jul 21 21:36:33.299970 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:36:33.299907 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-9e03c3-openshift-default-749c85bd65-5hqkn" Jul 21 21:36:33.300691 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:36:33.300672 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-9e03c3-openshift-default-749c85bd65-5hqkn" Jul 21 21:36:42.729727 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:36:42.729641 2575 scope.go:117] "RemoveContainer" containerID="f13f97833a2e9d27dc985ccb0e959bd46a1d77af9ea7f51038c7dc10aff73b28" Jul 21 21:36:42.738240 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:36:42.737928 2575 scope.go:117] "RemoveContainer" containerID="e605ece58077f3c68c569b3fc931f53b30c7fa8343c3561de05d35e640b9241f" Jul 21 21:38:08.100015 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:38:08.099936 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-bacc91-openshift-default-6f644c796c-lns9p"] Jul 21 21:38:08.100502 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:38:08.100185 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-bacc91-openshift-default-6f644c796c-lns9p" podUID="65aa8de5-9a34-4c5a-ade1-d0fa36ae1f19" containerName="istio-proxy" containerID="cri-o://ab6e71afbbbc218e7407612b6e43560bb9b1376c1cf7fbfc8a1fbc1ecb4205cd" gracePeriod=30 Jul 21 21:38:08.499457 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:38:08.499427 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-9e03c3-w9zm9"] Jul 21 21:38:08.503012 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:38:08.502986 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-9e03c3-w9zm9" Jul 21 21:38:08.506134 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:38:08.506112 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 21 21:38:08.506252 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:38:08.506161 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9fxhr\"" Jul 21 21:38:08.511931 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:38:08.511905 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-9e03c3-w9zm9"] Jul 21 21:38:08.574021 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:38:08.573989 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d247d063-d1fa-4971-a5a7-3539cee7fdf9-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-9e03c3-w9zm9\" (UID: \"d247d063-d1fa-4971-a5a7-3539cee7fdf9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-9e03c3-w9zm9" Jul 21 21:38:08.675337 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:38:08.675300 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d247d063-d1fa-4971-a5a7-3539cee7fdf9-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-9e03c3-w9zm9\" (UID: \"d247d063-d1fa-4971-a5a7-3539cee7fdf9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-9e03c3-w9zm9" Jul 21 21:38:08.675956 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:38:08.675933 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d247d063-d1fa-4971-a5a7-3539cee7fdf9-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-9e03c3-w9zm9\" (UID: \"d247d063-d1fa-4971-a5a7-3539cee7fdf9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-9e03c3-w9zm9" Jul 21 21:38:08.814506 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:38:08.814437 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-9e03c3-w9zm9" Jul 21 21:38:08.946860 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:38:08.946835 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-9e03c3-w9zm9"] Jul 21 21:38:08.949132 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:38:08.949105 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd247d063_d1fa_4971_a5a7_3539cee7fdf9.slice/crio-43a395d145fd7f65cafbc6f5af560646679e2dfea00dd86574c364f4f3051ae2 WatchSource:0}: Error finding container 43a395d145fd7f65cafbc6f5af560646679e2dfea00dd86574c364f4f3051ae2: Status 404 returned error can't find the container with id 43a395d145fd7f65cafbc6f5af560646679e2dfea00dd86574c364f4f3051ae2 Jul 21 21:38:09.442345 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:38:09.442241 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-9e03c3-w9zm9" event={"ID":"d247d063-d1fa-4971-a5a7-3539cee7fdf9","Type":"ContainerStarted","Data":"fdaeb42e6695f521567b2c92f2d680d23330075235d0eed54434b21b6d39886b"} Jul 21 21:38:09.442918 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:38:09.442890 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-9e03c3-w9zm9" event={"ID":"d247d063-d1fa-4971-a5a7-3539cee7fdf9","Type":"ContainerStarted","Data":"43a395d145fd7f65cafbc6f5af560646679e2dfea00dd86574c364f4f3051ae2"} Jul 21 21:38:09.466708 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:38:09.466646 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-9e03c3-w9zm9" podStartSLOduration=1.466627897 podStartE2EDuration="1.466627897s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 21:38:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 21:38:09.465289682 +0000 UTC m=+2247.400039346" watchObservedRunningTime="2026-07-21 21:38:09.466627897 +0000 UTC m=+2247.401377563" Jul 21 21:38:10.449209 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:38:10.449164 2575 generic.go:356] "Generic (PLEG): container finished" podID="d247d063-d1fa-4971-a5a7-3539cee7fdf9" containerID="fdaeb42e6695f521567b2c92f2d680d23330075235d0eed54434b21b6d39886b" exitCode=0 Jul 21 21:38:10.449594 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:38:10.449212 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-9e03c3-w9zm9" event={"ID":"d247d063-d1fa-4971-a5a7-3539cee7fdf9","Type":"ContainerDied","Data":"fdaeb42e6695f521567b2c92f2d680d23330075235d0eed54434b21b6d39886b"} Jul 21 21:38:11.587000 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:38:11.586978 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-9e03c3-w9zm9" Jul 21 21:38:11.706800 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:38:11.706720 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d247d063-d1fa-4971-a5a7-3539cee7fdf9-maas-api-service-ca\") pod \"d247d063-d1fa-4971-a5a7-3539cee7fdf9\" (UID: \"d247d063-d1fa-4971-a5a7-3539cee7fdf9\") " Jul 21 21:38:11.707100 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:38:11.707080 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d247d063-d1fa-4971-a5a7-3539cee7fdf9-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "d247d063-d1fa-4971-a5a7-3539cee7fdf9" (UID: "d247d063-d1fa-4971-a5a7-3539cee7fdf9"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 21 21:38:11.807478 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:38:11.807441 2575 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d247d063-d1fa-4971-a5a7-3539cee7fdf9-maas-api-service-ca\") on node \"ip-10-0-140-97.ec2.internal\" DevicePath \"\"" Jul 21 21:38:12.461604 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:38:12.461575 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-9e03c3-w9zm9" Jul 21 21:38:12.461604 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:38:12.461585 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-9e03c3-w9zm9" event={"ID":"d247d063-d1fa-4971-a5a7-3539cee7fdf9","Type":"ContainerDied","Data":"43a395d145fd7f65cafbc6f5af560646679e2dfea00dd86574c364f4f3051ae2"} Jul 21 21:38:12.461806 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:38:12.461635 2575 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="43a395d145fd7f65cafbc6f5af560646679e2dfea00dd86574c364f4f3051ae2" Jul 21 21:38:13.361335 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:38:13.361314 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-bacc91-openshift-default-6f644c796c-lns9p" Jul 21 21:38:13.422293 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:38:13.422218 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/65aa8de5-9a34-4c5a-ade1-d0fa36ae1f19-istiod-ca-cert\") pod \"65aa8de5-9a34-4c5a-ade1-d0fa36ae1f19\" (UID: \"65aa8de5-9a34-4c5a-ade1-d0fa36ae1f19\") " Jul 21 21:38:13.422293 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:38:13.422248 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/65aa8de5-9a34-4c5a-ade1-d0fa36ae1f19-istio-token\") pod \"65aa8de5-9a34-4c5a-ade1-d0fa36ae1f19\" (UID: \"65aa8de5-9a34-4c5a-ade1-d0fa36ae1f19\") " Jul 21 21:38:13.422480 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:38:13.422302 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-x7t88\" (UniqueName: \"kubernetes.io/projected/65aa8de5-9a34-4c5a-ade1-d0fa36ae1f19-kube-api-access-x7t88\") pod \"65aa8de5-9a34-4c5a-ade1-d0fa36ae1f19\" (UID: \"65aa8de5-9a34-4c5a-ade1-d0fa36ae1f19\") " Jul 21 21:38:13.422480 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:38:13.422353 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/65aa8de5-9a34-4c5a-ade1-d0fa36ae1f19-istio-podinfo\") pod \"65aa8de5-9a34-4c5a-ade1-d0fa36ae1f19\" (UID: \"65aa8de5-9a34-4c5a-ade1-d0fa36ae1f19\") " Jul 21 21:38:13.422581 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:38:13.422479 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/65aa8de5-9a34-4c5a-ade1-d0fa36ae1f19-istio-envoy\") pod \"65aa8de5-9a34-4c5a-ade1-d0fa36ae1f19\" (UID: \"65aa8de5-9a34-4c5a-ade1-d0fa36ae1f19\") " Jul 21 21:38:13.422634 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:38:13.422615 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/65aa8de5-9a34-4c5a-ade1-d0fa36ae1f19-istio-data\") pod \"65aa8de5-9a34-4c5a-ade1-d0fa36ae1f19\" (UID: \"65aa8de5-9a34-4c5a-ade1-d0fa36ae1f19\") " Jul 21 21:38:13.422696 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:38:13.422643 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/65aa8de5-9a34-4c5a-ade1-d0fa36ae1f19-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "65aa8de5-9a34-4c5a-ade1-d0fa36ae1f19" (UID: "65aa8de5-9a34-4c5a-ade1-d0fa36ae1f19"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 21 21:38:13.422696 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:38:13.422657 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/65aa8de5-9a34-4c5a-ade1-d0fa36ae1f19-credential-socket\") pod \"65aa8de5-9a34-4c5a-ade1-d0fa36ae1f19\" (UID: \"65aa8de5-9a34-4c5a-ade1-d0fa36ae1f19\") " Jul 21 21:38:13.422798 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:38:13.422700 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/65aa8de5-9a34-4c5a-ade1-d0fa36ae1f19-workload-socket\") pod \"65aa8de5-9a34-4c5a-ade1-d0fa36ae1f19\" (UID: \"65aa8de5-9a34-4c5a-ade1-d0fa36ae1f19\") " Jul 21 21:38:13.422798 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:38:13.422763 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/65aa8de5-9a34-4c5a-ade1-d0fa36ae1f19-workload-certs\") pod \"65aa8de5-9a34-4c5a-ade1-d0fa36ae1f19\" (UID: \"65aa8de5-9a34-4c5a-ade1-d0fa36ae1f19\") " Jul 21 21:38:13.423169 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:38:13.422999 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/65aa8de5-9a34-4c5a-ade1-d0fa36ae1f19-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "65aa8de5-9a34-4c5a-ade1-d0fa36ae1f19" (UID: "65aa8de5-9a34-4c5a-ade1-d0fa36ae1f19"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 21:38:13.423169 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:38:13.423012 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/65aa8de5-9a34-4c5a-ade1-d0fa36ae1f19-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "65aa8de5-9a34-4c5a-ade1-d0fa36ae1f19" (UID: "65aa8de5-9a34-4c5a-ade1-d0fa36ae1f19"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 21:38:13.423494 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:38:13.423221 2575 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/65aa8de5-9a34-4c5a-ade1-d0fa36ae1f19-credential-socket\") on node \"ip-10-0-140-97.ec2.internal\" DevicePath \"\"" Jul 21 21:38:13.423494 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:38:13.423240 2575 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/65aa8de5-9a34-4c5a-ade1-d0fa36ae1f19-workload-socket\") on node \"ip-10-0-140-97.ec2.internal\" DevicePath \"\"" Jul 21 21:38:13.423494 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:38:13.423253 2575 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/65aa8de5-9a34-4c5a-ade1-d0fa36ae1f19-istiod-ca-cert\") on node \"ip-10-0-140-97.ec2.internal\" DevicePath \"\"" Jul 21 21:38:13.423691 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:38:13.423631 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/65aa8de5-9a34-4c5a-ade1-d0fa36ae1f19-istio-data" (OuterVolumeSpecName: "istio-data") pod "65aa8de5-9a34-4c5a-ade1-d0fa36ae1f19" (UID: "65aa8de5-9a34-4c5a-ade1-d0fa36ae1f19"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 21:38:13.423769 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:38:13.423750 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/65aa8de5-9a34-4c5a-ade1-d0fa36ae1f19-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "65aa8de5-9a34-4c5a-ade1-d0fa36ae1f19" (UID: "65aa8de5-9a34-4c5a-ade1-d0fa36ae1f19"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 21:38:13.424668 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:38:13.424638 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/65aa8de5-9a34-4c5a-ade1-d0fa36ae1f19-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "65aa8de5-9a34-4c5a-ade1-d0fa36ae1f19" (UID: "65aa8de5-9a34-4c5a-ade1-d0fa36ae1f19"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 21 21:38:13.424800 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:38:13.424779 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/65aa8de5-9a34-4c5a-ade1-d0fa36ae1f19-istio-token" (OuterVolumeSpecName: "istio-token") pod "65aa8de5-9a34-4c5a-ade1-d0fa36ae1f19" (UID: "65aa8de5-9a34-4c5a-ade1-d0fa36ae1f19"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 21 21:38:13.425003 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:38:13.424978 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/65aa8de5-9a34-4c5a-ade1-d0fa36ae1f19-kube-api-access-x7t88" (OuterVolumeSpecName: "kube-api-access-x7t88") pod "65aa8de5-9a34-4c5a-ade1-d0fa36ae1f19" (UID: "65aa8de5-9a34-4c5a-ade1-d0fa36ae1f19"). InnerVolumeSpecName "kube-api-access-x7t88". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 21 21:38:13.425003 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:38:13.424984 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/65aa8de5-9a34-4c5a-ade1-d0fa36ae1f19-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "65aa8de5-9a34-4c5a-ade1-d0fa36ae1f19" (UID: "65aa8de5-9a34-4c5a-ade1-d0fa36ae1f19"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 21:38:13.468027 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:38:13.467994 2575 generic.go:356] "Generic (PLEG): container finished" podID="65aa8de5-9a34-4c5a-ade1-d0fa36ae1f19" containerID="ab6e71afbbbc218e7407612b6e43560bb9b1376c1cf7fbfc8a1fbc1ecb4205cd" exitCode=0 Jul 21 21:38:13.468166 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:38:13.468042 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-bacc91-openshift-default-6f644c796c-lns9p" Jul 21 21:38:13.468166 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:38:13.468071 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-bacc91-openshift-default-6f644c796c-lns9p" event={"ID":"65aa8de5-9a34-4c5a-ade1-d0fa36ae1f19","Type":"ContainerDied","Data":"ab6e71afbbbc218e7407612b6e43560bb9b1376c1cf7fbfc8a1fbc1ecb4205cd"} Jul 21 21:38:13.468166 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:38:13.468124 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-bacc91-openshift-default-6f644c796c-lns9p" event={"ID":"65aa8de5-9a34-4c5a-ade1-d0fa36ae1f19","Type":"ContainerDied","Data":"17db0c7002294ba00873a077e1ec0d1eff9b9bbf749222f83941246539dfd3ce"} Jul 21 21:38:13.468166 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:38:13.468145 2575 scope.go:117] "RemoveContainer" containerID="ab6e71afbbbc218e7407612b6e43560bb9b1376c1cf7fbfc8a1fbc1ecb4205cd" Jul 21 21:38:13.477336 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:38:13.477319 2575 scope.go:117] "RemoveContainer" containerID="561cc3063f47d5995f56807df316ac0a3dc23026ba22446c3b2f1f8d21c89e88" Jul 21 21:38:13.484970 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:38:13.484949 2575 scope.go:117] "RemoveContainer" containerID="ab6e71afbbbc218e7407612b6e43560bb9b1376c1cf7fbfc8a1fbc1ecb4205cd" Jul 21 21:38:13.485253 ip-10-0-140-97 kubenswrapper[2575]: E0721 21:38:13.485227 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ab6e71afbbbc218e7407612b6e43560bb9b1376c1cf7fbfc8a1fbc1ecb4205cd\": container with ID starting with ab6e71afbbbc218e7407612b6e43560bb9b1376c1cf7fbfc8a1fbc1ecb4205cd not found: ID does not exist" containerID="ab6e71afbbbc218e7407612b6e43560bb9b1376c1cf7fbfc8a1fbc1ecb4205cd" Jul 21 21:38:13.485322 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:38:13.485254 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ab6e71afbbbc218e7407612b6e43560bb9b1376c1cf7fbfc8a1fbc1ecb4205cd"} err="failed to get container status \"ab6e71afbbbc218e7407612b6e43560bb9b1376c1cf7fbfc8a1fbc1ecb4205cd\": rpc error: code = NotFound desc = could not find container \"ab6e71afbbbc218e7407612b6e43560bb9b1376c1cf7fbfc8a1fbc1ecb4205cd\": container with ID starting with ab6e71afbbbc218e7407612b6e43560bb9b1376c1cf7fbfc8a1fbc1ecb4205cd not found: ID does not exist" Jul 21 21:38:13.485322 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:38:13.485275 2575 scope.go:117] "RemoveContainer" containerID="561cc3063f47d5995f56807df316ac0a3dc23026ba22446c3b2f1f8d21c89e88" Jul 21 21:38:13.485483 ip-10-0-140-97 kubenswrapper[2575]: E0721 21:38:13.485465 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"561cc3063f47d5995f56807df316ac0a3dc23026ba22446c3b2f1f8d21c89e88\": container with ID starting with 561cc3063f47d5995f56807df316ac0a3dc23026ba22446c3b2f1f8d21c89e88 not found: ID does not exist" containerID="561cc3063f47d5995f56807df316ac0a3dc23026ba22446c3b2f1f8d21c89e88" Jul 21 21:38:13.485563 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:38:13.485493 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"561cc3063f47d5995f56807df316ac0a3dc23026ba22446c3b2f1f8d21c89e88"} err="failed to get container status \"561cc3063f47d5995f56807df316ac0a3dc23026ba22446c3b2f1f8d21c89e88\": rpc error: code = NotFound desc = could not find container \"561cc3063f47d5995f56807df316ac0a3dc23026ba22446c3b2f1f8d21c89e88\": container with ID starting with 561cc3063f47d5995f56807df316ac0a3dc23026ba22446c3b2f1f8d21c89e88 not found: ID does not exist" Jul 21 21:38:13.496112 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:38:13.496090 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-bacc91-openshift-default-6f644c796c-lns9p"] Jul 21 21:38:13.512698 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:38:13.512673 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-bacc91-openshift-default-6f644c796c-lns9p"] Jul 21 21:38:13.524461 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:38:13.524438 2575 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/65aa8de5-9a34-4c5a-ade1-d0fa36ae1f19-istio-data\") on node \"ip-10-0-140-97.ec2.internal\" DevicePath \"\"" Jul 21 21:38:13.524461 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:38:13.524462 2575 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/65aa8de5-9a34-4c5a-ade1-d0fa36ae1f19-workload-certs\") on node \"ip-10-0-140-97.ec2.internal\" DevicePath \"\"" Jul 21 21:38:13.524612 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:38:13.524470 2575 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/65aa8de5-9a34-4c5a-ade1-d0fa36ae1f19-istio-token\") on node \"ip-10-0-140-97.ec2.internal\" DevicePath \"\"" Jul 21 21:38:13.524612 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:38:13.524480 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x7t88\" (UniqueName: \"kubernetes.io/projected/65aa8de5-9a34-4c5a-ade1-d0fa36ae1f19-kube-api-access-x7t88\") on node \"ip-10-0-140-97.ec2.internal\" DevicePath \"\"" Jul 21 21:38:13.524612 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:38:13.524489 2575 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/65aa8de5-9a34-4c5a-ade1-d0fa36ae1f19-istio-podinfo\") on node \"ip-10-0-140-97.ec2.internal\" DevicePath \"\"" Jul 21 21:38:13.524612 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:38:13.524498 2575 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/65aa8de5-9a34-4c5a-ade1-d0fa36ae1f19-istio-envoy\") on node \"ip-10-0-140-97.ec2.internal\" DevicePath \"\"" Jul 21 21:38:14.741059 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:38:14.741021 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="65aa8de5-9a34-4c5a-ade1-d0fa36ae1f19" path="/var/lib/kubelet/pods/65aa8de5-9a34-4c5a-ade1-d0fa36ae1f19/volumes" Jul 21 21:38:23.587567 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:38:23.587520 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-9e03c3-w9zm9"] Jul 21 21:38:23.590893 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:38:23.590860 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-9e03c3-w9zm9"] Jul 21 21:38:24.741025 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:38:24.740991 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d247d063-d1fa-4971-a5a7-3539cee7fdf9" path="/var/lib/kubelet/pods/d247d063-d1fa-4971-a5a7-3539cee7fdf9/volumes" Jul 21 21:38:30.394484 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:38:30.394440 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-9e03c3-openshift-default-749c85bd65-5hqkn"] Jul 21 21:38:30.394973 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:38:30.394733 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-9e03c3-openshift-default-749c85bd65-5hqkn" podUID="d9e2c1bc-e15c-4f6e-a12b-4e1ef6acd86e" containerName="istio-proxy" containerID="cri-o://e86c3fb18bdae205740fac3bc1abe49ccca4ebeadf7681b73ada3b03f0232d23" gracePeriod=30 Jul 21 21:38:31.343145 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:38:31.343103 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-226c3e-openshift-default-75654f7bc7-7dlg2"] Jul 21 21:38:31.343603 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:38:31.343585 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="65aa8de5-9a34-4c5a-ade1-d0fa36ae1f19" containerName="istio-proxy" Jul 21 21:38:31.343603 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:38:31.343604 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="65aa8de5-9a34-4c5a-ade1-d0fa36ae1f19" containerName="istio-proxy" Jul 21 21:38:31.343760 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:38:31.343622 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d247d063-d1fa-4971-a5a7-3539cee7fdf9" containerName="revoke-keys" Jul 21 21:38:31.343760 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:38:31.343630 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="d247d063-d1fa-4971-a5a7-3539cee7fdf9" containerName="revoke-keys" Jul 21 21:38:31.343760 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:38:31.343647 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="65aa8de5-9a34-4c5a-ade1-d0fa36ae1f19" containerName="istio-proxy" Jul 21 21:38:31.343760 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:38:31.343655 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="65aa8de5-9a34-4c5a-ade1-d0fa36ae1f19" containerName="istio-proxy" Jul 21 21:38:31.343760 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:38:31.343745 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="65aa8de5-9a34-4c5a-ade1-d0fa36ae1f19" containerName="istio-proxy" Jul 21 21:38:31.343760 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:38:31.343759 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="d247d063-d1fa-4971-a5a7-3539cee7fdf9" containerName="revoke-keys" Jul 21 21:38:31.347204 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:38:31.347183 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-226c3e-openshift-default-75654f7bc7-7dlg2" Jul 21 21:38:31.351423 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:38:31.351401 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-226c3e-openshift-default-dockercfg-hwzbv\"" Jul 21 21:38:31.361500 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:38:31.361472 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-226c3e-openshift-default-75654f7bc7-7dlg2"] Jul 21 21:38:31.493991 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:38:31.493943 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9b214bc0-2416-47ec-9eb7-3ff2a0caed51-istio-envoy\") pod \"e2e-api-a-226c3e-openshift-default-75654f7bc7-7dlg2\" (UID: \"9b214bc0-2416-47ec-9eb7-3ff2a0caed51\") " pod="openshift-ingress/e2e-api-a-226c3e-openshift-default-75654f7bc7-7dlg2" Jul 21 21:38:31.493991 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:38:31.493992 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9b214bc0-2416-47ec-9eb7-3ff2a0caed51-istiod-ca-cert\") pod \"e2e-api-a-226c3e-openshift-default-75654f7bc7-7dlg2\" (UID: \"9b214bc0-2416-47ec-9eb7-3ff2a0caed51\") " pod="openshift-ingress/e2e-api-a-226c3e-openshift-default-75654f7bc7-7dlg2" Jul 21 21:38:31.494405 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:38:31.494039 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b214bc0-2416-47ec-9eb7-3ff2a0caed51-workload-socket\") pod \"e2e-api-a-226c3e-openshift-default-75654f7bc7-7dlg2\" (UID: \"9b214bc0-2416-47ec-9eb7-3ff2a0caed51\") " pod="openshift-ingress/e2e-api-a-226c3e-openshift-default-75654f7bc7-7dlg2" Jul 21 21:38:31.494405 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:38:31.494057 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9b214bc0-2416-47ec-9eb7-3ff2a0caed51-workload-certs\") pod \"e2e-api-a-226c3e-openshift-default-75654f7bc7-7dlg2\" (UID: \"9b214bc0-2416-47ec-9eb7-3ff2a0caed51\") " pod="openshift-ingress/e2e-api-a-226c3e-openshift-default-75654f7bc7-7dlg2" Jul 21 21:38:31.494405 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:38:31.494084 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9b214bc0-2416-47ec-9eb7-3ff2a0caed51-istio-token\") pod \"e2e-api-a-226c3e-openshift-default-75654f7bc7-7dlg2\" (UID: \"9b214bc0-2416-47ec-9eb7-3ff2a0caed51\") " pod="openshift-ingress/e2e-api-a-226c3e-openshift-default-75654f7bc7-7dlg2" Jul 21 21:38:31.494405 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:38:31.494117 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9b214bc0-2416-47ec-9eb7-3ff2a0caed51-istio-podinfo\") pod \"e2e-api-a-226c3e-openshift-default-75654f7bc7-7dlg2\" (UID: \"9b214bc0-2416-47ec-9eb7-3ff2a0caed51\") " pod="openshift-ingress/e2e-api-a-226c3e-openshift-default-75654f7bc7-7dlg2" Jul 21 21:38:31.494405 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:38:31.494191 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9b214bc0-2416-47ec-9eb7-3ff2a0caed51-istio-data\") pod \"e2e-api-a-226c3e-openshift-default-75654f7bc7-7dlg2\" (UID: \"9b214bc0-2416-47ec-9eb7-3ff2a0caed51\") " pod="openshift-ingress/e2e-api-a-226c3e-openshift-default-75654f7bc7-7dlg2" Jul 21 21:38:31.494405 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:38:31.494237 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b214bc0-2416-47ec-9eb7-3ff2a0caed51-credential-socket\") pod \"e2e-api-a-226c3e-openshift-default-75654f7bc7-7dlg2\" (UID: \"9b214bc0-2416-47ec-9eb7-3ff2a0caed51\") " pod="openshift-ingress/e2e-api-a-226c3e-openshift-default-75654f7bc7-7dlg2" Jul 21 21:38:31.494405 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:38:31.494261 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xm88z\" (UniqueName: \"kubernetes.io/projected/9b214bc0-2416-47ec-9eb7-3ff2a0caed51-kube-api-access-xm88z\") pod \"e2e-api-a-226c3e-openshift-default-75654f7bc7-7dlg2\" (UID: \"9b214bc0-2416-47ec-9eb7-3ff2a0caed51\") " pod="openshift-ingress/e2e-api-a-226c3e-openshift-default-75654f7bc7-7dlg2" Jul 21 21:38:31.594906 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:38:31.594818 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b214bc0-2416-47ec-9eb7-3ff2a0caed51-workload-socket\") pod \"e2e-api-a-226c3e-openshift-default-75654f7bc7-7dlg2\" (UID: \"9b214bc0-2416-47ec-9eb7-3ff2a0caed51\") " pod="openshift-ingress/e2e-api-a-226c3e-openshift-default-75654f7bc7-7dlg2" Jul 21 21:38:31.594906 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:38:31.594860 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9b214bc0-2416-47ec-9eb7-3ff2a0caed51-workload-certs\") pod \"e2e-api-a-226c3e-openshift-default-75654f7bc7-7dlg2\" (UID: \"9b214bc0-2416-47ec-9eb7-3ff2a0caed51\") " pod="openshift-ingress/e2e-api-a-226c3e-openshift-default-75654f7bc7-7dlg2" Jul 21 21:38:31.594906 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:38:31.594901 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9b214bc0-2416-47ec-9eb7-3ff2a0caed51-istio-token\") pod \"e2e-api-a-226c3e-openshift-default-75654f7bc7-7dlg2\" (UID: \"9b214bc0-2416-47ec-9eb7-3ff2a0caed51\") " pod="openshift-ingress/e2e-api-a-226c3e-openshift-default-75654f7bc7-7dlg2" Jul 21 21:38:31.595164 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:38:31.594947 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9b214bc0-2416-47ec-9eb7-3ff2a0caed51-istio-podinfo\") pod \"e2e-api-a-226c3e-openshift-default-75654f7bc7-7dlg2\" (UID: \"9b214bc0-2416-47ec-9eb7-3ff2a0caed51\") " pod="openshift-ingress/e2e-api-a-226c3e-openshift-default-75654f7bc7-7dlg2" Jul 21 21:38:31.595164 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:38:31.594998 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9b214bc0-2416-47ec-9eb7-3ff2a0caed51-istio-data\") pod \"e2e-api-a-226c3e-openshift-default-75654f7bc7-7dlg2\" (UID: \"9b214bc0-2416-47ec-9eb7-3ff2a0caed51\") " pod="openshift-ingress/e2e-api-a-226c3e-openshift-default-75654f7bc7-7dlg2" Jul 21 21:38:31.595164 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:38:31.595042 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b214bc0-2416-47ec-9eb7-3ff2a0caed51-credential-socket\") pod \"e2e-api-a-226c3e-openshift-default-75654f7bc7-7dlg2\" (UID: \"9b214bc0-2416-47ec-9eb7-3ff2a0caed51\") " pod="openshift-ingress/e2e-api-a-226c3e-openshift-default-75654f7bc7-7dlg2" Jul 21 21:38:31.595164 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:38:31.595066 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xm88z\" (UniqueName: \"kubernetes.io/projected/9b214bc0-2416-47ec-9eb7-3ff2a0caed51-kube-api-access-xm88z\") pod \"e2e-api-a-226c3e-openshift-default-75654f7bc7-7dlg2\" (UID: \"9b214bc0-2416-47ec-9eb7-3ff2a0caed51\") " pod="openshift-ingress/e2e-api-a-226c3e-openshift-default-75654f7bc7-7dlg2" Jul 21 21:38:31.595164 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:38:31.595125 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9b214bc0-2416-47ec-9eb7-3ff2a0caed51-istio-envoy\") pod \"e2e-api-a-226c3e-openshift-default-75654f7bc7-7dlg2\" (UID: \"9b214bc0-2416-47ec-9eb7-3ff2a0caed51\") " pod="openshift-ingress/e2e-api-a-226c3e-openshift-default-75654f7bc7-7dlg2" Jul 21 21:38:31.595164 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:38:31.595149 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9b214bc0-2416-47ec-9eb7-3ff2a0caed51-istiod-ca-cert\") pod \"e2e-api-a-226c3e-openshift-default-75654f7bc7-7dlg2\" (UID: \"9b214bc0-2416-47ec-9eb7-3ff2a0caed51\") " pod="openshift-ingress/e2e-api-a-226c3e-openshift-default-75654f7bc7-7dlg2" Jul 21 21:38:31.595473 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:38:31.595314 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b214bc0-2416-47ec-9eb7-3ff2a0caed51-workload-socket\") pod \"e2e-api-a-226c3e-openshift-default-75654f7bc7-7dlg2\" (UID: \"9b214bc0-2416-47ec-9eb7-3ff2a0caed51\") " pod="openshift-ingress/e2e-api-a-226c3e-openshift-default-75654f7bc7-7dlg2" Jul 21 21:38:31.595473 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:38:31.595368 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9b214bc0-2416-47ec-9eb7-3ff2a0caed51-workload-certs\") pod \"e2e-api-a-226c3e-openshift-default-75654f7bc7-7dlg2\" (UID: \"9b214bc0-2416-47ec-9eb7-3ff2a0caed51\") " pod="openshift-ingress/e2e-api-a-226c3e-openshift-default-75654f7bc7-7dlg2" Jul 21 21:38:31.595473 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:38:31.595411 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9b214bc0-2416-47ec-9eb7-3ff2a0caed51-istio-data\") pod \"e2e-api-a-226c3e-openshift-default-75654f7bc7-7dlg2\" (UID: \"9b214bc0-2416-47ec-9eb7-3ff2a0caed51\") " pod="openshift-ingress/e2e-api-a-226c3e-openshift-default-75654f7bc7-7dlg2" Jul 21 21:38:31.595619 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:38:31.595487 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b214bc0-2416-47ec-9eb7-3ff2a0caed51-credential-socket\") pod \"e2e-api-a-226c3e-openshift-default-75654f7bc7-7dlg2\" (UID: \"9b214bc0-2416-47ec-9eb7-3ff2a0caed51\") " pod="openshift-ingress/e2e-api-a-226c3e-openshift-default-75654f7bc7-7dlg2" Jul 21 21:38:31.595771 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:38:31.595754 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9b214bc0-2416-47ec-9eb7-3ff2a0caed51-istiod-ca-cert\") pod \"e2e-api-a-226c3e-openshift-default-75654f7bc7-7dlg2\" (UID: \"9b214bc0-2416-47ec-9eb7-3ff2a0caed51\") " pod="openshift-ingress/e2e-api-a-226c3e-openshift-default-75654f7bc7-7dlg2" Jul 21 21:38:31.597522 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:38:31.597492 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9b214bc0-2416-47ec-9eb7-3ff2a0caed51-istio-envoy\") pod \"e2e-api-a-226c3e-openshift-default-75654f7bc7-7dlg2\" (UID: \"9b214bc0-2416-47ec-9eb7-3ff2a0caed51\") " pod="openshift-ingress/e2e-api-a-226c3e-openshift-default-75654f7bc7-7dlg2" Jul 21 21:38:31.597789 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:38:31.597768 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9b214bc0-2416-47ec-9eb7-3ff2a0caed51-istio-podinfo\") pod \"e2e-api-a-226c3e-openshift-default-75654f7bc7-7dlg2\" (UID: \"9b214bc0-2416-47ec-9eb7-3ff2a0caed51\") " pod="openshift-ingress/e2e-api-a-226c3e-openshift-default-75654f7bc7-7dlg2" Jul 21 21:38:31.606168 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:38:31.606147 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xm88z\" (UniqueName: \"kubernetes.io/projected/9b214bc0-2416-47ec-9eb7-3ff2a0caed51-kube-api-access-xm88z\") pod \"e2e-api-a-226c3e-openshift-default-75654f7bc7-7dlg2\" (UID: \"9b214bc0-2416-47ec-9eb7-3ff2a0caed51\") " pod="openshift-ingress/e2e-api-a-226c3e-openshift-default-75654f7bc7-7dlg2" Jul 21 21:38:31.606282 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:38:31.606217 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9b214bc0-2416-47ec-9eb7-3ff2a0caed51-istio-token\") pod \"e2e-api-a-226c3e-openshift-default-75654f7bc7-7dlg2\" (UID: \"9b214bc0-2416-47ec-9eb7-3ff2a0caed51\") " pod="openshift-ingress/e2e-api-a-226c3e-openshift-default-75654f7bc7-7dlg2" Jul 21 21:38:31.657259 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:38:31.657222 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-226c3e-openshift-default-75654f7bc7-7dlg2" Jul 21 21:38:31.801807 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:38:31.801776 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-226c3e-openshift-default-75654f7bc7-7dlg2"] Jul 21 21:38:31.803896 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:38:31.803870 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9b214bc0_2416_47ec_9eb7_3ff2a0caed51.slice/crio-f5c011fb58d42aa970fe585d18b63bd0647ebf23a8e3eef187e13ee21222b56f WatchSource:0}: Error finding container f5c011fb58d42aa970fe585d18b63bd0647ebf23a8e3eef187e13ee21222b56f: Status 404 returned error can't find the container with id f5c011fb58d42aa970fe585d18b63bd0647ebf23a8e3eef187e13ee21222b56f Jul 21 21:38:31.805933 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:38:31.805902 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 21 21:38:31.805995 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:38:31.805969 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 21 21:38:31.806040 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:38:31.805995 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 21 21:38:32.573104 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:38:32.573068 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-226c3e-openshift-default-75654f7bc7-7dlg2" event={"ID":"9b214bc0-2416-47ec-9eb7-3ff2a0caed51","Type":"ContainerStarted","Data":"9910c2af718415f981f1e75eda8224e35d2c32554396e73052d17839e75172bf"} Jul 21 21:38:32.573466 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:38:32.573123 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-226c3e-openshift-default-75654f7bc7-7dlg2" event={"ID":"9b214bc0-2416-47ec-9eb7-3ff2a0caed51","Type":"ContainerStarted","Data":"f5c011fb58d42aa970fe585d18b63bd0647ebf23a8e3eef187e13ee21222b56f"} Jul 21 21:38:32.611379 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:38:32.611325 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-226c3e-openshift-default-75654f7bc7-7dlg2" podStartSLOduration=1.611309973 podStartE2EDuration="1.611309973s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 21:38:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 21:38:32.61074979 +0000 UTC m=+2270.545499469" watchObservedRunningTime="2026-07-21 21:38:32.611309973 +0000 UTC m=+2270.546059639" Jul 21 21:38:32.657740 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:38:32.657709 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-226c3e-openshift-default-75654f7bc7-7dlg2" Jul 21 21:38:32.662680 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:38:32.662658 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-226c3e-openshift-default-75654f7bc7-7dlg2" Jul 21 21:38:33.301768 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:38:33.301728 2575 patch_prober.go:28] interesting pod/e2e-inf-b-9e03c3-openshift-default-749c85bd65-5hqkn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 21 21:38:33.301925 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:38:33.301793 2575 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-9e03c3-openshift-default-749c85bd65-5hqkn" podUID="d9e2c1bc-e15c-4f6e-a12b-4e1ef6acd86e" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 21 21:38:33.578363 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:38:33.578278 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-226c3e-openshift-default-75654f7bc7-7dlg2" Jul 21 21:38:33.579415 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:38:33.579392 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-226c3e-openshift-default-75654f7bc7-7dlg2" Jul 21 21:38:35.591415 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:38:35.591374 2575 generic.go:356] "Generic (PLEG): container finished" podID="d9e2c1bc-e15c-4f6e-a12b-4e1ef6acd86e" containerID="e86c3fb18bdae205740fac3bc1abe49ccca4ebeadf7681b73ada3b03f0232d23" exitCode=0 Jul 21 21:38:35.591859 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:38:35.591445 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-9e03c3-openshift-default-749c85bd65-5hqkn" event={"ID":"d9e2c1bc-e15c-4f6e-a12b-4e1ef6acd86e","Type":"ContainerDied","Data":"e86c3fb18bdae205740fac3bc1abe49ccca4ebeadf7681b73ada3b03f0232d23"} Jul 21 21:38:35.591859 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:38:35.591521 2575 scope.go:117] "RemoveContainer" containerID="d998f38b69de69dfc8e03ec86c13bce81373cb8689ec473e6d37a518b0e4c139" Jul 21 21:38:35.650885 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:38:35.650856 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-9e03c3-openshift-default-749c85bd65-5hqkn" Jul 21 21:38:35.837984 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:38:35.837890 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9e2c1bc-e15c-4f6e-a12b-4e1ef6acd86e-workload-socket\") pod \"d9e2c1bc-e15c-4f6e-a12b-4e1ef6acd86e\" (UID: \"d9e2c1bc-e15c-4f6e-a12b-4e1ef6acd86e\") " Jul 21 21:38:35.837984 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:38:35.837937 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d9e2c1bc-e15c-4f6e-a12b-4e1ef6acd86e-istio-token\") pod \"d9e2c1bc-e15c-4f6e-a12b-4e1ef6acd86e\" (UID: \"d9e2c1bc-e15c-4f6e-a12b-4e1ef6acd86e\") " Jul 21 21:38:35.837984 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:38:35.837967 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d9e2c1bc-e15c-4f6e-a12b-4e1ef6acd86e-istio-data\") pod \"d9e2c1bc-e15c-4f6e-a12b-4e1ef6acd86e\" (UID: \"d9e2c1bc-e15c-4f6e-a12b-4e1ef6acd86e\") " Jul 21 21:38:35.838299 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:38:35.838046 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d9e2c1bc-e15c-4f6e-a12b-4e1ef6acd86e-workload-certs\") pod \"d9e2c1bc-e15c-4f6e-a12b-4e1ef6acd86e\" (UID: \"d9e2c1bc-e15c-4f6e-a12b-4e1ef6acd86e\") " Jul 21 21:38:35.838299 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:38:35.838102 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9e2c1bc-e15c-4f6e-a12b-4e1ef6acd86e-credential-socket\") pod \"d9e2c1bc-e15c-4f6e-a12b-4e1ef6acd86e\" (UID: \"d9e2c1bc-e15c-4f6e-a12b-4e1ef6acd86e\") " Jul 21 21:38:35.838299 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:38:35.838134 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d9e2c1bc-e15c-4f6e-a12b-4e1ef6acd86e-istio-envoy\") pod \"d9e2c1bc-e15c-4f6e-a12b-4e1ef6acd86e\" (UID: \"d9e2c1bc-e15c-4f6e-a12b-4e1ef6acd86e\") " Jul 21 21:38:35.838299 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:38:35.838172 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d9e2c1bc-e15c-4f6e-a12b-4e1ef6acd86e-istio-podinfo\") pod \"d9e2c1bc-e15c-4f6e-a12b-4e1ef6acd86e\" (UID: \"d9e2c1bc-e15c-4f6e-a12b-4e1ef6acd86e\") " Jul 21 21:38:35.838299 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:38:35.838195 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d9e2c1bc-e15c-4f6e-a12b-4e1ef6acd86e-istiod-ca-cert\") pod \"d9e2c1bc-e15c-4f6e-a12b-4e1ef6acd86e\" (UID: \"d9e2c1bc-e15c-4f6e-a12b-4e1ef6acd86e\") " Jul 21 21:38:35.838299 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:38:35.838198 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d9e2c1bc-e15c-4f6e-a12b-4e1ef6acd86e-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "d9e2c1bc-e15c-4f6e-a12b-4e1ef6acd86e" (UID: "d9e2c1bc-e15c-4f6e-a12b-4e1ef6acd86e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 21:38:35.838299 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:38:35.838255 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5kg5z\" (UniqueName: \"kubernetes.io/projected/d9e2c1bc-e15c-4f6e-a12b-4e1ef6acd86e-kube-api-access-5kg5z\") pod \"d9e2c1bc-e15c-4f6e-a12b-4e1ef6acd86e\" (UID: \"d9e2c1bc-e15c-4f6e-a12b-4e1ef6acd86e\") " Jul 21 21:38:35.838658 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:38:35.838381 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d9e2c1bc-e15c-4f6e-a12b-4e1ef6acd86e-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "d9e2c1bc-e15c-4f6e-a12b-4e1ef6acd86e" (UID: "d9e2c1bc-e15c-4f6e-a12b-4e1ef6acd86e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 21:38:35.838658 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:38:35.838580 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d9e2c1bc-e15c-4f6e-a12b-4e1ef6acd86e-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "d9e2c1bc-e15c-4f6e-a12b-4e1ef6acd86e" (UID: "d9e2c1bc-e15c-4f6e-a12b-4e1ef6acd86e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 21:38:35.838837 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:38:35.838818 2575 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d9e2c1bc-e15c-4f6e-a12b-4e1ef6acd86e-workload-certs\") on node \"ip-10-0-140-97.ec2.internal\" DevicePath \"\"" Jul 21 21:38:35.838890 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:38:35.838844 2575 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9e2c1bc-e15c-4f6e-a12b-4e1ef6acd86e-credential-socket\") on node \"ip-10-0-140-97.ec2.internal\" DevicePath \"\"" Jul 21 21:38:35.838890 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:38:35.838859 2575 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9e2c1bc-e15c-4f6e-a12b-4e1ef6acd86e-workload-socket\") on node \"ip-10-0-140-97.ec2.internal\" DevicePath \"\"" Jul 21 21:38:35.839323 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:38:35.839299 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d9e2c1bc-e15c-4f6e-a12b-4e1ef6acd86e-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "d9e2c1bc-e15c-4f6e-a12b-4e1ef6acd86e" (UID: "d9e2c1bc-e15c-4f6e-a12b-4e1ef6acd86e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 21 21:38:35.839496 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:38:35.839467 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d9e2c1bc-e15c-4f6e-a12b-4e1ef6acd86e-istio-data" (OuterVolumeSpecName: "istio-data") pod "d9e2c1bc-e15c-4f6e-a12b-4e1ef6acd86e" (UID: "d9e2c1bc-e15c-4f6e-a12b-4e1ef6acd86e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 21:38:35.841190 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:38:35.841162 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d9e2c1bc-e15c-4f6e-a12b-4e1ef6acd86e-kube-api-access-5kg5z" (OuterVolumeSpecName: "kube-api-access-5kg5z") pod "d9e2c1bc-e15c-4f6e-a12b-4e1ef6acd86e" (UID: "d9e2c1bc-e15c-4f6e-a12b-4e1ef6acd86e"). InnerVolumeSpecName "kube-api-access-5kg5z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 21 21:38:35.841298 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:38:35.841167 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d9e2c1bc-e15c-4f6e-a12b-4e1ef6acd86e-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "d9e2c1bc-e15c-4f6e-a12b-4e1ef6acd86e" (UID: "d9e2c1bc-e15c-4f6e-a12b-4e1ef6acd86e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 21:38:35.841298 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:38:35.841205 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d9e2c1bc-e15c-4f6e-a12b-4e1ef6acd86e-istio-token" (OuterVolumeSpecName: "istio-token") pod "d9e2c1bc-e15c-4f6e-a12b-4e1ef6acd86e" (UID: "d9e2c1bc-e15c-4f6e-a12b-4e1ef6acd86e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 21 21:38:35.841373 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:38:35.841307 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d9e2c1bc-e15c-4f6e-a12b-4e1ef6acd86e-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "d9e2c1bc-e15c-4f6e-a12b-4e1ef6acd86e" (UID: "d9e2c1bc-e15c-4f6e-a12b-4e1ef6acd86e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 21 21:38:35.939619 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:38:35.939581 2575 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d9e2c1bc-e15c-4f6e-a12b-4e1ef6acd86e-istio-envoy\") on node \"ip-10-0-140-97.ec2.internal\" DevicePath \"\"" Jul 21 21:38:35.939619 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:38:35.939616 2575 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d9e2c1bc-e15c-4f6e-a12b-4e1ef6acd86e-istio-podinfo\") on node \"ip-10-0-140-97.ec2.internal\" DevicePath \"\"" Jul 21 21:38:35.939619 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:38:35.939627 2575 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d9e2c1bc-e15c-4f6e-a12b-4e1ef6acd86e-istiod-ca-cert\") on node \"ip-10-0-140-97.ec2.internal\" DevicePath \"\"" Jul 21 21:38:35.939865 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:38:35.939639 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5kg5z\" (UniqueName: \"kubernetes.io/projected/d9e2c1bc-e15c-4f6e-a12b-4e1ef6acd86e-kube-api-access-5kg5z\") on node \"ip-10-0-140-97.ec2.internal\" DevicePath \"\"" Jul 21 21:38:35.939865 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:38:35.939651 2575 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d9e2c1bc-e15c-4f6e-a12b-4e1ef6acd86e-istio-token\") on node \"ip-10-0-140-97.ec2.internal\" DevicePath \"\"" Jul 21 21:38:35.939865 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:38:35.939662 2575 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d9e2c1bc-e15c-4f6e-a12b-4e1ef6acd86e-istio-data\") on node \"ip-10-0-140-97.ec2.internal\" DevicePath \"\"" Jul 21 21:38:36.597302 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:38:36.597274 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-9e03c3-openshift-default-749c85bd65-5hqkn" Jul 21 21:38:36.597735 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:38:36.597308 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-9e03c3-openshift-default-749c85bd65-5hqkn" event={"ID":"d9e2c1bc-e15c-4f6e-a12b-4e1ef6acd86e","Type":"ContainerDied","Data":"b66a0ea0b87ef67e760a48d68cf5710d1820ae619d29a8c22955e0821730b308"} Jul 21 21:38:36.597735 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:38:36.597367 2575 scope.go:117] "RemoveContainer" containerID="e86c3fb18bdae205740fac3bc1abe49ccca4ebeadf7681b73ada3b03f0232d23" Jul 21 21:38:36.619155 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:38:36.619124 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-9e03c3-openshift-default-749c85bd65-5hqkn"] Jul 21 21:38:36.623184 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:38:36.623153 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-9e03c3-openshift-default-749c85bd65-5hqkn"] Jul 21 21:38:36.741895 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:38:36.741865 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d9e2c1bc-e15c-4f6e-a12b-4e1ef6acd86e" path="/var/lib/kubelet/pods/d9e2c1bc-e15c-4f6e-a12b-4e1ef6acd86e/volumes" Jul 21 21:38:39.879312 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:38:39.879283 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-5898c4-openshift-default-c87765695-vcj2j"] Jul 21 21:38:39.879717 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:38:39.879684 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d9e2c1bc-e15c-4f6e-a12b-4e1ef6acd86e" containerName="istio-proxy" Jul 21 21:38:39.879717 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:38:39.879696 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="d9e2c1bc-e15c-4f6e-a12b-4e1ef6acd86e" containerName="istio-proxy" Jul 21 21:38:39.879717 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:38:39.879712 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d9e2c1bc-e15c-4f6e-a12b-4e1ef6acd86e" containerName="istio-proxy" Jul 21 21:38:39.879888 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:38:39.879721 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="d9e2c1bc-e15c-4f6e-a12b-4e1ef6acd86e" containerName="istio-proxy" Jul 21 21:38:39.879888 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:38:39.879849 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="d9e2c1bc-e15c-4f6e-a12b-4e1ef6acd86e" containerName="istio-proxy" Jul 21 21:38:39.879888 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:38:39.879865 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="65aa8de5-9a34-4c5a-ade1-d0fa36ae1f19" containerName="istio-proxy" Jul 21 21:38:39.879888 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:38:39.879875 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="d9e2c1bc-e15c-4f6e-a12b-4e1ef6acd86e" containerName="istio-proxy" Jul 21 21:38:39.884352 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:38:39.884332 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-5898c4-openshift-default-c87765695-vcj2j" Jul 21 21:38:39.887629 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:38:39.887610 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-5898c4-openshift-default-dockercfg-9pdpj\"" Jul 21 21:38:39.899716 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:38:39.899694 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-5898c4-openshift-default-c87765695-vcj2j"] Jul 21 21:38:39.980476 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:38:39.980429 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e034e76-c6e3-48c6-ae67-8f2ffc04bc62-workload-socket\") pod \"e2e-api-b-5898c4-openshift-default-c87765695-vcj2j\" (UID: \"8e034e76-c6e3-48c6-ae67-8f2ffc04bc62\") " pod="openshift-ingress/e2e-api-b-5898c4-openshift-default-c87765695-vcj2j" Jul 21 21:38:39.980719 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:38:39.980483 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8e034e76-c6e3-48c6-ae67-8f2ffc04bc62-istio-data\") pod \"e2e-api-b-5898c4-openshift-default-c87765695-vcj2j\" (UID: \"8e034e76-c6e3-48c6-ae67-8f2ffc04bc62\") " pod="openshift-ingress/e2e-api-b-5898c4-openshift-default-c87765695-vcj2j" Jul 21 21:38:39.980719 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:38:39.980598 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x5vgc\" (UniqueName: \"kubernetes.io/projected/8e034e76-c6e3-48c6-ae67-8f2ffc04bc62-kube-api-access-x5vgc\") pod \"e2e-api-b-5898c4-openshift-default-c87765695-vcj2j\" (UID: \"8e034e76-c6e3-48c6-ae67-8f2ffc04bc62\") " pod="openshift-ingress/e2e-api-b-5898c4-openshift-default-c87765695-vcj2j" Jul 21 21:38:39.980719 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:38:39.980649 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8e034e76-c6e3-48c6-ae67-8f2ffc04bc62-workload-certs\") pod \"e2e-api-b-5898c4-openshift-default-c87765695-vcj2j\" (UID: \"8e034e76-c6e3-48c6-ae67-8f2ffc04bc62\") " pod="openshift-ingress/e2e-api-b-5898c4-openshift-default-c87765695-vcj2j" Jul 21 21:38:39.980909 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:38:39.980782 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8e034e76-c6e3-48c6-ae67-8f2ffc04bc62-istio-podinfo\") pod \"e2e-api-b-5898c4-openshift-default-c87765695-vcj2j\" (UID: \"8e034e76-c6e3-48c6-ae67-8f2ffc04bc62\") " pod="openshift-ingress/e2e-api-b-5898c4-openshift-default-c87765695-vcj2j" Jul 21 21:38:39.980909 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:38:39.980833 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8e034e76-c6e3-48c6-ae67-8f2ffc04bc62-istio-envoy\") pod \"e2e-api-b-5898c4-openshift-default-c87765695-vcj2j\" (UID: \"8e034e76-c6e3-48c6-ae67-8f2ffc04bc62\") " pod="openshift-ingress/e2e-api-b-5898c4-openshift-default-c87765695-vcj2j" Jul 21 21:38:39.981020 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:38:39.980922 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e034e76-c6e3-48c6-ae67-8f2ffc04bc62-credential-socket\") pod \"e2e-api-b-5898c4-openshift-default-c87765695-vcj2j\" (UID: \"8e034e76-c6e3-48c6-ae67-8f2ffc04bc62\") " pod="openshift-ingress/e2e-api-b-5898c4-openshift-default-c87765695-vcj2j" Jul 21 21:38:39.981020 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:38:39.980962 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8e034e76-c6e3-48c6-ae67-8f2ffc04bc62-istiod-ca-cert\") pod \"e2e-api-b-5898c4-openshift-default-c87765695-vcj2j\" (UID: \"8e034e76-c6e3-48c6-ae67-8f2ffc04bc62\") " pod="openshift-ingress/e2e-api-b-5898c4-openshift-default-c87765695-vcj2j" Jul 21 21:38:39.981020 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:38:39.980999 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8e034e76-c6e3-48c6-ae67-8f2ffc04bc62-istio-token\") pod \"e2e-api-b-5898c4-openshift-default-c87765695-vcj2j\" (UID: \"8e034e76-c6e3-48c6-ae67-8f2ffc04bc62\") " pod="openshift-ingress/e2e-api-b-5898c4-openshift-default-c87765695-vcj2j" Jul 21 21:38:40.081668 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:38:40.081635 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8e034e76-c6e3-48c6-ae67-8f2ffc04bc62-istio-podinfo\") pod \"e2e-api-b-5898c4-openshift-default-c87765695-vcj2j\" (UID: \"8e034e76-c6e3-48c6-ae67-8f2ffc04bc62\") " pod="openshift-ingress/e2e-api-b-5898c4-openshift-default-c87765695-vcj2j" Jul 21 21:38:40.081826 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:38:40.081685 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8e034e76-c6e3-48c6-ae67-8f2ffc04bc62-istio-envoy\") pod \"e2e-api-b-5898c4-openshift-default-c87765695-vcj2j\" (UID: \"8e034e76-c6e3-48c6-ae67-8f2ffc04bc62\") " pod="openshift-ingress/e2e-api-b-5898c4-openshift-default-c87765695-vcj2j" Jul 21 21:38:40.081826 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:38:40.081711 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e034e76-c6e3-48c6-ae67-8f2ffc04bc62-credential-socket\") pod \"e2e-api-b-5898c4-openshift-default-c87765695-vcj2j\" (UID: \"8e034e76-c6e3-48c6-ae67-8f2ffc04bc62\") " pod="openshift-ingress/e2e-api-b-5898c4-openshift-default-c87765695-vcj2j" Jul 21 21:38:40.081826 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:38:40.081808 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8e034e76-c6e3-48c6-ae67-8f2ffc04bc62-istiod-ca-cert\") pod \"e2e-api-b-5898c4-openshift-default-c87765695-vcj2j\" (UID: \"8e034e76-c6e3-48c6-ae67-8f2ffc04bc62\") " pod="openshift-ingress/e2e-api-b-5898c4-openshift-default-c87765695-vcj2j" Jul 21 21:38:40.082010 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:38:40.081869 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8e034e76-c6e3-48c6-ae67-8f2ffc04bc62-istio-token\") pod \"e2e-api-b-5898c4-openshift-default-c87765695-vcj2j\" (UID: \"8e034e76-c6e3-48c6-ae67-8f2ffc04bc62\") " pod="openshift-ingress/e2e-api-b-5898c4-openshift-default-c87765695-vcj2j" Jul 21 21:38:40.082010 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:38:40.081904 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e034e76-c6e3-48c6-ae67-8f2ffc04bc62-workload-socket\") pod \"e2e-api-b-5898c4-openshift-default-c87765695-vcj2j\" (UID: \"8e034e76-c6e3-48c6-ae67-8f2ffc04bc62\") " pod="openshift-ingress/e2e-api-b-5898c4-openshift-default-c87765695-vcj2j" Jul 21 21:38:40.082010 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:38:40.081923 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8e034e76-c6e3-48c6-ae67-8f2ffc04bc62-istio-data\") pod \"e2e-api-b-5898c4-openshift-default-c87765695-vcj2j\" (UID: \"8e034e76-c6e3-48c6-ae67-8f2ffc04bc62\") " pod="openshift-ingress/e2e-api-b-5898c4-openshift-default-c87765695-vcj2j" Jul 21 21:38:40.082010 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:38:40.081951 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x5vgc\" (UniqueName: \"kubernetes.io/projected/8e034e76-c6e3-48c6-ae67-8f2ffc04bc62-kube-api-access-x5vgc\") pod \"e2e-api-b-5898c4-openshift-default-c87765695-vcj2j\" (UID: \"8e034e76-c6e3-48c6-ae67-8f2ffc04bc62\") " pod="openshift-ingress/e2e-api-b-5898c4-openshift-default-c87765695-vcj2j" Jul 21 21:38:40.082010 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:38:40.081989 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8e034e76-c6e3-48c6-ae67-8f2ffc04bc62-workload-certs\") pod \"e2e-api-b-5898c4-openshift-default-c87765695-vcj2j\" (UID: \"8e034e76-c6e3-48c6-ae67-8f2ffc04bc62\") " pod="openshift-ingress/e2e-api-b-5898c4-openshift-default-c87765695-vcj2j" Jul 21 21:38:40.082292 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:38:40.082080 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e034e76-c6e3-48c6-ae67-8f2ffc04bc62-credential-socket\") pod \"e2e-api-b-5898c4-openshift-default-c87765695-vcj2j\" (UID: \"8e034e76-c6e3-48c6-ae67-8f2ffc04bc62\") " pod="openshift-ingress/e2e-api-b-5898c4-openshift-default-c87765695-vcj2j" Jul 21 21:38:40.082292 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:38:40.082264 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e034e76-c6e3-48c6-ae67-8f2ffc04bc62-workload-socket\") pod \"e2e-api-b-5898c4-openshift-default-c87765695-vcj2j\" (UID: \"8e034e76-c6e3-48c6-ae67-8f2ffc04bc62\") " pod="openshift-ingress/e2e-api-b-5898c4-openshift-default-c87765695-vcj2j" Jul 21 21:38:40.082456 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:38:40.082349 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8e034e76-c6e3-48c6-ae67-8f2ffc04bc62-workload-certs\") pod \"e2e-api-b-5898c4-openshift-default-c87765695-vcj2j\" (UID: \"8e034e76-c6e3-48c6-ae67-8f2ffc04bc62\") " pod="openshift-ingress/e2e-api-b-5898c4-openshift-default-c87765695-vcj2j" Jul 21 21:38:40.082522 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:38:40.082476 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8e034e76-c6e3-48c6-ae67-8f2ffc04bc62-istiod-ca-cert\") pod \"e2e-api-b-5898c4-openshift-default-c87765695-vcj2j\" (UID: \"8e034e76-c6e3-48c6-ae67-8f2ffc04bc62\") " pod="openshift-ingress/e2e-api-b-5898c4-openshift-default-c87765695-vcj2j" Jul 21 21:38:40.082522 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:38:40.082488 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8e034e76-c6e3-48c6-ae67-8f2ffc04bc62-istio-data\") pod \"e2e-api-b-5898c4-openshift-default-c87765695-vcj2j\" (UID: \"8e034e76-c6e3-48c6-ae67-8f2ffc04bc62\") " pod="openshift-ingress/e2e-api-b-5898c4-openshift-default-c87765695-vcj2j" Jul 21 21:38:40.084016 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:38:40.083995 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8e034e76-c6e3-48c6-ae67-8f2ffc04bc62-istio-envoy\") pod \"e2e-api-b-5898c4-openshift-default-c87765695-vcj2j\" (UID: \"8e034e76-c6e3-48c6-ae67-8f2ffc04bc62\") " pod="openshift-ingress/e2e-api-b-5898c4-openshift-default-c87765695-vcj2j" Jul 21 21:38:40.084238 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:38:40.084220 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8e034e76-c6e3-48c6-ae67-8f2ffc04bc62-istio-podinfo\") pod \"e2e-api-b-5898c4-openshift-default-c87765695-vcj2j\" (UID: \"8e034e76-c6e3-48c6-ae67-8f2ffc04bc62\") " pod="openshift-ingress/e2e-api-b-5898c4-openshift-default-c87765695-vcj2j" Jul 21 21:38:40.090582 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:38:40.090556 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8e034e76-c6e3-48c6-ae67-8f2ffc04bc62-istio-token\") pod \"e2e-api-b-5898c4-openshift-default-c87765695-vcj2j\" (UID: \"8e034e76-c6e3-48c6-ae67-8f2ffc04bc62\") " pod="openshift-ingress/e2e-api-b-5898c4-openshift-default-c87765695-vcj2j" Jul 21 21:38:40.090810 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:38:40.090791 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x5vgc\" (UniqueName: \"kubernetes.io/projected/8e034e76-c6e3-48c6-ae67-8f2ffc04bc62-kube-api-access-x5vgc\") pod \"e2e-api-b-5898c4-openshift-default-c87765695-vcj2j\" (UID: \"8e034e76-c6e3-48c6-ae67-8f2ffc04bc62\") " pod="openshift-ingress/e2e-api-b-5898c4-openshift-default-c87765695-vcj2j" Jul 21 21:38:40.194669 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:38:40.194580 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-5898c4-openshift-default-c87765695-vcj2j" Jul 21 21:38:40.351750 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:38:40.351725 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-5898c4-openshift-default-c87765695-vcj2j"] Jul 21 21:38:40.353773 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:38:40.353743 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8e034e76_c6e3_48c6_ae67_8f2ffc04bc62.slice/crio-c96d3877637184432c4f296ab48ee581f72679592cbc445e85e322d3910eec02 WatchSource:0}: Error finding container c96d3877637184432c4f296ab48ee581f72679592cbc445e85e322d3910eec02: Status 404 returned error can't find the container with id c96d3877637184432c4f296ab48ee581f72679592cbc445e85e322d3910eec02 Jul 21 21:38:40.355884 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:38:40.355850 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 21 21:38:40.355971 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:38:40.355918 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 21 21:38:40.355971 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:38:40.355944 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 21 21:38:40.623835 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:38:40.623799 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-5898c4-openshift-default-c87765695-vcj2j" event={"ID":"8e034e76-c6e3-48c6-ae67-8f2ffc04bc62","Type":"ContainerStarted","Data":"d7a914984dda85e049baff9ff10b6f93397e85ff21397b31431f8568a2d0d6fa"} Jul 21 21:38:40.624170 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:38:40.623861 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-5898c4-openshift-default-c87765695-vcj2j" event={"ID":"8e034e76-c6e3-48c6-ae67-8f2ffc04bc62","Type":"ContainerStarted","Data":"c96d3877637184432c4f296ab48ee581f72679592cbc445e85e322d3910eec02"} Jul 21 21:38:40.681713 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:38:40.681656 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-5898c4-openshift-default-c87765695-vcj2j" podStartSLOduration=1.6816380880000001 podStartE2EDuration="1.681638088s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 21:38:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 21:38:40.678144483 +0000 UTC m=+2278.612894168" watchObservedRunningTime="2026-07-21 21:38:40.681638088 +0000 UTC m=+2278.616387754" Jul 21 21:38:41.194991 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:38:41.194949 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-5898c4-openshift-default-c87765695-vcj2j" Jul 21 21:38:41.200047 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:38:41.200020 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-5898c4-openshift-default-c87765695-vcj2j" Jul 21 21:38:41.636467 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:38:41.636430 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-5898c4-openshift-default-c87765695-vcj2j" Jul 21 21:38:41.637582 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:38:41.637520 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-5898c4-openshift-default-c87765695-vcj2j" Jul 21 21:38:42.775624 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:38:42.775589 2575 scope.go:117] "RemoveContainer" containerID="3dcf28709b8584aa0f45e8a04cba2661c3aed097aa31b5ae0782a6f80019282a" Jul 21 21:38:48.579466 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:38:48.579431 2575 patch_prober.go:28] interesting pod/e2e-api-a-226c3e-openshift-default-75654f7bc7-7dlg2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body= Jul 21 21:38:48.579905 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:38:48.579490 2575 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-226c3e-openshift-default-75654f7bc7-7dlg2" podUID="9b214bc0-2416-47ec-9eb7-3ff2a0caed51" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" Jul 21 21:38:48.681320 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:38:48.681285 2575 generic.go:356] "Generic (PLEG): container finished" podID="9b214bc0-2416-47ec-9eb7-3ff2a0caed51" containerID="9910c2af718415f981f1e75eda8224e35d2c32554396e73052d17839e75172bf" exitCode=0 Jul 21 21:38:48.681508 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:38:48.681369 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-226c3e-openshift-default-75654f7bc7-7dlg2" event={"ID":"9b214bc0-2416-47ec-9eb7-3ff2a0caed51","Type":"ContainerDied","Data":"9910c2af718415f981f1e75eda8224e35d2c32554396e73052d17839e75172bf"} Jul 21 21:38:48.681833 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:38:48.681819 2575 scope.go:117] "RemoveContainer" containerID="9910c2af718415f981f1e75eda8224e35d2c32554396e73052d17839e75172bf" Jul 21 21:38:48.682806 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:38:48.682780 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 21 21:38:48.682858 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:38:48.682847 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 21 21:38:48.682902 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:38:48.682879 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 21 21:38:49.689782 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:38:49.689750 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-226c3e-openshift-default-75654f7bc7-7dlg2" event={"ID":"9b214bc0-2416-47ec-9eb7-3ff2a0caed51","Type":"ContainerStarted","Data":"f6fa9e452477938adae3a9d7b3d4ddeacc1c2191e7fac571786f2d5ce772f8cd"} Jul 21 21:38:50.658429 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:38:50.658381 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-226c3e-openshift-default-75654f7bc7-7dlg2" Jul 21 21:38:50.658727 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:38:50.658698 2575 patch_prober.go:28] interesting pod/e2e-api-a-226c3e-openshift-default-75654f7bc7-7dlg2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body= Jul 21 21:38:50.658893 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:38:50.658867 2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-226c3e-openshift-default-75654f7bc7-7dlg2" podUID="9b214bc0-2416-47ec-9eb7-3ff2a0caed51" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" Jul 21 21:38:51.658273 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:38:51.658234 2575 patch_prober.go:28] interesting pod/e2e-api-a-226c3e-openshift-default-75654f7bc7-7dlg2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body= Jul 21 21:38:51.658882 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:38:51.658848 2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-226c3e-openshift-default-75654f7bc7-7dlg2" podUID="9b214bc0-2416-47ec-9eb7-3ff2a0caed51" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" Jul 21 21:38:52.658491 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:38:52.658454 2575 patch_prober.go:28] interesting pod/e2e-api-a-226c3e-openshift-default-75654f7bc7-7dlg2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body= Jul 21 21:38:52.659070 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:38:52.659036 2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-226c3e-openshift-default-75654f7bc7-7dlg2" podUID="9b214bc0-2416-47ec-9eb7-3ff2a0caed51" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" Jul 21 21:38:53.659110 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:38:53.658596 2575 patch_prober.go:28] interesting pod/e2e-api-a-226c3e-openshift-default-75654f7bc7-7dlg2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body= Jul 21 21:38:53.659110 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:38:53.658682 2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-226c3e-openshift-default-75654f7bc7-7dlg2" podUID="9b214bc0-2416-47ec-9eb7-3ff2a0caed51" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" Jul 21 21:38:54.661971 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:38:54.661935 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-226c3e-openshift-default-75654f7bc7-7dlg2" Jul 21 21:38:54.662460 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:38:54.662439 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-226c3e-openshift-default-75654f7bc7-7dlg2" Jul 21 21:38:54.663057 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:38:54.663036 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-226c3e-openshift-default-75654f7bc7-7dlg2" Jul 21 21:38:55.724756 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:38:55.724721 2575 generic.go:356] "Generic (PLEG): container finished" podID="8e034e76-c6e3-48c6-ae67-8f2ffc04bc62" containerID="d7a914984dda85e049baff9ff10b6f93397e85ff21397b31431f8568a2d0d6fa" exitCode=0 Jul 21 21:38:55.725194 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:38:55.724791 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-5898c4-openshift-default-c87765695-vcj2j" event={"ID":"8e034e76-c6e3-48c6-ae67-8f2ffc04bc62","Type":"ContainerDied","Data":"d7a914984dda85e049baff9ff10b6f93397e85ff21397b31431f8568a2d0d6fa"} Jul 21 21:38:55.725329 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:38:55.725315 2575 scope.go:117] "RemoveContainer" containerID="d7a914984dda85e049baff9ff10b6f93397e85ff21397b31431f8568a2d0d6fa" Jul 21 21:38:55.726295 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:38:55.726267 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 21 21:38:55.726369 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:38:55.726350 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 21 21:38:55.726415 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:38:55.726401 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 21 21:38:56.637172 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:38:56.637118 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-5898c4-openshift-default-c87765695-vcj2j" Jul 21 21:38:56.731082 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:38:56.731050 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-5898c4-openshift-default-c87765695-vcj2j" event={"ID":"8e034e76-c6e3-48c6-ae67-8f2ffc04bc62","Type":"ContainerStarted","Data":"5e3ef35fc244a528f60136f25df645c2319164789191e03dc6acba6e3d135db8"} Jul 21 21:38:57.195399 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:38:57.195357 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-5898c4-openshift-default-c87765695-vcj2j" Jul 21 21:38:57.196416 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:38:57.196392 2575 patch_prober.go:28] interesting pod/e2e-api-b-5898c4-openshift-default-c87765695-vcj2j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" start-of-body= Jul 21 21:38:57.196487 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:38:57.196443 2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-5898c4-openshift-default-c87765695-vcj2j" podUID="8e034e76-c6e3-48c6-ae67-8f2ffc04bc62" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" Jul 21 21:38:58.195600 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:38:58.195521 2575 patch_prober.go:28] interesting pod/e2e-api-b-5898c4-openshift-default-c87765695-vcj2j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" start-of-body= Jul 21 21:38:58.195996 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:38:58.195614 2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-5898c4-openshift-default-c87765695-vcj2j" podUID="8e034e76-c6e3-48c6-ae67-8f2ffc04bc62" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" Jul 21 21:38:59.195300 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:38:59.195261 2575 patch_prober.go:28] interesting pod/e2e-api-b-5898c4-openshift-default-c87765695-vcj2j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" start-of-body= Jul 21 21:38:59.195485 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:38:59.195328 2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-5898c4-openshift-default-c87765695-vcj2j" podUID="8e034e76-c6e3-48c6-ae67-8f2ffc04bc62" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" Jul 21 21:39:00.195525 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:39:00.195483 2575 patch_prober.go:28] interesting pod/e2e-api-b-5898c4-openshift-default-c87765695-vcj2j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" start-of-body= Jul 21 21:39:00.196027 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:39:00.195583 2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-5898c4-openshift-default-c87765695-vcj2j" podUID="8e034e76-c6e3-48c6-ae67-8f2ffc04bc62" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" Jul 21 21:39:01.198679 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:39:01.198648 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-5898c4-openshift-default-c87765695-vcj2j" Jul 21 21:39:01.199070 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:39:01.199028 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-5898c4-openshift-default-c87765695-vcj2j" Jul 21 21:39:01.199686 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:39:01.199665 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-5898c4-openshift-default-c87765695-vcj2j" Jul 21 21:39:21.837228 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:39:21.837195 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-226c3e-openshift-default-75654f7bc7-7dlg2"] Jul 21 21:39:21.837782 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:39:21.837401 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-226c3e-openshift-default-75654f7bc7-7dlg2" podUID="9b214bc0-2416-47ec-9eb7-3ff2a0caed51" containerName="istio-proxy" containerID="cri-o://f6fa9e452477938adae3a9d7b3d4ddeacc1c2191e7fac571786f2d5ce772f8cd" gracePeriod=30 Jul 21 21:39:22.226861 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:39:22.226824 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-5898c4-4h9hd"] Jul 21 21:39:22.230695 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:39:22.230679 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-5898c4-4h9hd" Jul 21 21:39:22.233445 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:39:22.233413 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9fxhr\"" Jul 21 21:39:22.233587 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:39:22.233477 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 21 21:39:22.241353 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:39:22.241328 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-5898c4-4h9hd"] Jul 21 21:39:22.301625 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:39:22.301588 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b2691f37-18af-42ec-a441-0d7eee0442d3-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-5898c4-4h9hd\" (UID: \"b2691f37-18af-42ec-a441-0d7eee0442d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-5898c4-4h9hd" Jul 21 21:39:22.402336 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:39:22.402297 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b2691f37-18af-42ec-a441-0d7eee0442d3-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-5898c4-4h9hd\" (UID: \"b2691f37-18af-42ec-a441-0d7eee0442d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-5898c4-4h9hd" Jul 21 21:39:22.403002 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:39:22.402980 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b2691f37-18af-42ec-a441-0d7eee0442d3-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-5898c4-4h9hd\" (UID: \"b2691f37-18af-42ec-a441-0d7eee0442d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-5898c4-4h9hd" Jul 21 21:39:22.541432 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:39:22.541333 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-5898c4-4h9hd" Jul 21 21:39:22.685554 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:39:22.685513 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-5898c4-4h9hd"] Jul 21 21:39:22.686953 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:39:22.686928 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb2691f37_18af_42ec_a441_0d7eee0442d3.slice/crio-bf89e57722effaa5574e6389bb2f979681cf73a4704c78d4bfd6bebb709027c2 WatchSource:0}: Error finding container bf89e57722effaa5574e6389bb2f979681cf73a4704c78d4bfd6bebb709027c2: Status 404 returned error can't find the container with id bf89e57722effaa5574e6389bb2f979681cf73a4704c78d4bfd6bebb709027c2 Jul 21 21:39:22.869851 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:39:22.869818 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-5898c4-4h9hd" event={"ID":"b2691f37-18af-42ec-a441-0d7eee0442d3","Type":"ContainerStarted","Data":"f9b62bbc510310956c353686071201576e714df9a0cb546ea4c54da4ac468aee"} Jul 21 21:39:22.870236 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:39:22.869872 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-5898c4-4h9hd" event={"ID":"b2691f37-18af-42ec-a441-0d7eee0442d3","Type":"ContainerStarted","Data":"bf89e57722effaa5574e6389bb2f979681cf73a4704c78d4bfd6bebb709027c2"} Jul 21 21:39:22.888434 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:39:22.888383 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-5898c4-4h9hd" podStartSLOduration=0.88836794 podStartE2EDuration="888.36794ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 21:39:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 21:39:22.887641961 +0000 UTC m=+2320.822391628" watchObservedRunningTime="2026-07-21 21:39:22.88836794 +0000 UTC m=+2320.823117619" Jul 21 21:39:23.875770 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:39:23.875737 2575 generic.go:356] "Generic (PLEG): container finished" podID="b2691f37-18af-42ec-a441-0d7eee0442d3" containerID="f9b62bbc510310956c353686071201576e714df9a0cb546ea4c54da4ac468aee" exitCode=0 Jul 21 21:39:23.876118 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:39:23.875819 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-5898c4-4h9hd" event={"ID":"b2691f37-18af-42ec-a441-0d7eee0442d3","Type":"ContainerDied","Data":"f9b62bbc510310956c353686071201576e714df9a0cb546ea4c54da4ac468aee"} Jul 21 21:39:24.665464 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:39:24.665426 2575 patch_prober.go:28] interesting pod/e2e-api-a-226c3e-openshift-default-75654f7bc7-7dlg2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=upstream connect error or disconnect/reset before headers. reset reason: remote connection failure, transport failure reason: delayed connect error: Connection refused Jul 21 21:39:24.665655 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:39:24.665507 2575 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-226c3e-openshift-default-75654f7bc7-7dlg2" podUID="9b214bc0-2416-47ec-9eb7-3ff2a0caed51" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 21 21:39:25.019966 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:39:25.019941 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-5898c4-4h9hd" Jul 21 21:39:25.130621 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:39:25.130582 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b2691f37-18af-42ec-a441-0d7eee0442d3-maas-api-service-ca\") pod \"b2691f37-18af-42ec-a441-0d7eee0442d3\" (UID: \"b2691f37-18af-42ec-a441-0d7eee0442d3\") " Jul 21 21:39:25.130949 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:39:25.130921 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b2691f37-18af-42ec-a441-0d7eee0442d3-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "b2691f37-18af-42ec-a441-0d7eee0442d3" (UID: "b2691f37-18af-42ec-a441-0d7eee0442d3"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 21 21:39:25.232163 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:39:25.232075 2575 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b2691f37-18af-42ec-a441-0d7eee0442d3-maas-api-service-ca\") on node \"ip-10-0-140-97.ec2.internal\" DevicePath \"\"" Jul 21 21:39:25.886834 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:39:25.886799 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-5898c4-4h9hd" event={"ID":"b2691f37-18af-42ec-a441-0d7eee0442d3","Type":"ContainerDied","Data":"bf89e57722effaa5574e6389bb2f979681cf73a4704c78d4bfd6bebb709027c2"} Jul 21 21:39:25.886834 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:39:25.886821 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-5898c4-4h9hd" Jul 21 21:39:25.887042 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:39:25.886850 2575 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bf89e57722effaa5574e6389bb2f979681cf73a4704c78d4bfd6bebb709027c2" Jul 21 21:39:27.095673 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:39:27.095647 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-226c3e-openshift-default-75654f7bc7-7dlg2" Jul 21 21:39:27.150188 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:39:27.150155 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9b214bc0-2416-47ec-9eb7-3ff2a0caed51-istio-data\") pod \"9b214bc0-2416-47ec-9eb7-3ff2a0caed51\" (UID: \"9b214bc0-2416-47ec-9eb7-3ff2a0caed51\") " Jul 21 21:39:27.150188 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:39:27.150190 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9b214bc0-2416-47ec-9eb7-3ff2a0caed51-istio-envoy\") pod \"9b214bc0-2416-47ec-9eb7-3ff2a0caed51\" (UID: \"9b214bc0-2416-47ec-9eb7-3ff2a0caed51\") " Jul 21 21:39:27.150423 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:39:27.150226 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b214bc0-2416-47ec-9eb7-3ff2a0caed51-workload-socket\") pod \"9b214bc0-2416-47ec-9eb7-3ff2a0caed51\" (UID: \"9b214bc0-2416-47ec-9eb7-3ff2a0caed51\") " Jul 21 21:39:27.150423 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:39:27.150256 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9b214bc0-2416-47ec-9eb7-3ff2a0caed51-istiod-ca-cert\") pod \"9b214bc0-2416-47ec-9eb7-3ff2a0caed51\" (UID: \"9b214bc0-2416-47ec-9eb7-3ff2a0caed51\") " Jul 21 21:39:27.150423 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:39:27.150282 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9b214bc0-2416-47ec-9eb7-3ff2a0caed51-istio-podinfo\") pod \"9b214bc0-2416-47ec-9eb7-3ff2a0caed51\" (UID: \"9b214bc0-2416-47ec-9eb7-3ff2a0caed51\") " Jul 21 21:39:27.150423 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:39:27.150313 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b214bc0-2416-47ec-9eb7-3ff2a0caed51-credential-socket\") pod \"9b214bc0-2416-47ec-9eb7-3ff2a0caed51\" (UID: \"9b214bc0-2416-47ec-9eb7-3ff2a0caed51\") " Jul 21 21:39:27.150423 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:39:27.150338 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xm88z\" (UniqueName: \"kubernetes.io/projected/9b214bc0-2416-47ec-9eb7-3ff2a0caed51-kube-api-access-xm88z\") pod \"9b214bc0-2416-47ec-9eb7-3ff2a0caed51\" (UID: \"9b214bc0-2416-47ec-9eb7-3ff2a0caed51\") " Jul 21 21:39:27.150423 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:39:27.150387 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9b214bc0-2416-47ec-9eb7-3ff2a0caed51-istio-token\") pod \"9b214bc0-2416-47ec-9eb7-3ff2a0caed51\" (UID: \"9b214bc0-2416-47ec-9eb7-3ff2a0caed51\") " Jul 21 21:39:27.150776 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:39:27.150458 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9b214bc0-2416-47ec-9eb7-3ff2a0caed51-workload-certs\") pod \"9b214bc0-2416-47ec-9eb7-3ff2a0caed51\" (UID: \"9b214bc0-2416-47ec-9eb7-3ff2a0caed51\") " Jul 21 21:39:27.150840 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:39:27.150790 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b214bc0-2416-47ec-9eb7-3ff2a0caed51-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "9b214bc0-2416-47ec-9eb7-3ff2a0caed51" (UID: "9b214bc0-2416-47ec-9eb7-3ff2a0caed51"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 21:39:27.150840 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:39:27.150798 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9b214bc0-2416-47ec-9eb7-3ff2a0caed51-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "9b214bc0-2416-47ec-9eb7-3ff2a0caed51" (UID: "9b214bc0-2416-47ec-9eb7-3ff2a0caed51"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 21 21:39:27.150948 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:39:27.150879 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b214bc0-2416-47ec-9eb7-3ff2a0caed51-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "9b214bc0-2416-47ec-9eb7-3ff2a0caed51" (UID: "9b214bc0-2416-47ec-9eb7-3ff2a0caed51"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 21:39:27.151066 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:39:27.151002 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b214bc0-2416-47ec-9eb7-3ff2a0caed51-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "9b214bc0-2416-47ec-9eb7-3ff2a0caed51" (UID: "9b214bc0-2416-47ec-9eb7-3ff2a0caed51"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 21:39:27.151066 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:39:27.151052 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b214bc0-2416-47ec-9eb7-3ff2a0caed51-istio-data" (OuterVolumeSpecName: "istio-data") pod "9b214bc0-2416-47ec-9eb7-3ff2a0caed51" (UID: "9b214bc0-2416-47ec-9eb7-3ff2a0caed51"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 21:39:27.152625 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:39:27.152594 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9b214bc0-2416-47ec-9eb7-3ff2a0caed51-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "9b214bc0-2416-47ec-9eb7-3ff2a0caed51" (UID: "9b214bc0-2416-47ec-9eb7-3ff2a0caed51"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 21 21:39:27.152768 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:39:27.152753 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9b214bc0-2416-47ec-9eb7-3ff2a0caed51-istio-token" (OuterVolumeSpecName: "istio-token") pod "9b214bc0-2416-47ec-9eb7-3ff2a0caed51" (UID: "9b214bc0-2416-47ec-9eb7-3ff2a0caed51"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 21 21:39:27.152943 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:39:27.152912 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9b214bc0-2416-47ec-9eb7-3ff2a0caed51-kube-api-access-xm88z" (OuterVolumeSpecName: "kube-api-access-xm88z") pod "9b214bc0-2416-47ec-9eb7-3ff2a0caed51" (UID: "9b214bc0-2416-47ec-9eb7-3ff2a0caed51"). InnerVolumeSpecName "kube-api-access-xm88z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 21 21:39:27.153013 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:39:27.152937 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b214bc0-2416-47ec-9eb7-3ff2a0caed51-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "9b214bc0-2416-47ec-9eb7-3ff2a0caed51" (UID: "9b214bc0-2416-47ec-9eb7-3ff2a0caed51"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 21:39:27.251500 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:39:27.251463 2575 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9b214bc0-2416-47ec-9eb7-3ff2a0caed51-istio-data\") on node \"ip-10-0-140-97.ec2.internal\" DevicePath \"\"" Jul 21 21:39:27.251500 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:39:27.251493 2575 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9b214bc0-2416-47ec-9eb7-3ff2a0caed51-istio-envoy\") on node \"ip-10-0-140-97.ec2.internal\" DevicePath \"\"" Jul 21 21:39:27.251500 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:39:27.251501 2575 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b214bc0-2416-47ec-9eb7-3ff2a0caed51-workload-socket\") on node \"ip-10-0-140-97.ec2.internal\" DevicePath \"\"" Jul 21 21:39:27.251500 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:39:27.251509 2575 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9b214bc0-2416-47ec-9eb7-3ff2a0caed51-istiod-ca-cert\") on node \"ip-10-0-140-97.ec2.internal\" DevicePath \"\"" Jul 21 21:39:27.251784 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:39:27.251517 2575 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9b214bc0-2416-47ec-9eb7-3ff2a0caed51-istio-podinfo\") on node \"ip-10-0-140-97.ec2.internal\" DevicePath \"\"" Jul 21 21:39:27.251784 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:39:27.251525 2575 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b214bc0-2416-47ec-9eb7-3ff2a0caed51-credential-socket\") on node \"ip-10-0-140-97.ec2.internal\" DevicePath \"\"" Jul 21 21:39:27.251784 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:39:27.251556 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xm88z\" (UniqueName: \"kubernetes.io/projected/9b214bc0-2416-47ec-9eb7-3ff2a0caed51-kube-api-access-xm88z\") on node \"ip-10-0-140-97.ec2.internal\" DevicePath \"\"" Jul 21 21:39:27.251784 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:39:27.251564 2575 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9b214bc0-2416-47ec-9eb7-3ff2a0caed51-istio-token\") on node \"ip-10-0-140-97.ec2.internal\" DevicePath \"\"" Jul 21 21:39:27.251784 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:39:27.251572 2575 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9b214bc0-2416-47ec-9eb7-3ff2a0caed51-workload-certs\") on node \"ip-10-0-140-97.ec2.internal\" DevicePath \"\"" Jul 21 21:39:27.898014 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:39:27.897978 2575 generic.go:356] "Generic (PLEG): container finished" podID="9b214bc0-2416-47ec-9eb7-3ff2a0caed51" containerID="f6fa9e452477938adae3a9d7b3d4ddeacc1c2191e7fac571786f2d5ce772f8cd" exitCode=0 Jul 21 21:39:27.898267 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:39:27.898026 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-226c3e-openshift-default-75654f7bc7-7dlg2" Jul 21 21:39:27.898267 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:39:27.898058 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-226c3e-openshift-default-75654f7bc7-7dlg2" event={"ID":"9b214bc0-2416-47ec-9eb7-3ff2a0caed51","Type":"ContainerDied","Data":"f6fa9e452477938adae3a9d7b3d4ddeacc1c2191e7fac571786f2d5ce772f8cd"} Jul 21 21:39:27.898267 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:39:27.898110 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-226c3e-openshift-default-75654f7bc7-7dlg2" event={"ID":"9b214bc0-2416-47ec-9eb7-3ff2a0caed51","Type":"ContainerDied","Data":"f5c011fb58d42aa970fe585d18b63bd0647ebf23a8e3eef187e13ee21222b56f"} Jul 21 21:39:27.898267 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:39:27.898135 2575 scope.go:117] "RemoveContainer" containerID="f6fa9e452477938adae3a9d7b3d4ddeacc1c2191e7fac571786f2d5ce772f8cd" Jul 21 21:39:27.909446 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:39:27.909423 2575 scope.go:117] "RemoveContainer" containerID="9910c2af718415f981f1e75eda8224e35d2c32554396e73052d17839e75172bf" Jul 21 21:39:27.918417 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:39:27.918396 2575 scope.go:117] "RemoveContainer" containerID="f6fa9e452477938adae3a9d7b3d4ddeacc1c2191e7fac571786f2d5ce772f8cd" Jul 21 21:39:27.918879 ip-10-0-140-97 kubenswrapper[2575]: E0721 21:39:27.918853 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f6fa9e452477938adae3a9d7b3d4ddeacc1c2191e7fac571786f2d5ce772f8cd\": container with ID starting with f6fa9e452477938adae3a9d7b3d4ddeacc1c2191e7fac571786f2d5ce772f8cd not found: ID does not exist" containerID="f6fa9e452477938adae3a9d7b3d4ddeacc1c2191e7fac571786f2d5ce772f8cd" Jul 21 21:39:27.918979 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:39:27.918890 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f6fa9e452477938adae3a9d7b3d4ddeacc1c2191e7fac571786f2d5ce772f8cd"} err="failed to get container status \"f6fa9e452477938adae3a9d7b3d4ddeacc1c2191e7fac571786f2d5ce772f8cd\": rpc error: code = NotFound desc = could not find container \"f6fa9e452477938adae3a9d7b3d4ddeacc1c2191e7fac571786f2d5ce772f8cd\": container with ID starting with f6fa9e452477938adae3a9d7b3d4ddeacc1c2191e7fac571786f2d5ce772f8cd not found: ID does not exist" Jul 21 21:39:27.918979 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:39:27.918912 2575 scope.go:117] "RemoveContainer" containerID="9910c2af718415f981f1e75eda8224e35d2c32554396e73052d17839e75172bf" Jul 21 21:39:27.919186 ip-10-0-140-97 kubenswrapper[2575]: E0721 21:39:27.919167 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9910c2af718415f981f1e75eda8224e35d2c32554396e73052d17839e75172bf\": container with ID starting with 9910c2af718415f981f1e75eda8224e35d2c32554396e73052d17839e75172bf not found: ID does not exist" containerID="9910c2af718415f981f1e75eda8224e35d2c32554396e73052d17839e75172bf" Jul 21 21:39:27.919233 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:39:27.919192 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9910c2af718415f981f1e75eda8224e35d2c32554396e73052d17839e75172bf"} err="failed to get container status \"9910c2af718415f981f1e75eda8224e35d2c32554396e73052d17839e75172bf\": rpc error: code = NotFound desc = could not find container \"9910c2af718415f981f1e75eda8224e35d2c32554396e73052d17839e75172bf\": container with ID starting with 9910c2af718415f981f1e75eda8224e35d2c32554396e73052d17839e75172bf not found: ID does not exist" Jul 21 21:39:27.920412 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:39:27.920392 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-226c3e-openshift-default-75654f7bc7-7dlg2"] Jul 21 21:39:27.924795 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:39:27.924772 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-226c3e-openshift-default-75654f7bc7-7dlg2"] Jul 21 21:39:28.740662 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:39:28.740630 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9b214bc0-2416-47ec-9eb7-3ff2a0caed51" path="/var/lib/kubelet/pods/9b214bc0-2416-47ec-9eb7-3ff2a0caed51/volumes" Jul 21 21:39:42.307441 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:39:42.307354 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-5898c4-4h9hd"] Jul 21 21:39:42.310174 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:39:42.310144 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-5898c4-4h9hd"] Jul 21 21:39:42.743148 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:39:42.743114 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b2691f37-18af-42ec-a441-0d7eee0442d3" path="/var/lib/kubelet/pods/b2691f37-18af-42ec-a441-0d7eee0442d3/volumes" Jul 21 21:39:48.763562 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:39:48.763500 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-5898c4-openshift-default-c87765695-vcj2j"] Jul 21 21:39:48.763968 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:39:48.763786 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-5898c4-openshift-default-c87765695-vcj2j" podUID="8e034e76-c6e3-48c6-ae67-8f2ffc04bc62" containerName="istio-proxy" containerID="cri-o://5e3ef35fc244a528f60136f25df645c2319164789191e03dc6acba6e3d135db8" gracePeriod=30 Jul 21 21:39:49.707678 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:39:49.707644 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-e584ac-openshift-default-5d8bd494c-sslp5"] Jul 21 21:39:49.708040 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:39:49.708028 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b2691f37-18af-42ec-a441-0d7eee0442d3" containerName="revoke-keys" Jul 21 21:39:49.708086 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:39:49.708042 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="b2691f37-18af-42ec-a441-0d7eee0442d3" containerName="revoke-keys" Jul 21 21:39:49.708086 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:39:49.708057 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9b214bc0-2416-47ec-9eb7-3ff2a0caed51" containerName="istio-proxy" Jul 21 21:39:49.708086 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:39:49.708063 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="9b214bc0-2416-47ec-9eb7-3ff2a0caed51" containerName="istio-proxy" Jul 21 21:39:49.708086 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:39:49.708073 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9b214bc0-2416-47ec-9eb7-3ff2a0caed51" containerName="istio-proxy" Jul 21 21:39:49.708086 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:39:49.708079 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="9b214bc0-2416-47ec-9eb7-3ff2a0caed51" containerName="istio-proxy" Jul 21 21:39:49.708242 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:39:49.708141 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="9b214bc0-2416-47ec-9eb7-3ff2a0caed51" containerName="istio-proxy" Jul 21 21:39:49.708242 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:39:49.708149 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="9b214bc0-2416-47ec-9eb7-3ff2a0caed51" containerName="istio-proxy" Jul 21 21:39:49.708242 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:39:49.708155 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="b2691f37-18af-42ec-a441-0d7eee0442d3" containerName="revoke-keys" Jul 21 21:39:49.711242 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:39:49.711222 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-e584ac-openshift-default-5d8bd494c-sslp5" Jul 21 21:39:49.714170 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:39:49.714147 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-e584ac-openshift-default-dockercfg-s2mhf\"" Jul 21 21:39:49.722606 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:39:49.722581 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-e584ac-openshift-default-5d8bd494c-sslp5"] Jul 21 21:39:49.872935 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:39:49.872895 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4f4a0c63-45f1-4f45-b038-c2e2433900f6-istio-envoy\") pod \"e2e-shared-a-e584ac-openshift-default-5d8bd494c-sslp5\" (UID: \"4f4a0c63-45f1-4f45-b038-c2e2433900f6\") " pod="openshift-ingress/e2e-shared-a-e584ac-openshift-default-5d8bd494c-sslp5" Jul 21 21:39:49.873444 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:39:49.873090 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h7llv\" (UniqueName: \"kubernetes.io/projected/4f4a0c63-45f1-4f45-b038-c2e2433900f6-kube-api-access-h7llv\") pod \"e2e-shared-a-e584ac-openshift-default-5d8bd494c-sslp5\" (UID: \"4f4a0c63-45f1-4f45-b038-c2e2433900f6\") " pod="openshift-ingress/e2e-shared-a-e584ac-openshift-default-5d8bd494c-sslp5" Jul 21 21:39:49.873444 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:39:49.873137 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4f4a0c63-45f1-4f45-b038-c2e2433900f6-istiod-ca-cert\") pod \"e2e-shared-a-e584ac-openshift-default-5d8bd494c-sslp5\" (UID: \"4f4a0c63-45f1-4f45-b038-c2e2433900f6\") " pod="openshift-ingress/e2e-shared-a-e584ac-openshift-default-5d8bd494c-sslp5" Jul 21 21:39:49.873444 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:39:49.873217 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4f4a0c63-45f1-4f45-b038-c2e2433900f6-istio-token\") pod \"e2e-shared-a-e584ac-openshift-default-5d8bd494c-sslp5\" (UID: \"4f4a0c63-45f1-4f45-b038-c2e2433900f6\") " pod="openshift-ingress/e2e-shared-a-e584ac-openshift-default-5d8bd494c-sslp5" Jul 21 21:39:49.873444 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:39:49.873274 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4f4a0c63-45f1-4f45-b038-c2e2433900f6-istio-data\") pod \"e2e-shared-a-e584ac-openshift-default-5d8bd494c-sslp5\" (UID: \"4f4a0c63-45f1-4f45-b038-c2e2433900f6\") " pod="openshift-ingress/e2e-shared-a-e584ac-openshift-default-5d8bd494c-sslp5" Jul 21 21:39:49.873444 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:39:49.873311 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f4a0c63-45f1-4f45-b038-c2e2433900f6-credential-socket\") pod \"e2e-shared-a-e584ac-openshift-default-5d8bd494c-sslp5\" (UID: \"4f4a0c63-45f1-4f45-b038-c2e2433900f6\") " pod="openshift-ingress/e2e-shared-a-e584ac-openshift-default-5d8bd494c-sslp5" Jul 21 21:39:49.873444 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:39:49.873375 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4f4a0c63-45f1-4f45-b038-c2e2433900f6-workload-certs\") pod \"e2e-shared-a-e584ac-openshift-default-5d8bd494c-sslp5\" (UID: \"4f4a0c63-45f1-4f45-b038-c2e2433900f6\") " pod="openshift-ingress/e2e-shared-a-e584ac-openshift-default-5d8bd494c-sslp5" Jul 21 21:39:49.873444 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:39:49.873406 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4f4a0c63-45f1-4f45-b038-c2e2433900f6-istio-podinfo\") pod \"e2e-shared-a-e584ac-openshift-default-5d8bd494c-sslp5\" (UID: \"4f4a0c63-45f1-4f45-b038-c2e2433900f6\") " pod="openshift-ingress/e2e-shared-a-e584ac-openshift-default-5d8bd494c-sslp5" Jul 21 21:39:49.873786 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:39:49.873481 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f4a0c63-45f1-4f45-b038-c2e2433900f6-workload-socket\") pod \"e2e-shared-a-e584ac-openshift-default-5d8bd494c-sslp5\" (UID: \"4f4a0c63-45f1-4f45-b038-c2e2433900f6\") " pod="openshift-ingress/e2e-shared-a-e584ac-openshift-default-5d8bd494c-sslp5" Jul 21 21:39:49.975010 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:39:49.974919 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4f4a0c63-45f1-4f45-b038-c2e2433900f6-istio-envoy\") pod \"e2e-shared-a-e584ac-openshift-default-5d8bd494c-sslp5\" (UID: \"4f4a0c63-45f1-4f45-b038-c2e2433900f6\") " pod="openshift-ingress/e2e-shared-a-e584ac-openshift-default-5d8bd494c-sslp5" Jul 21 21:39:49.975010 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:39:49.974972 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h7llv\" (UniqueName: \"kubernetes.io/projected/4f4a0c63-45f1-4f45-b038-c2e2433900f6-kube-api-access-h7llv\") pod \"e2e-shared-a-e584ac-openshift-default-5d8bd494c-sslp5\" (UID: \"4f4a0c63-45f1-4f45-b038-c2e2433900f6\") " pod="openshift-ingress/e2e-shared-a-e584ac-openshift-default-5d8bd494c-sslp5" Jul 21 21:39:49.975010 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:39:49.974993 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4f4a0c63-45f1-4f45-b038-c2e2433900f6-istiod-ca-cert\") pod \"e2e-shared-a-e584ac-openshift-default-5d8bd494c-sslp5\" (UID: \"4f4a0c63-45f1-4f45-b038-c2e2433900f6\") " pod="openshift-ingress/e2e-shared-a-e584ac-openshift-default-5d8bd494c-sslp5" Jul 21 21:39:49.975308 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:39:49.975026 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4f4a0c63-45f1-4f45-b038-c2e2433900f6-istio-token\") pod \"e2e-shared-a-e584ac-openshift-default-5d8bd494c-sslp5\" (UID: \"4f4a0c63-45f1-4f45-b038-c2e2433900f6\") " pod="openshift-ingress/e2e-shared-a-e584ac-openshift-default-5d8bd494c-sslp5" Jul 21 21:39:49.975308 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:39:49.975060 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4f4a0c63-45f1-4f45-b038-c2e2433900f6-istio-data\") pod \"e2e-shared-a-e584ac-openshift-default-5d8bd494c-sslp5\" (UID: \"4f4a0c63-45f1-4f45-b038-c2e2433900f6\") " pod="openshift-ingress/e2e-shared-a-e584ac-openshift-default-5d8bd494c-sslp5" Jul 21 21:39:49.975308 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:39:49.975081 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f4a0c63-45f1-4f45-b038-c2e2433900f6-credential-socket\") pod \"e2e-shared-a-e584ac-openshift-default-5d8bd494c-sslp5\" (UID: \"4f4a0c63-45f1-4f45-b038-c2e2433900f6\") " pod="openshift-ingress/e2e-shared-a-e584ac-openshift-default-5d8bd494c-sslp5" Jul 21 21:39:49.975308 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:39:49.975110 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4f4a0c63-45f1-4f45-b038-c2e2433900f6-workload-certs\") pod \"e2e-shared-a-e584ac-openshift-default-5d8bd494c-sslp5\" (UID: \"4f4a0c63-45f1-4f45-b038-c2e2433900f6\") " pod="openshift-ingress/e2e-shared-a-e584ac-openshift-default-5d8bd494c-sslp5" Jul 21 21:39:49.975522 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:39:49.975353 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4f4a0c63-45f1-4f45-b038-c2e2433900f6-istio-podinfo\") pod \"e2e-shared-a-e584ac-openshift-default-5d8bd494c-sslp5\" (UID: \"4f4a0c63-45f1-4f45-b038-c2e2433900f6\") " pod="openshift-ingress/e2e-shared-a-e584ac-openshift-default-5d8bd494c-sslp5" Jul 21 21:39:49.975522 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:39:49.975456 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f4a0c63-45f1-4f45-b038-c2e2433900f6-workload-socket\") pod \"e2e-shared-a-e584ac-openshift-default-5d8bd494c-sslp5\" (UID: \"4f4a0c63-45f1-4f45-b038-c2e2433900f6\") " pod="openshift-ingress/e2e-shared-a-e584ac-openshift-default-5d8bd494c-sslp5" Jul 21 21:39:49.975522 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:39:49.975487 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4f4a0c63-45f1-4f45-b038-c2e2433900f6-istio-data\") pod \"e2e-shared-a-e584ac-openshift-default-5d8bd494c-sslp5\" (UID: \"4f4a0c63-45f1-4f45-b038-c2e2433900f6\") " pod="openshift-ingress/e2e-shared-a-e584ac-openshift-default-5d8bd494c-sslp5" Jul 21 21:39:49.975701 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:39:49.975494 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4f4a0c63-45f1-4f45-b038-c2e2433900f6-workload-certs\") pod \"e2e-shared-a-e584ac-openshift-default-5d8bd494c-sslp5\" (UID: \"4f4a0c63-45f1-4f45-b038-c2e2433900f6\") " pod="openshift-ingress/e2e-shared-a-e584ac-openshift-default-5d8bd494c-sslp5" Jul 21 21:39:49.975788 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:39:49.975752 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f4a0c63-45f1-4f45-b038-c2e2433900f6-credential-socket\") pod \"e2e-shared-a-e584ac-openshift-default-5d8bd494c-sslp5\" (UID: \"4f4a0c63-45f1-4f45-b038-c2e2433900f6\") " pod="openshift-ingress/e2e-shared-a-e584ac-openshift-default-5d8bd494c-sslp5" Jul 21 21:39:49.975788 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:39:49.975767 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f4a0c63-45f1-4f45-b038-c2e2433900f6-workload-socket\") pod \"e2e-shared-a-e584ac-openshift-default-5d8bd494c-sslp5\" (UID: \"4f4a0c63-45f1-4f45-b038-c2e2433900f6\") " pod="openshift-ingress/e2e-shared-a-e584ac-openshift-default-5d8bd494c-sslp5" Jul 21 21:39:49.975953 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:39:49.975932 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4f4a0c63-45f1-4f45-b038-c2e2433900f6-istiod-ca-cert\") pod \"e2e-shared-a-e584ac-openshift-default-5d8bd494c-sslp5\" (UID: \"4f4a0c63-45f1-4f45-b038-c2e2433900f6\") " pod="openshift-ingress/e2e-shared-a-e584ac-openshift-default-5d8bd494c-sslp5" Jul 21 21:39:49.977317 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:39:49.977290 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4f4a0c63-45f1-4f45-b038-c2e2433900f6-istio-envoy\") pod \"e2e-shared-a-e584ac-openshift-default-5d8bd494c-sslp5\" (UID: \"4f4a0c63-45f1-4f45-b038-c2e2433900f6\") " pod="openshift-ingress/e2e-shared-a-e584ac-openshift-default-5d8bd494c-sslp5" Jul 21 21:39:49.977730 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:39:49.977714 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4f4a0c63-45f1-4f45-b038-c2e2433900f6-istio-podinfo\") pod \"e2e-shared-a-e584ac-openshift-default-5d8bd494c-sslp5\" (UID: \"4f4a0c63-45f1-4f45-b038-c2e2433900f6\") " pod="openshift-ingress/e2e-shared-a-e584ac-openshift-default-5d8bd494c-sslp5" Jul 21 21:39:49.985018 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:39:49.984980 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h7llv\" (UniqueName: \"kubernetes.io/projected/4f4a0c63-45f1-4f45-b038-c2e2433900f6-kube-api-access-h7llv\") pod \"e2e-shared-a-e584ac-openshift-default-5d8bd494c-sslp5\" (UID: \"4f4a0c63-45f1-4f45-b038-c2e2433900f6\") " pod="openshift-ingress/e2e-shared-a-e584ac-openshift-default-5d8bd494c-sslp5" Jul 21 21:39:49.985864 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:39:49.985840 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4f4a0c63-45f1-4f45-b038-c2e2433900f6-istio-token\") pod \"e2e-shared-a-e584ac-openshift-default-5d8bd494c-sslp5\" (UID: \"4f4a0c63-45f1-4f45-b038-c2e2433900f6\") " pod="openshift-ingress/e2e-shared-a-e584ac-openshift-default-5d8bd494c-sslp5" Jul 21 21:39:50.022735 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:39:50.022698 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-e584ac-openshift-default-5d8bd494c-sslp5" Jul 21 21:39:50.171636 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:39:50.171610 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-e584ac-openshift-default-5d8bd494c-sslp5"] Jul 21 21:39:50.174051 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:39:50.174017 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4f4a0c63_45f1_4f45_b038_c2e2433900f6.slice/crio-bc9f4f4cdad840b7308c9b0b8d5f933e2a6d58870ae06909b6c6390b91e13e3d WatchSource:0}: Error finding container bc9f4f4cdad840b7308c9b0b8d5f933e2a6d58870ae06909b6c6390b91e13e3d: Status 404 returned error can't find the container with id bc9f4f4cdad840b7308c9b0b8d5f933e2a6d58870ae06909b6c6390b91e13e3d Jul 21 21:39:50.176236 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:39:50.176190 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 21 21:39:50.176312 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:39:50.176294 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 21 21:39:50.176361 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:39:50.176345 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 21 21:39:51.016401 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:39:51.016355 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-e584ac-openshift-default-5d8bd494c-sslp5" event={"ID":"4f4a0c63-45f1-4f45-b038-c2e2433900f6","Type":"ContainerStarted","Data":"00e47cff51276e6433b46cce20e4f05f30947f56940c942df6f6792f5831402f"} Jul 21 21:39:51.016848 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:39:51.016420 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-e584ac-openshift-default-5d8bd494c-sslp5" event={"ID":"4f4a0c63-45f1-4f45-b038-c2e2433900f6","Type":"ContainerStarted","Data":"bc9f4f4cdad840b7308c9b0b8d5f933e2a6d58870ae06909b6c6390b91e13e3d"} Jul 21 21:39:51.023302 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:39:51.023279 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-e584ac-openshift-default-5d8bd494c-sslp5" Jul 21 21:39:51.028090 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:39:51.028065 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-e584ac-openshift-default-5d8bd494c-sslp5" Jul 21 21:39:51.040053 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:39:51.039994 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-e584ac-openshift-default-5d8bd494c-sslp5" podStartSLOduration=2.039975325 podStartE2EDuration="2.039975325s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 21:39:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 21:39:51.038280823 +0000 UTC m=+2348.973030490" watchObservedRunningTime="2026-07-21 21:39:51.039975325 +0000 UTC m=+2348.974724993" Jul 21 21:39:52.021313 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:39:52.021273 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-e584ac-openshift-default-5d8bd494c-sslp5" Jul 21 21:39:52.022232 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:39:52.022211 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-e584ac-openshift-default-5d8bd494c-sslp5" Jul 21 21:39:54.033764 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:39:54.033671 2575 generic.go:356] "Generic (PLEG): container finished" podID="8e034e76-c6e3-48c6-ae67-8f2ffc04bc62" containerID="5e3ef35fc244a528f60136f25df645c2319164789191e03dc6acba6e3d135db8" exitCode=0 Jul 21 21:39:54.033764 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:39:54.033743 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-5898c4-openshift-default-c87765695-vcj2j" event={"ID":"8e034e76-c6e3-48c6-ae67-8f2ffc04bc62","Type":"ContainerDied","Data":"5e3ef35fc244a528f60136f25df645c2319164789191e03dc6acba6e3d135db8"} Jul 21 21:39:54.034128 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:39:54.033840 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-5898c4-openshift-default-c87765695-vcj2j" event={"ID":"8e034e76-c6e3-48c6-ae67-8f2ffc04bc62","Type":"ContainerDied","Data":"c96d3877637184432c4f296ab48ee581f72679592cbc445e85e322d3910eec02"} Jul 21 21:39:54.034128 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:39:54.033865 2575 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c96d3877637184432c4f296ab48ee581f72679592cbc445e85e322d3910eec02" Jul 21 21:39:54.034128 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:39:54.033885 2575 scope.go:117] "RemoveContainer" containerID="d7a914984dda85e049baff9ff10b6f93397e85ff21397b31431f8568a2d0d6fa" Jul 21 21:39:54.036074 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:39:54.036053 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-5898c4-openshift-default-c87765695-vcj2j" Jul 21 21:39:54.119774 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:39:54.119738 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-x5vgc\" (UniqueName: \"kubernetes.io/projected/8e034e76-c6e3-48c6-ae67-8f2ffc04bc62-kube-api-access-x5vgc\") pod \"8e034e76-c6e3-48c6-ae67-8f2ffc04bc62\" (UID: \"8e034e76-c6e3-48c6-ae67-8f2ffc04bc62\") " Jul 21 21:39:54.119954 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:39:54.119787 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8e034e76-c6e3-48c6-ae67-8f2ffc04bc62-istio-token\") pod \"8e034e76-c6e3-48c6-ae67-8f2ffc04bc62\" (UID: \"8e034e76-c6e3-48c6-ae67-8f2ffc04bc62\") " Jul 21 21:39:54.119954 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:39:54.119818 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8e034e76-c6e3-48c6-ae67-8f2ffc04bc62-workload-certs\") pod \"8e034e76-c6e3-48c6-ae67-8f2ffc04bc62\" (UID: \"8e034e76-c6e3-48c6-ae67-8f2ffc04bc62\") " Jul 21 21:39:54.120041 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:39:54.120005 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8e034e76-c6e3-48c6-ae67-8f2ffc04bc62-istio-podinfo\") pod \"8e034e76-c6e3-48c6-ae67-8f2ffc04bc62\" (UID: \"8e034e76-c6e3-48c6-ae67-8f2ffc04bc62\") " Jul 21 21:39:54.120098 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:39:54.120078 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8e034e76-c6e3-48c6-ae67-8f2ffc04bc62-istio-envoy\") pod \"8e034e76-c6e3-48c6-ae67-8f2ffc04bc62\" (UID: \"8e034e76-c6e3-48c6-ae67-8f2ffc04bc62\") " Jul 21 21:39:54.120155 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:39:54.120102 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e034e76-c6e3-48c6-ae67-8f2ffc04bc62-credential-socket\") pod \"8e034e76-c6e3-48c6-ae67-8f2ffc04bc62\" (UID: \"8e034e76-c6e3-48c6-ae67-8f2ffc04bc62\") " Jul 21 21:39:54.120155 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:39:54.120127 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e034e76-c6e3-48c6-ae67-8f2ffc04bc62-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "8e034e76-c6e3-48c6-ae67-8f2ffc04bc62" (UID: "8e034e76-c6e3-48c6-ae67-8f2ffc04bc62"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 21:39:54.120155 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:39:54.120140 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e034e76-c6e3-48c6-ae67-8f2ffc04bc62-workload-socket\") pod \"8e034e76-c6e3-48c6-ae67-8f2ffc04bc62\" (UID: \"8e034e76-c6e3-48c6-ae67-8f2ffc04bc62\") " Jul 21 21:39:54.120311 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:39:54.120212 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8e034e76-c6e3-48c6-ae67-8f2ffc04bc62-istio-data\") pod \"8e034e76-c6e3-48c6-ae67-8f2ffc04bc62\" (UID: \"8e034e76-c6e3-48c6-ae67-8f2ffc04bc62\") " Jul 21 21:39:54.120792 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:39:54.120358 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e034e76-c6e3-48c6-ae67-8f2ffc04bc62-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "8e034e76-c6e3-48c6-ae67-8f2ffc04bc62" (UID: "8e034e76-c6e3-48c6-ae67-8f2ffc04bc62"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 21:39:54.120792 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:39:54.120394 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e034e76-c6e3-48c6-ae67-8f2ffc04bc62-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "8e034e76-c6e3-48c6-ae67-8f2ffc04bc62" (UID: "8e034e76-c6e3-48c6-ae67-8f2ffc04bc62"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 21:39:54.120792 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:39:54.120730 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8e034e76-c6e3-48c6-ae67-8f2ffc04bc62-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "8e034e76-c6e3-48c6-ae67-8f2ffc04bc62" (UID: "8e034e76-c6e3-48c6-ae67-8f2ffc04bc62"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 21 21:39:54.121086 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:39:54.121070 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8e034e76-c6e3-48c6-ae67-8f2ffc04bc62-istiod-ca-cert\") pod \"8e034e76-c6e3-48c6-ae67-8f2ffc04bc62\" (UID: \"8e034e76-c6e3-48c6-ae67-8f2ffc04bc62\") " Jul 21 21:39:54.121216 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:39:54.121189 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e034e76-c6e3-48c6-ae67-8f2ffc04bc62-istio-data" (OuterVolumeSpecName: "istio-data") pod "8e034e76-c6e3-48c6-ae67-8f2ffc04bc62" (UID: "8e034e76-c6e3-48c6-ae67-8f2ffc04bc62"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 21:39:54.121748 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:39:54.121583 2575 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8e034e76-c6e3-48c6-ae67-8f2ffc04bc62-workload-certs\") on node \"ip-10-0-140-97.ec2.internal\" DevicePath \"\"" Jul 21 21:39:54.121748 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:39:54.121609 2575 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e034e76-c6e3-48c6-ae67-8f2ffc04bc62-credential-socket\") on node \"ip-10-0-140-97.ec2.internal\" DevicePath \"\"" Jul 21 21:39:54.121748 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:39:54.121624 2575 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e034e76-c6e3-48c6-ae67-8f2ffc04bc62-workload-socket\") on node \"ip-10-0-140-97.ec2.internal\" DevicePath \"\"" Jul 21 21:39:54.121748 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:39:54.121638 2575 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8e034e76-c6e3-48c6-ae67-8f2ffc04bc62-istio-data\") on node \"ip-10-0-140-97.ec2.internal\" DevicePath \"\"" Jul 21 21:39:54.121748 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:39:54.121650 2575 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8e034e76-c6e3-48c6-ae67-8f2ffc04bc62-istiod-ca-cert\") on node \"ip-10-0-140-97.ec2.internal\" DevicePath \"\"" Jul 21 21:39:54.122502 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:39:54.122475 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e034e76-c6e3-48c6-ae67-8f2ffc04bc62-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "8e034e76-c6e3-48c6-ae67-8f2ffc04bc62" (UID: "8e034e76-c6e3-48c6-ae67-8f2ffc04bc62"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 21:39:54.122502 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:39:54.122496 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8e034e76-c6e3-48c6-ae67-8f2ffc04bc62-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "8e034e76-c6e3-48c6-ae67-8f2ffc04bc62" (UID: "8e034e76-c6e3-48c6-ae67-8f2ffc04bc62"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 21 21:39:54.122687 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:39:54.122590 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8e034e76-c6e3-48c6-ae67-8f2ffc04bc62-istio-token" (OuterVolumeSpecName: "istio-token") pod "8e034e76-c6e3-48c6-ae67-8f2ffc04bc62" (UID: "8e034e76-c6e3-48c6-ae67-8f2ffc04bc62"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 21 21:39:54.122687 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:39:54.122592 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8e034e76-c6e3-48c6-ae67-8f2ffc04bc62-kube-api-access-x5vgc" (OuterVolumeSpecName: "kube-api-access-x5vgc") pod "8e034e76-c6e3-48c6-ae67-8f2ffc04bc62" (UID: "8e034e76-c6e3-48c6-ae67-8f2ffc04bc62"). InnerVolumeSpecName "kube-api-access-x5vgc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 21 21:39:54.222380 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:39:54.222338 2575 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8e034e76-c6e3-48c6-ae67-8f2ffc04bc62-istio-token\") on node \"ip-10-0-140-97.ec2.internal\" DevicePath \"\"" Jul 21 21:39:54.222380 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:39:54.222372 2575 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8e034e76-c6e3-48c6-ae67-8f2ffc04bc62-istio-podinfo\") on node \"ip-10-0-140-97.ec2.internal\" DevicePath \"\"" Jul 21 21:39:54.222380 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:39:54.222383 2575 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8e034e76-c6e3-48c6-ae67-8f2ffc04bc62-istio-envoy\") on node \"ip-10-0-140-97.ec2.internal\" DevicePath \"\"" Jul 21 21:39:54.222380 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:39:54.222392 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x5vgc\" (UniqueName: \"kubernetes.io/projected/8e034e76-c6e3-48c6-ae67-8f2ffc04bc62-kube-api-access-x5vgc\") on node \"ip-10-0-140-97.ec2.internal\" DevicePath \"\"" Jul 21 21:39:55.040755 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:39:55.040674 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-5898c4-openshift-default-c87765695-vcj2j" Jul 21 21:39:55.074265 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:39:55.074227 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-5898c4-openshift-default-c87765695-vcj2j"] Jul 21 21:39:55.079609 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:39:55.079576 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-5898c4-openshift-default-c87765695-vcj2j"] Jul 21 21:39:56.740805 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:39:56.740769 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8e034e76-c6e3-48c6-ae67-8f2ffc04bc62" path="/var/lib/kubelet/pods/8e034e76-c6e3-48c6-ae67-8f2ffc04bc62/volumes" Jul 21 21:39:58.139258 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:39:58.139221 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-24ec74-openshift-default-58d5566c95-chm82"] Jul 21 21:39:58.139827 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:39:58.139807 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8e034e76-c6e3-48c6-ae67-8f2ffc04bc62" containerName="istio-proxy" Jul 21 21:39:58.139923 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:39:58.139831 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="8e034e76-c6e3-48c6-ae67-8f2ffc04bc62" containerName="istio-proxy" Jul 21 21:39:58.139979 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:39:58.139964 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="8e034e76-c6e3-48c6-ae67-8f2ffc04bc62" containerName="istio-proxy" Jul 21 21:39:58.144632 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:39:58.144600 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-24ec74-openshift-default-58d5566c95-chm82" Jul 21 21:39:58.147428 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:39:58.147404 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-24ec74-openshift-default-dockercfg-fklgb\"" Jul 21 21:39:58.159834 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:39:58.159797 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-24ec74-openshift-default-58d5566c95-chm82"] Jul 21 21:39:58.264211 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:39:58.264161 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d6099e69-dcca-4458-a948-29493d41605f-istio-podinfo\") pod \"e2e-shared-b-24ec74-openshift-default-58d5566c95-chm82\" (UID: \"d6099e69-dcca-4458-a948-29493d41605f\") " pod="openshift-ingress/e2e-shared-b-24ec74-openshift-default-58d5566c95-chm82" Jul 21 21:39:58.264418 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:39:58.264223 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6099e69-dcca-4458-a948-29493d41605f-credential-socket\") pod \"e2e-shared-b-24ec74-openshift-default-58d5566c95-chm82\" (UID: \"d6099e69-dcca-4458-a948-29493d41605f\") " pod="openshift-ingress/e2e-shared-b-24ec74-openshift-default-58d5566c95-chm82" Jul 21 21:39:58.264418 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:39:58.264282 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d6099e69-dcca-4458-a948-29493d41605f-istio-data\") pod \"e2e-shared-b-24ec74-openshift-default-58d5566c95-chm82\" (UID: \"d6099e69-dcca-4458-a948-29493d41605f\") " pod="openshift-ingress/e2e-shared-b-24ec74-openshift-default-58d5566c95-chm82" Jul 21 21:39:58.264418 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:39:58.264319 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d6099e69-dcca-4458-a948-29493d41605f-istio-token\") pod \"e2e-shared-b-24ec74-openshift-default-58d5566c95-chm82\" (UID: \"d6099e69-dcca-4458-a948-29493d41605f\") " pod="openshift-ingress/e2e-shared-b-24ec74-openshift-default-58d5566c95-chm82" Jul 21 21:39:58.264625 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:39:58.264421 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6099e69-dcca-4458-a948-29493d41605f-workload-socket\") pod \"e2e-shared-b-24ec74-openshift-default-58d5566c95-chm82\" (UID: \"d6099e69-dcca-4458-a948-29493d41605f\") " pod="openshift-ingress/e2e-shared-b-24ec74-openshift-default-58d5566c95-chm82" Jul 21 21:39:58.264625 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:39:58.264460 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d6099e69-dcca-4458-a948-29493d41605f-istio-envoy\") pod \"e2e-shared-b-24ec74-openshift-default-58d5566c95-chm82\" (UID: \"d6099e69-dcca-4458-a948-29493d41605f\") " pod="openshift-ingress/e2e-shared-b-24ec74-openshift-default-58d5566c95-chm82" Jul 21 21:39:58.264625 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:39:58.264489 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dddwv\" (UniqueName: \"kubernetes.io/projected/d6099e69-dcca-4458-a948-29493d41605f-kube-api-access-dddwv\") pod \"e2e-shared-b-24ec74-openshift-default-58d5566c95-chm82\" (UID: \"d6099e69-dcca-4458-a948-29493d41605f\") " pod="openshift-ingress/e2e-shared-b-24ec74-openshift-default-58d5566c95-chm82" Jul 21 21:39:58.264625 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:39:58.264516 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d6099e69-dcca-4458-a948-29493d41605f-istiod-ca-cert\") pod \"e2e-shared-b-24ec74-openshift-default-58d5566c95-chm82\" (UID: \"d6099e69-dcca-4458-a948-29493d41605f\") " pod="openshift-ingress/e2e-shared-b-24ec74-openshift-default-58d5566c95-chm82" Jul 21 21:39:58.264625 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:39:58.264573 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d6099e69-dcca-4458-a948-29493d41605f-workload-certs\") pod \"e2e-shared-b-24ec74-openshift-default-58d5566c95-chm82\" (UID: \"d6099e69-dcca-4458-a948-29493d41605f\") " pod="openshift-ingress/e2e-shared-b-24ec74-openshift-default-58d5566c95-chm82" Jul 21 21:39:58.365396 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:39:58.365349 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d6099e69-dcca-4458-a948-29493d41605f-istio-envoy\") pod \"e2e-shared-b-24ec74-openshift-default-58d5566c95-chm82\" (UID: \"d6099e69-dcca-4458-a948-29493d41605f\") " pod="openshift-ingress/e2e-shared-b-24ec74-openshift-default-58d5566c95-chm82" Jul 21 21:39:58.365657 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:39:58.365636 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dddwv\" (UniqueName: \"kubernetes.io/projected/d6099e69-dcca-4458-a948-29493d41605f-kube-api-access-dddwv\") pod \"e2e-shared-b-24ec74-openshift-default-58d5566c95-chm82\" (UID: \"d6099e69-dcca-4458-a948-29493d41605f\") " pod="openshift-ingress/e2e-shared-b-24ec74-openshift-default-58d5566c95-chm82" Jul 21 21:39:58.365804 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:39:58.365785 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d6099e69-dcca-4458-a948-29493d41605f-istiod-ca-cert\") pod \"e2e-shared-b-24ec74-openshift-default-58d5566c95-chm82\" (UID: \"d6099e69-dcca-4458-a948-29493d41605f\") " pod="openshift-ingress/e2e-shared-b-24ec74-openshift-default-58d5566c95-chm82" Jul 21 21:39:58.365935 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:39:58.365921 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d6099e69-dcca-4458-a948-29493d41605f-workload-certs\") pod \"e2e-shared-b-24ec74-openshift-default-58d5566c95-chm82\" (UID: \"d6099e69-dcca-4458-a948-29493d41605f\") " pod="openshift-ingress/e2e-shared-b-24ec74-openshift-default-58d5566c95-chm82" Jul 21 21:39:58.366059 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:39:58.366045 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d6099e69-dcca-4458-a948-29493d41605f-istio-podinfo\") pod \"e2e-shared-b-24ec74-openshift-default-58d5566c95-chm82\" (UID: \"d6099e69-dcca-4458-a948-29493d41605f\") " pod="openshift-ingress/e2e-shared-b-24ec74-openshift-default-58d5566c95-chm82" Jul 21 21:39:58.366167 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:39:58.366154 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6099e69-dcca-4458-a948-29493d41605f-credential-socket\") pod \"e2e-shared-b-24ec74-openshift-default-58d5566c95-chm82\" (UID: \"d6099e69-dcca-4458-a948-29493d41605f\") " pod="openshift-ingress/e2e-shared-b-24ec74-openshift-default-58d5566c95-chm82" Jul 21 21:39:58.366299 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:39:58.366286 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d6099e69-dcca-4458-a948-29493d41605f-istio-data\") pod \"e2e-shared-b-24ec74-openshift-default-58d5566c95-chm82\" (UID: \"d6099e69-dcca-4458-a948-29493d41605f\") " pod="openshift-ingress/e2e-shared-b-24ec74-openshift-default-58d5566c95-chm82" Jul 21 21:39:58.366446 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:39:58.366430 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d6099e69-dcca-4458-a948-29493d41605f-istio-token\") pod \"e2e-shared-b-24ec74-openshift-default-58d5566c95-chm82\" (UID: \"d6099e69-dcca-4458-a948-29493d41605f\") " pod="openshift-ingress/e2e-shared-b-24ec74-openshift-default-58d5566c95-chm82" Jul 21 21:39:58.366666 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:39:58.366650 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6099e69-dcca-4458-a948-29493d41605f-workload-socket\") pod \"e2e-shared-b-24ec74-openshift-default-58d5566c95-chm82\" (UID: \"d6099e69-dcca-4458-a948-29493d41605f\") " pod="openshift-ingress/e2e-shared-b-24ec74-openshift-default-58d5566c95-chm82" Jul 21 21:39:58.366812 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:39:58.366707 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d6099e69-dcca-4458-a948-29493d41605f-istio-data\") pod \"e2e-shared-b-24ec74-openshift-default-58d5566c95-chm82\" (UID: \"d6099e69-dcca-4458-a948-29493d41605f\") " pod="openshift-ingress/e2e-shared-b-24ec74-openshift-default-58d5566c95-chm82" Jul 21 21:39:58.367814 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:39:58.367288 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d6099e69-dcca-4458-a948-29493d41605f-workload-certs\") pod \"e2e-shared-b-24ec74-openshift-default-58d5566c95-chm82\" (UID: \"d6099e69-dcca-4458-a948-29493d41605f\") " pod="openshift-ingress/e2e-shared-b-24ec74-openshift-default-58d5566c95-chm82" Jul 21 21:39:58.367814 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:39:58.367686 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6099e69-dcca-4458-a948-29493d41605f-credential-socket\") pod \"e2e-shared-b-24ec74-openshift-default-58d5566c95-chm82\" (UID: \"d6099e69-dcca-4458-a948-29493d41605f\") " pod="openshift-ingress/e2e-shared-b-24ec74-openshift-default-58d5566c95-chm82" Jul 21 21:39:58.367814 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:39:58.367759 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d6099e69-dcca-4458-a948-29493d41605f-istiod-ca-cert\") pod \"e2e-shared-b-24ec74-openshift-default-58d5566c95-chm82\" (UID: \"d6099e69-dcca-4458-a948-29493d41605f\") " pod="openshift-ingress/e2e-shared-b-24ec74-openshift-default-58d5566c95-chm82" Jul 21 21:39:58.368028 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:39:58.367941 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6099e69-dcca-4458-a948-29493d41605f-workload-socket\") pod \"e2e-shared-b-24ec74-openshift-default-58d5566c95-chm82\" (UID: \"d6099e69-dcca-4458-a948-29493d41605f\") " pod="openshift-ingress/e2e-shared-b-24ec74-openshift-default-58d5566c95-chm82" Jul 21 21:39:58.368292 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:39:58.368269 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d6099e69-dcca-4458-a948-29493d41605f-istio-envoy\") pod \"e2e-shared-b-24ec74-openshift-default-58d5566c95-chm82\" (UID: \"d6099e69-dcca-4458-a948-29493d41605f\") " pod="openshift-ingress/e2e-shared-b-24ec74-openshift-default-58d5566c95-chm82" Jul 21 21:39:58.370294 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:39:58.370276 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d6099e69-dcca-4458-a948-29493d41605f-istio-podinfo\") pod \"e2e-shared-b-24ec74-openshift-default-58d5566c95-chm82\" (UID: \"d6099e69-dcca-4458-a948-29493d41605f\") " pod="openshift-ingress/e2e-shared-b-24ec74-openshift-default-58d5566c95-chm82" Jul 21 21:39:58.374748 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:39:58.374727 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dddwv\" (UniqueName: \"kubernetes.io/projected/d6099e69-dcca-4458-a948-29493d41605f-kube-api-access-dddwv\") pod \"e2e-shared-b-24ec74-openshift-default-58d5566c95-chm82\" (UID: \"d6099e69-dcca-4458-a948-29493d41605f\") " pod="openshift-ingress/e2e-shared-b-24ec74-openshift-default-58d5566c95-chm82" Jul 21 21:39:58.374925 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:39:58.374907 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d6099e69-dcca-4458-a948-29493d41605f-istio-token\") pod \"e2e-shared-b-24ec74-openshift-default-58d5566c95-chm82\" (UID: \"d6099e69-dcca-4458-a948-29493d41605f\") " pod="openshift-ingress/e2e-shared-b-24ec74-openshift-default-58d5566c95-chm82" Jul 21 21:39:58.455906 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:39:58.455521 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-24ec74-openshift-default-58d5566c95-chm82" Jul 21 21:39:58.632418 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:39:58.632384 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-24ec74-openshift-default-58d5566c95-chm82"] Jul 21 21:39:58.633853 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:39:58.633819 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd6099e69_dcca_4458_a948_29493d41605f.slice/crio-02e3e63fb47253900348a79ab9251b8a9731405518429714964bb106925ed5bf WatchSource:0}: Error finding container 02e3e63fb47253900348a79ab9251b8a9731405518429714964bb106925ed5bf: Status 404 returned error can't find the container with id 02e3e63fb47253900348a79ab9251b8a9731405518429714964bb106925ed5bf Jul 21 21:39:58.635943 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:39:58.635909 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 21 21:39:58.636025 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:39:58.635978 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 21 21:39:58.636025 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:39:58.636008 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 21 21:39:59.069815 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:39:59.069709 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-24ec74-openshift-default-58d5566c95-chm82" event={"ID":"d6099e69-dcca-4458-a948-29493d41605f","Type":"ContainerStarted","Data":"2c7a272169452bb4d238b2e59dd9df6a920e179a53c23109a606877d1736dfde"} Jul 21 21:39:59.069969 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:39:59.069816 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-24ec74-openshift-default-58d5566c95-chm82" event={"ID":"d6099e69-dcca-4458-a948-29493d41605f","Type":"ContainerStarted","Data":"02e3e63fb47253900348a79ab9251b8a9731405518429714964bb106925ed5bf"} Jul 21 21:39:59.091360 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:39:59.091307 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-24ec74-openshift-default-58d5566c95-chm82" podStartSLOduration=1.091289803 podStartE2EDuration="1.091289803s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 21:39:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 21:39:59.088839469 +0000 UTC m=+2357.023589179" watchObservedRunningTime="2026-07-21 21:39:59.091289803 +0000 UTC m=+2357.026039468" Jul 21 21:39:59.456712 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:39:59.456666 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-24ec74-openshift-default-58d5566c95-chm82" Jul 21 21:39:59.468066 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:39:59.468028 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-24ec74-openshift-default-58d5566c95-chm82" Jul 21 21:40:00.075944 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:40:00.075909 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-24ec74-openshift-default-58d5566c95-chm82" Jul 21 21:40:00.076906 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:40:00.076882 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-24ec74-openshift-default-58d5566c95-chm82" Jul 21 21:40:05.106798 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:40:05.106761 2575 generic.go:356] "Generic (PLEG): container finished" podID="4f4a0c63-45f1-4f45-b038-c2e2433900f6" containerID="00e47cff51276e6433b46cce20e4f05f30947f56940c942df6f6792f5831402f" exitCode=0 Jul 21 21:40:05.107296 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:40:05.106848 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-e584ac-openshift-default-5d8bd494c-sslp5" event={"ID":"4f4a0c63-45f1-4f45-b038-c2e2433900f6","Type":"ContainerDied","Data":"00e47cff51276e6433b46cce20e4f05f30947f56940c942df6f6792f5831402f"} Jul 21 21:40:05.107296 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:40:05.107265 2575 scope.go:117] "RemoveContainer" containerID="00e47cff51276e6433b46cce20e4f05f30947f56940c942df6f6792f5831402f" Jul 21 21:40:05.108348 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:40:05.108291 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 21 21:40:05.108418 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:40:05.108370 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 21 21:40:05.108418 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:40:05.108403 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 21 21:40:06.114424 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:40:06.114392 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-e584ac-openshift-default-5d8bd494c-sslp5" event={"ID":"4f4a0c63-45f1-4f45-b038-c2e2433900f6","Type":"ContainerStarted","Data":"fb6cad868ca2384d672a281e4adc5bdfbe09d88cf7147e7a595207177e4d18bf"} Jul 21 21:40:07.021806 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:40:07.021766 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-e584ac-openshift-default-5d8bd494c-sslp5" Jul 21 21:40:07.023055 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:40:07.023031 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-e584ac-openshift-default-5d8bd494c-sslp5" Jul 21 21:40:07.024217 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:40:07.024189 2575 patch_prober.go:28] interesting pod/e2e-shared-a-e584ac-openshift-default-5d8bd494c-sslp5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" start-of-body= Jul 21 21:40:07.024325 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:40:07.024237 2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-e584ac-openshift-default-5d8bd494c-sslp5" podUID="4f4a0c63-45f1-4f45-b038-c2e2433900f6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" Jul 21 21:40:08.023521 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:40:08.023480 2575 patch_prober.go:28] interesting pod/e2e-shared-a-e584ac-openshift-default-5d8bd494c-sslp5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" start-of-body= Jul 21 21:40:08.023943 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:40:08.023567 2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-e584ac-openshift-default-5d8bd494c-sslp5" podUID="4f4a0c63-45f1-4f45-b038-c2e2433900f6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" Jul 21 21:40:09.023915 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:40:09.023872 2575 patch_prober.go:28] interesting pod/e2e-shared-a-e584ac-openshift-default-5d8bd494c-sslp5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" start-of-body= Jul 21 21:40:09.024331 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:40:09.023939 2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-e584ac-openshift-default-5d8bd494c-sslp5" podUID="4f4a0c63-45f1-4f45-b038-c2e2433900f6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" Jul 21 21:40:10.023617 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:40:10.023569 2575 patch_prober.go:28] interesting pod/e2e-shared-a-e584ac-openshift-default-5d8bd494c-sslp5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" start-of-body= Jul 21 21:40:10.023805 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:40:10.023695 2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-e584ac-openshift-default-5d8bd494c-sslp5" podUID="4f4a0c63-45f1-4f45-b038-c2e2433900f6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" Jul 21 21:40:11.027325 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:40:11.027292 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-e584ac-openshift-default-5d8bd494c-sslp5" Jul 21 21:40:11.028361 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:40:11.028340 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-e584ac-openshift-default-5d8bd494c-sslp5" Jul 21 21:40:13.156467 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:40:13.156435 2575 generic.go:356] "Generic (PLEG): container finished" podID="d6099e69-dcca-4458-a948-29493d41605f" containerID="2c7a272169452bb4d238b2e59dd9df6a920e179a53c23109a606877d1736dfde" exitCode=0 Jul 21 21:40:13.156896 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:40:13.156509 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-24ec74-openshift-default-58d5566c95-chm82" event={"ID":"d6099e69-dcca-4458-a948-29493d41605f","Type":"ContainerDied","Data":"2c7a272169452bb4d238b2e59dd9df6a920e179a53c23109a606877d1736dfde"} Jul 21 21:40:13.156953 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:40:13.156939 2575 scope.go:117] "RemoveContainer" containerID="2c7a272169452bb4d238b2e59dd9df6a920e179a53c23109a606877d1736dfde" Jul 21 21:40:13.158015 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:40:13.157984 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 21 21:40:13.158088 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:40:13.158072 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 21 21:40:13.158124 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:40:13.158115 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 21 21:40:14.168636 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:40:14.167663 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-24ec74-openshift-default-58d5566c95-chm82" event={"ID":"d6099e69-dcca-4458-a948-29493d41605f","Type":"ContainerStarted","Data":"73aaeebdf509e0530e7116e04e1ca9cf9b3597c17cc517918c8ccc27bc7a0551"} Jul 21 21:40:14.457901 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:40:14.457334 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-24ec74-openshift-default-58d5566c95-chm82" Jul 21 21:40:14.457901 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:40:14.457478 2575 patch_prober.go:28] interesting pod/e2e-shared-b-24ec74-openshift-default-58d5566c95-chm82 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body= Jul 21 21:40:14.457901 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:40:14.457549 2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-24ec74-openshift-default-58d5566c95-chm82" podUID="d6099e69-dcca-4458-a948-29493d41605f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" Jul 21 21:40:15.076640 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:40:15.076595 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-24ec74-openshift-default-58d5566c95-chm82" Jul 21 21:40:15.457241 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:40:15.456729 2575 patch_prober.go:28] interesting pod/e2e-shared-b-24ec74-openshift-default-58d5566c95-chm82 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body= Jul 21 21:40:15.457241 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:40:15.456816 2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-24ec74-openshift-default-58d5566c95-chm82" podUID="d6099e69-dcca-4458-a948-29493d41605f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" Jul 21 21:40:16.457309 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:40:16.457267 2575 patch_prober.go:28] interesting pod/e2e-shared-b-24ec74-openshift-default-58d5566c95-chm82 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body= Jul 21 21:40:16.457930 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:40:16.457899 2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-24ec74-openshift-default-58d5566c95-chm82" podUID="d6099e69-dcca-4458-a948-29493d41605f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" Jul 21 21:40:17.457392 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:40:17.457350 2575 patch_prober.go:28] interesting pod/e2e-shared-b-24ec74-openshift-default-58d5566c95-chm82 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body= Jul 21 21:40:17.458001 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:40:17.457966 2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-24ec74-openshift-default-58d5566c95-chm82" podUID="d6099e69-dcca-4458-a948-29493d41605f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" Jul 21 21:40:18.461277 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:40:18.461242 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-24ec74-openshift-default-58d5566c95-chm82" Jul 21 21:40:18.462442 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:40:18.462418 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-24ec74-openshift-default-58d5566c95-chm82" Jul 21 21:40:42.818731 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:40:42.818702 2575 scope.go:117] "RemoveContainer" containerID="0ca95462f3794f500fab6414e73ee23af4be57fb468300187a0ac19ee010e348" Jul 21 21:40:53.141507 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:40:53.141473 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-24ec74/auth-test-model-24ec74-kserve-544cd55f55-tslnz"] Jul 21 21:40:53.142117 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:40:53.142083 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8e034e76-c6e3-48c6-ae67-8f2ffc04bc62" containerName="istio-proxy" Jul 21 21:40:53.142117 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:40:53.142109 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="8e034e76-c6e3-48c6-ae67-8f2ffc04bc62" containerName="istio-proxy" Jul 21 21:40:53.142334 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:40:53.142214 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="8e034e76-c6e3-48c6-ae67-8f2ffc04bc62" containerName="istio-proxy" Jul 21 21:40:53.144269 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:40:53.144251 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-24ec74/auth-test-model-24ec74-kserve-544cd55f55-tslnz" Jul 21 21:40:53.148005 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:40:53.147982 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-24ec74\"/\"openshift-service-ca.crt\"" Jul 21 21:40:53.148165 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:40:53.148039 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-24ec74\"/\"kube-root-ca.crt\"" Jul 21 21:40:53.148289 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:40:53.148044 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-24ec74\"/\"auth-test-model-24ec74-kserve-self-signed-certs\"" Jul 21 21:40:53.148289 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:40:53.148065 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-24ec74\"/\"default-dockercfg-4x6vd\"" Jul 21 21:40:53.158944 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:40:53.158921 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-24ec74/auth-test-model-24ec74-kserve-544cd55f55-tslnz"] Jul 21 21:40:53.207702 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:40:53.207673 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7e008490-1285-461c-80a1-6aa0295f7f76-model-cache\") pod \"auth-test-model-24ec74-kserve-544cd55f55-tslnz\" (UID: \"7e008490-1285-461c-80a1-6aa0295f7f76\") " pod="ai-tenant-e2e-shared-b-24ec74/auth-test-model-24ec74-kserve-544cd55f55-tslnz" Jul 21 21:40:53.207850 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:40:53.207707 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7e008490-1285-461c-80a1-6aa0295f7f76-dshm\") pod \"auth-test-model-24ec74-kserve-544cd55f55-tslnz\" (UID: \"7e008490-1285-461c-80a1-6aa0295f7f76\") " pod="ai-tenant-e2e-shared-b-24ec74/auth-test-model-24ec74-kserve-544cd55f55-tslnz" Jul 21 21:40:53.207850 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:40:53.207742 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-khqxw\" (UniqueName: \"kubernetes.io/projected/7e008490-1285-461c-80a1-6aa0295f7f76-kube-api-access-khqxw\") pod \"auth-test-model-24ec74-kserve-544cd55f55-tslnz\" (UID: \"7e008490-1285-461c-80a1-6aa0295f7f76\") " pod="ai-tenant-e2e-shared-b-24ec74/auth-test-model-24ec74-kserve-544cd55f55-tslnz" Jul 21 21:40:53.207850 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:40:53.207795 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7e008490-1285-461c-80a1-6aa0295f7f76-tmp-dir\") pod \"auth-test-model-24ec74-kserve-544cd55f55-tslnz\" (UID: \"7e008490-1285-461c-80a1-6aa0295f7f76\") " pod="ai-tenant-e2e-shared-b-24ec74/auth-test-model-24ec74-kserve-544cd55f55-tslnz" Jul 21 21:40:53.207961 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:40:53.207881 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7e008490-1285-461c-80a1-6aa0295f7f76-home\") pod \"auth-test-model-24ec74-kserve-544cd55f55-tslnz\" (UID: \"7e008490-1285-461c-80a1-6aa0295f7f76\") " pod="ai-tenant-e2e-shared-b-24ec74/auth-test-model-24ec74-kserve-544cd55f55-tslnz" Jul 21 21:40:53.207961 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:40:53.207916 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7e008490-1285-461c-80a1-6aa0295f7f76-tls-certs\") pod \"auth-test-model-24ec74-kserve-544cd55f55-tslnz\" (UID: \"7e008490-1285-461c-80a1-6aa0295f7f76\") " pod="ai-tenant-e2e-shared-b-24ec74/auth-test-model-24ec74-kserve-544cd55f55-tslnz" Jul 21 21:40:53.309030 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:40:53.308993 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-khqxw\" (UniqueName: \"kubernetes.io/projected/7e008490-1285-461c-80a1-6aa0295f7f76-kube-api-access-khqxw\") pod \"auth-test-model-24ec74-kserve-544cd55f55-tslnz\" (UID: \"7e008490-1285-461c-80a1-6aa0295f7f76\") " pod="ai-tenant-e2e-shared-b-24ec74/auth-test-model-24ec74-kserve-544cd55f55-tslnz" Jul 21 21:40:53.309180 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:40:53.309047 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7e008490-1285-461c-80a1-6aa0295f7f76-tmp-dir\") pod \"auth-test-model-24ec74-kserve-544cd55f55-tslnz\" (UID: \"7e008490-1285-461c-80a1-6aa0295f7f76\") " pod="ai-tenant-e2e-shared-b-24ec74/auth-test-model-24ec74-kserve-544cd55f55-tslnz" Jul 21 21:40:53.309222 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:40:53.309184 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7e008490-1285-461c-80a1-6aa0295f7f76-home\") pod \"auth-test-model-24ec74-kserve-544cd55f55-tslnz\" (UID: \"7e008490-1285-461c-80a1-6aa0295f7f76\") " pod="ai-tenant-e2e-shared-b-24ec74/auth-test-model-24ec74-kserve-544cd55f55-tslnz" Jul 21 21:40:53.309222 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:40:53.309211 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7e008490-1285-461c-80a1-6aa0295f7f76-tls-certs\") pod \"auth-test-model-24ec74-kserve-544cd55f55-tslnz\" (UID: \"7e008490-1285-461c-80a1-6aa0295f7f76\") " pod="ai-tenant-e2e-shared-b-24ec74/auth-test-model-24ec74-kserve-544cd55f55-tslnz" Jul 21 21:40:53.309294 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:40:53.309241 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7e008490-1285-461c-80a1-6aa0295f7f76-model-cache\") pod \"auth-test-model-24ec74-kserve-544cd55f55-tslnz\" (UID: \"7e008490-1285-461c-80a1-6aa0295f7f76\") " pod="ai-tenant-e2e-shared-b-24ec74/auth-test-model-24ec74-kserve-544cd55f55-tslnz" Jul 21 21:40:53.309294 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:40:53.309271 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7e008490-1285-461c-80a1-6aa0295f7f76-dshm\") pod \"auth-test-model-24ec74-kserve-544cd55f55-tslnz\" (UID: \"7e008490-1285-461c-80a1-6aa0295f7f76\") " pod="ai-tenant-e2e-shared-b-24ec74/auth-test-model-24ec74-kserve-544cd55f55-tslnz" Jul 21 21:40:53.309407 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:40:53.309387 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7e008490-1285-461c-80a1-6aa0295f7f76-tmp-dir\") pod \"auth-test-model-24ec74-kserve-544cd55f55-tslnz\" (UID: \"7e008490-1285-461c-80a1-6aa0295f7f76\") " pod="ai-tenant-e2e-shared-b-24ec74/auth-test-model-24ec74-kserve-544cd55f55-tslnz" Jul 21 21:40:53.309513 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:40:53.309495 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7e008490-1285-461c-80a1-6aa0295f7f76-home\") pod \"auth-test-model-24ec74-kserve-544cd55f55-tslnz\" (UID: \"7e008490-1285-461c-80a1-6aa0295f7f76\") " pod="ai-tenant-e2e-shared-b-24ec74/auth-test-model-24ec74-kserve-544cd55f55-tslnz" Jul 21 21:40:53.309635 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:40:53.309618 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7e008490-1285-461c-80a1-6aa0295f7f76-model-cache\") pod \"auth-test-model-24ec74-kserve-544cd55f55-tslnz\" (UID: \"7e008490-1285-461c-80a1-6aa0295f7f76\") " pod="ai-tenant-e2e-shared-b-24ec74/auth-test-model-24ec74-kserve-544cd55f55-tslnz" Jul 21 21:40:53.314300 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:40:53.312894 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7e008490-1285-461c-80a1-6aa0295f7f76-dshm\") pod \"auth-test-model-24ec74-kserve-544cd55f55-tslnz\" (UID: \"7e008490-1285-461c-80a1-6aa0295f7f76\") " pod="ai-tenant-e2e-shared-b-24ec74/auth-test-model-24ec74-kserve-544cd55f55-tslnz" Jul 21 21:40:53.317855 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:40:53.317826 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7e008490-1285-461c-80a1-6aa0295f7f76-tls-certs\") pod \"auth-test-model-24ec74-kserve-544cd55f55-tslnz\" (UID: \"7e008490-1285-461c-80a1-6aa0295f7f76\") " pod="ai-tenant-e2e-shared-b-24ec74/auth-test-model-24ec74-kserve-544cd55f55-tslnz" Jul 21 21:40:53.319329 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:40:53.319305 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-khqxw\" (UniqueName: \"kubernetes.io/projected/7e008490-1285-461c-80a1-6aa0295f7f76-kube-api-access-khqxw\") pod \"auth-test-model-24ec74-kserve-544cd55f55-tslnz\" (UID: \"7e008490-1285-461c-80a1-6aa0295f7f76\") " pod="ai-tenant-e2e-shared-b-24ec74/auth-test-model-24ec74-kserve-544cd55f55-tslnz" Jul 21 21:40:53.453572 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:40:53.453474 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-24ec74/auth-test-model-24ec74-kserve-544cd55f55-tslnz" Jul 21 21:40:53.586226 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:40:53.586201 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-24ec74/auth-test-model-24ec74-kserve-544cd55f55-tslnz"] Jul 21 21:40:53.587919 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:40:53.587892 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7e008490_1285_461c_80a1_6aa0295f7f76.slice/crio-375de0e95eeaef55d81595e560e539c3077f6626955342ca62dba4f12630a235 WatchSource:0}: Error finding container 375de0e95eeaef55d81595e560e539c3077f6626955342ca62dba4f12630a235: Status 404 returned error can't find the container with id 375de0e95eeaef55d81595e560e539c3077f6626955342ca62dba4f12630a235 Jul 21 21:40:54.370932 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:40:54.370892 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-24ec74/auth-test-model-24ec74-kserve-544cd55f55-tslnz" event={"ID":"7e008490-1285-461c-80a1-6aa0295f7f76","Type":"ContainerStarted","Data":"6de4ffd2941667ca2b932ac14d13411c2d7b63b85e3e461b052c9585787e6447"} Jul 21 21:40:54.371338 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:40:54.370941 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-24ec74/auth-test-model-24ec74-kserve-544cd55f55-tslnz" event={"ID":"7e008490-1285-461c-80a1-6aa0295f7f76","Type":"ContainerStarted","Data":"375de0e95eeaef55d81595e560e539c3077f6626955342ca62dba4f12630a235"} Jul 21 21:40:54.390543 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:40:54.390491 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-24ec74/auth-test-model-24ec74-kserve-544cd55f55-tslnz" podStartSLOduration=1.390474974 podStartE2EDuration="1.390474974s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 21:40:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 21:40:54.390363803 +0000 UTC m=+2412.325113470" watchObservedRunningTime="2026-07-21 21:40:54.390474974 +0000 UTC m=+2412.325224640" Jul 21 21:41:03.454377 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:41:03.454338 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-24ec74/auth-test-model-24ec74-kserve-544cd55f55-tslnz" Jul 21 21:41:03.454377 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:41:03.454386 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-24ec74/auth-test-model-24ec74-kserve-544cd55f55-tslnz" Jul 21 21:41:03.466763 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:41:03.466740 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-24ec74/auth-test-model-24ec74-kserve-544cd55f55-tslnz" Jul 21 21:41:04.430913 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:41:04.430837 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-24ec74/auth-test-model-24ec74-kserve-544cd55f55-tslnz" Jul 21 21:43:05.196294 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:43:05.196221 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-24ec74/auth-test-model-24ec74-kserve-544cd55f55-tslnz"] Jul 21 21:43:20.216378 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:43:20.216321 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-24ec74/auth-test-model-24ec74-kserve-544cd55f55-tslnz" podUID="7e008490-1285-461c-80a1-6aa0295f7f76" containerName="main" containerID="cri-o://6de4ffd2941667ca2b932ac14d13411c2d7b63b85e3e461b052c9585787e6447" gracePeriod=45 Jul 21 21:43:20.467975 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:43:20.467925 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-24ec74/auth-test-model-24ec74-kserve-544cd55f55-tslnz" Jul 21 21:43:20.516894 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:43:20.516870 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7e008490-1285-461c-80a1-6aa0295f7f76-home\") pod \"7e008490-1285-461c-80a1-6aa0295f7f76\" (UID: \"7e008490-1285-461c-80a1-6aa0295f7f76\") " Jul 21 21:43:20.517073 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:43:20.516929 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-khqxw\" (UniqueName: \"kubernetes.io/projected/7e008490-1285-461c-80a1-6aa0295f7f76-kube-api-access-khqxw\") pod \"7e008490-1285-461c-80a1-6aa0295f7f76\" (UID: \"7e008490-1285-461c-80a1-6aa0295f7f76\") " Jul 21 21:43:20.517073 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:43:20.516992 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7e008490-1285-461c-80a1-6aa0295f7f76-tls-certs\") pod \"7e008490-1285-461c-80a1-6aa0295f7f76\" (UID: \"7e008490-1285-461c-80a1-6aa0295f7f76\") " Jul 21 21:43:20.517073 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:43:20.517039 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7e008490-1285-461c-80a1-6aa0295f7f76-model-cache\") pod \"7e008490-1285-461c-80a1-6aa0295f7f76\" (UID: \"7e008490-1285-461c-80a1-6aa0295f7f76\") " Jul 21 21:43:20.517240 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:43:20.517087 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7e008490-1285-461c-80a1-6aa0295f7f76-dshm\") pod \"7e008490-1285-461c-80a1-6aa0295f7f76\" (UID: \"7e008490-1285-461c-80a1-6aa0295f7f76\") " Jul 21 21:43:20.517240 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:43:20.517131 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e008490-1285-461c-80a1-6aa0295f7f76-home" (OuterVolumeSpecName: "home") pod "7e008490-1285-461c-80a1-6aa0295f7f76" (UID: "7e008490-1285-461c-80a1-6aa0295f7f76"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 21:43:20.517240 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:43:20.517139 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7e008490-1285-461c-80a1-6aa0295f7f76-tmp-dir\") pod \"7e008490-1285-461c-80a1-6aa0295f7f76\" (UID: \"7e008490-1285-461c-80a1-6aa0295f7f76\") " Jul 21 21:43:20.517400 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:43:20.517299 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e008490-1285-461c-80a1-6aa0295f7f76-model-cache" (OuterVolumeSpecName: "model-cache") pod "7e008490-1285-461c-80a1-6aa0295f7f76" (UID: "7e008490-1285-461c-80a1-6aa0295f7f76"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 21:43:20.517400 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:43:20.517350 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e008490-1285-461c-80a1-6aa0295f7f76-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "7e008490-1285-461c-80a1-6aa0295f7f76" (UID: "7e008490-1285-461c-80a1-6aa0295f7f76"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 21:43:20.517550 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:43:20.517514 2575 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7e008490-1285-461c-80a1-6aa0295f7f76-tmp-dir\") on node \"ip-10-0-140-97.ec2.internal\" DevicePath \"\"" Jul 21 21:43:20.517598 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:43:20.517557 2575 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7e008490-1285-461c-80a1-6aa0295f7f76-home\") on node \"ip-10-0-140-97.ec2.internal\" DevicePath \"\"" Jul 21 21:43:20.517598 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:43:20.517571 2575 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7e008490-1285-461c-80a1-6aa0295f7f76-model-cache\") on node \"ip-10-0-140-97.ec2.internal\" DevicePath \"\"" Jul 21 21:43:20.519208 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:43:20.519186 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7e008490-1285-461c-80a1-6aa0295f7f76-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "7e008490-1285-461c-80a1-6aa0295f7f76" (UID: "7e008490-1285-461c-80a1-6aa0295f7f76"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 21 21:43:20.519424 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:43:20.519404 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7e008490-1285-461c-80a1-6aa0295f7f76-kube-api-access-khqxw" (OuterVolumeSpecName: "kube-api-access-khqxw") pod "7e008490-1285-461c-80a1-6aa0295f7f76" (UID: "7e008490-1285-461c-80a1-6aa0295f7f76"). InnerVolumeSpecName "kube-api-access-khqxw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 21 21:43:20.519424 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:43:20.519411 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e008490-1285-461c-80a1-6aa0295f7f76-dshm" (OuterVolumeSpecName: "dshm") pod "7e008490-1285-461c-80a1-6aa0295f7f76" (UID: "7e008490-1285-461c-80a1-6aa0295f7f76"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 21:43:20.618937 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:43:20.618898 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-khqxw\" (UniqueName: \"kubernetes.io/projected/7e008490-1285-461c-80a1-6aa0295f7f76-kube-api-access-khqxw\") on node \"ip-10-0-140-97.ec2.internal\" DevicePath \"\"" Jul 21 21:43:20.618937 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:43:20.618936 2575 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7e008490-1285-461c-80a1-6aa0295f7f76-tls-certs\") on node \"ip-10-0-140-97.ec2.internal\" DevicePath \"\"" Jul 21 21:43:20.618937 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:43:20.618948 2575 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7e008490-1285-461c-80a1-6aa0295f7f76-dshm\") on node \"ip-10-0-140-97.ec2.internal\" DevicePath \"\"" Jul 21 21:43:21.108184 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:43:21.108152 2575 generic.go:356] "Generic (PLEG): container finished" podID="7e008490-1285-461c-80a1-6aa0295f7f76" containerID="6de4ffd2941667ca2b932ac14d13411c2d7b63b85e3e461b052c9585787e6447" exitCode=0 Jul 21 21:43:21.108347 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:43:21.108197 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-24ec74/auth-test-model-24ec74-kserve-544cd55f55-tslnz" event={"ID":"7e008490-1285-461c-80a1-6aa0295f7f76","Type":"ContainerDied","Data":"6de4ffd2941667ca2b932ac14d13411c2d7b63b85e3e461b052c9585787e6447"} Jul 21 21:43:21.108347 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:43:21.108200 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-24ec74/auth-test-model-24ec74-kserve-544cd55f55-tslnz" Jul 21 21:43:21.108347 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:43:21.108233 2575 scope.go:117] "RemoveContainer" containerID="6de4ffd2941667ca2b932ac14d13411c2d7b63b85e3e461b052c9585787e6447" Jul 21 21:43:21.108347 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:43:21.108223 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-24ec74/auth-test-model-24ec74-kserve-544cd55f55-tslnz" event={"ID":"7e008490-1285-461c-80a1-6aa0295f7f76","Type":"ContainerDied","Data":"375de0e95eeaef55d81595e560e539c3077f6626955342ca62dba4f12630a235"} Jul 21 21:43:21.116916 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:43:21.116897 2575 scope.go:117] "RemoveContainer" containerID="6de4ffd2941667ca2b932ac14d13411c2d7b63b85e3e461b052c9585787e6447" Jul 21 21:43:21.117181 ip-10-0-140-97 kubenswrapper[2575]: E0721 21:43:21.117161 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6de4ffd2941667ca2b932ac14d13411c2d7b63b85e3e461b052c9585787e6447\": container with ID starting with 6de4ffd2941667ca2b932ac14d13411c2d7b63b85e3e461b052c9585787e6447 not found: ID does not exist" containerID="6de4ffd2941667ca2b932ac14d13411c2d7b63b85e3e461b052c9585787e6447" Jul 21 21:43:21.117246 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:43:21.117193 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6de4ffd2941667ca2b932ac14d13411c2d7b63b85e3e461b052c9585787e6447"} err="failed to get container status \"6de4ffd2941667ca2b932ac14d13411c2d7b63b85e3e461b052c9585787e6447\": rpc error: code = NotFound desc = could not find container \"6de4ffd2941667ca2b932ac14d13411c2d7b63b85e3e461b052c9585787e6447\": container with ID starting with 6de4ffd2941667ca2b932ac14d13411c2d7b63b85e3e461b052c9585787e6447 not found: ID does not exist" Jul 21 21:43:21.126144 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:43:21.126091 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-24ec74/auth-test-model-24ec74-kserve-544cd55f55-tslnz"] Jul 21 21:43:21.128363 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:43:21.128341 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-24ec74/auth-test-model-24ec74-kserve-544cd55f55-tslnz"] Jul 21 21:43:22.742143 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:43:22.742111 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7e008490-1285-461c-80a1-6aa0295f7f76" path="/var/lib/kubelet/pods/7e008490-1285-461c-80a1-6aa0295f7f76/volumes" Jul 21 21:43:31.629591 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:43:31.629559 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-24ec74/sub-test-model-24ec74-kserve-5b69657f49-2d9mv"] Jul 21 21:43:31.630121 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:43:31.630099 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7e008490-1285-461c-80a1-6aa0295f7f76" containerName="main" Jul 21 21:43:31.630205 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:43:31.630125 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="7e008490-1285-461c-80a1-6aa0295f7f76" containerName="main" Jul 21 21:43:31.630244 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:43:31.630232 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="7e008490-1285-461c-80a1-6aa0295f7f76" containerName="main" Jul 21 21:43:31.634691 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:43:31.634670 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-24ec74/sub-test-model-24ec74-kserve-5b69657f49-2d9mv" Jul 21 21:43:31.637400 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:43:31.637378 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-24ec74\"/\"default-dockercfg-4x6vd\"" Jul 21 21:43:31.637400 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:43:31.637390 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-24ec74\"/\"kube-root-ca.crt\"" Jul 21 21:43:31.638424 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:43:31.638405 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-24ec74\"/\"openshift-service-ca.crt\"" Jul 21 21:43:31.638642 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:43:31.638625 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-24ec74\"/\"sub-test-model-24ec74-kserve-self-signed-certs\"" Jul 21 21:43:31.641769 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:43:31.641747 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-24ec74/sub-test-model-24ec74-kserve-5b69657f49-2d9mv"] Jul 21 21:43:31.721348 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:43:31.721318 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/50719c99-058b-4c65-81b9-b869332c1819-home\") pod \"sub-test-model-24ec74-kserve-5b69657f49-2d9mv\" (UID: \"50719c99-058b-4c65-81b9-b869332c1819\") " pod="ai-tenant-e2e-shared-b-24ec74/sub-test-model-24ec74-kserve-5b69657f49-2d9mv" Jul 21 21:43:31.721499 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:43:31.721379 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xrlbl\" (UniqueName: \"kubernetes.io/projected/50719c99-058b-4c65-81b9-b869332c1819-kube-api-access-xrlbl\") pod \"sub-test-model-24ec74-kserve-5b69657f49-2d9mv\" (UID: \"50719c99-058b-4c65-81b9-b869332c1819\") " pod="ai-tenant-e2e-shared-b-24ec74/sub-test-model-24ec74-kserve-5b69657f49-2d9mv" Jul 21 21:43:31.721499 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:43:31.721402 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/50719c99-058b-4c65-81b9-b869332c1819-model-cache\") pod \"sub-test-model-24ec74-kserve-5b69657f49-2d9mv\" (UID: \"50719c99-058b-4c65-81b9-b869332c1819\") " pod="ai-tenant-e2e-shared-b-24ec74/sub-test-model-24ec74-kserve-5b69657f49-2d9mv" Jul 21 21:43:31.721499 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:43:31.721419 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/50719c99-058b-4c65-81b9-b869332c1819-tmp-dir\") pod \"sub-test-model-24ec74-kserve-5b69657f49-2d9mv\" (UID: \"50719c99-058b-4c65-81b9-b869332c1819\") " pod="ai-tenant-e2e-shared-b-24ec74/sub-test-model-24ec74-kserve-5b69657f49-2d9mv" Jul 21 21:43:31.721499 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:43:31.721479 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/50719c99-058b-4c65-81b9-b869332c1819-dshm\") pod \"sub-test-model-24ec74-kserve-5b69657f49-2d9mv\" (UID: \"50719c99-058b-4c65-81b9-b869332c1819\") " pod="ai-tenant-e2e-shared-b-24ec74/sub-test-model-24ec74-kserve-5b69657f49-2d9mv" Jul 21 21:43:31.721686 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:43:31.721504 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/50719c99-058b-4c65-81b9-b869332c1819-tls-certs\") pod \"sub-test-model-24ec74-kserve-5b69657f49-2d9mv\" (UID: \"50719c99-058b-4c65-81b9-b869332c1819\") " pod="ai-tenant-e2e-shared-b-24ec74/sub-test-model-24ec74-kserve-5b69657f49-2d9mv" Jul 21 21:43:31.822202 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:43:31.822153 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/50719c99-058b-4c65-81b9-b869332c1819-home\") pod \"sub-test-model-24ec74-kserve-5b69657f49-2d9mv\" (UID: \"50719c99-058b-4c65-81b9-b869332c1819\") " pod="ai-tenant-e2e-shared-b-24ec74/sub-test-model-24ec74-kserve-5b69657f49-2d9mv" Jul 21 21:43:31.822394 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:43:31.822261 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xrlbl\" (UniqueName: \"kubernetes.io/projected/50719c99-058b-4c65-81b9-b869332c1819-kube-api-access-xrlbl\") pod \"sub-test-model-24ec74-kserve-5b69657f49-2d9mv\" (UID: \"50719c99-058b-4c65-81b9-b869332c1819\") " pod="ai-tenant-e2e-shared-b-24ec74/sub-test-model-24ec74-kserve-5b69657f49-2d9mv" Jul 21 21:43:31.822394 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:43:31.822293 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/50719c99-058b-4c65-81b9-b869332c1819-model-cache\") pod \"sub-test-model-24ec74-kserve-5b69657f49-2d9mv\" (UID: \"50719c99-058b-4c65-81b9-b869332c1819\") " pod="ai-tenant-e2e-shared-b-24ec74/sub-test-model-24ec74-kserve-5b69657f49-2d9mv" Jul 21 21:43:31.822394 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:43:31.822321 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/50719c99-058b-4c65-81b9-b869332c1819-tmp-dir\") pod \"sub-test-model-24ec74-kserve-5b69657f49-2d9mv\" (UID: \"50719c99-058b-4c65-81b9-b869332c1819\") " pod="ai-tenant-e2e-shared-b-24ec74/sub-test-model-24ec74-kserve-5b69657f49-2d9mv" Jul 21 21:43:31.822394 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:43:31.822389 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/50719c99-058b-4c65-81b9-b869332c1819-dshm\") pod \"sub-test-model-24ec74-kserve-5b69657f49-2d9mv\" (UID: \"50719c99-058b-4c65-81b9-b869332c1819\") " pod="ai-tenant-e2e-shared-b-24ec74/sub-test-model-24ec74-kserve-5b69657f49-2d9mv" Jul 21 21:43:31.822660 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:43:31.822413 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/50719c99-058b-4c65-81b9-b869332c1819-tls-certs\") pod \"sub-test-model-24ec74-kserve-5b69657f49-2d9mv\" (UID: \"50719c99-058b-4c65-81b9-b869332c1819\") " pod="ai-tenant-e2e-shared-b-24ec74/sub-test-model-24ec74-kserve-5b69657f49-2d9mv" Jul 21 21:43:31.822660 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:43:31.822645 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/50719c99-058b-4c65-81b9-b869332c1819-home\") pod \"sub-test-model-24ec74-kserve-5b69657f49-2d9mv\" (UID: \"50719c99-058b-4c65-81b9-b869332c1819\") " pod="ai-tenant-e2e-shared-b-24ec74/sub-test-model-24ec74-kserve-5b69657f49-2d9mv" Jul 21 21:43:31.822764 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:43:31.822716 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/50719c99-058b-4c65-81b9-b869332c1819-tmp-dir\") pod \"sub-test-model-24ec74-kserve-5b69657f49-2d9mv\" (UID: \"50719c99-058b-4c65-81b9-b869332c1819\") " pod="ai-tenant-e2e-shared-b-24ec74/sub-test-model-24ec74-kserve-5b69657f49-2d9mv" Jul 21 21:43:31.822928 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:43:31.822912 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/50719c99-058b-4c65-81b9-b869332c1819-model-cache\") pod \"sub-test-model-24ec74-kserve-5b69657f49-2d9mv\" (UID: \"50719c99-058b-4c65-81b9-b869332c1819\") " pod="ai-tenant-e2e-shared-b-24ec74/sub-test-model-24ec74-kserve-5b69657f49-2d9mv" Jul 21 21:43:31.824653 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:43:31.824632 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/50719c99-058b-4c65-81b9-b869332c1819-dshm\") pod \"sub-test-model-24ec74-kserve-5b69657f49-2d9mv\" (UID: \"50719c99-058b-4c65-81b9-b869332c1819\") " pod="ai-tenant-e2e-shared-b-24ec74/sub-test-model-24ec74-kserve-5b69657f49-2d9mv" Jul 21 21:43:31.824923 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:43:31.824902 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/50719c99-058b-4c65-81b9-b869332c1819-tls-certs\") pod \"sub-test-model-24ec74-kserve-5b69657f49-2d9mv\" (UID: \"50719c99-058b-4c65-81b9-b869332c1819\") " pod="ai-tenant-e2e-shared-b-24ec74/sub-test-model-24ec74-kserve-5b69657f49-2d9mv" Jul 21 21:43:31.832083 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:43:31.832025 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xrlbl\" (UniqueName: \"kubernetes.io/projected/50719c99-058b-4c65-81b9-b869332c1819-kube-api-access-xrlbl\") pod \"sub-test-model-24ec74-kserve-5b69657f49-2d9mv\" (UID: \"50719c99-058b-4c65-81b9-b869332c1819\") " pod="ai-tenant-e2e-shared-b-24ec74/sub-test-model-24ec74-kserve-5b69657f49-2d9mv" Jul 21 21:43:31.945087 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:43:31.944987 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-24ec74/sub-test-model-24ec74-kserve-5b69657f49-2d9mv" Jul 21 21:43:32.082669 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:43:32.082643 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-24ec74/sub-test-model-24ec74-kserve-5b69657f49-2d9mv"] Jul 21 21:43:32.084469 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:43:32.084438 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod50719c99_058b_4c65_81b9_b869332c1819.slice/crio-0d6bd7e6706bd67277d3ead3cc15534d203e15ad5964bb38afee53292fd62fb0 WatchSource:0}: Error finding container 0d6bd7e6706bd67277d3ead3cc15534d203e15ad5964bb38afee53292fd62fb0: Status 404 returned error can't find the container with id 0d6bd7e6706bd67277d3ead3cc15534d203e15ad5964bb38afee53292fd62fb0 Jul 21 21:43:32.176552 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:43:32.176486 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-24ec74/sub-test-model-24ec74-kserve-5b69657f49-2d9mv" event={"ID":"50719c99-058b-4c65-81b9-b869332c1819","Type":"ContainerStarted","Data":"bd81bc9bf399bc9be95d1726ed211f7a6944db9fa0b1fdf8f3f8488d3d9fccfe"} Jul 21 21:43:32.176719 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:43:32.176562 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-24ec74/sub-test-model-24ec74-kserve-5b69657f49-2d9mv" event={"ID":"50719c99-058b-4c65-81b9-b869332c1819","Type":"ContainerStarted","Data":"0d6bd7e6706bd67277d3ead3cc15534d203e15ad5964bb38afee53292fd62fb0"} Jul 21 21:43:32.195411 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:43:32.195319 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-24ec74/sub-test-model-24ec74-kserve-5b69657f49-2d9mv" podStartSLOduration=1.195305239 podStartE2EDuration="1.195305239s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 21:43:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 21:43:32.19330047 +0000 UTC m=+2570.128050136" watchObservedRunningTime="2026-07-21 21:43:32.195305239 +0000 UTC m=+2570.130054941" Jul 21 21:43:41.945705 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:43:41.945664 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-24ec74/sub-test-model-24ec74-kserve-5b69657f49-2d9mv" Jul 21 21:43:41.946289 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:43:41.945719 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-24ec74/sub-test-model-24ec74-kserve-5b69657f49-2d9mv" Jul 21 21:43:41.958150 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:43:41.958124 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-24ec74/sub-test-model-24ec74-kserve-5b69657f49-2d9mv" Jul 21 21:43:42.235862 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:43:42.235788 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-24ec74/sub-test-model-24ec74-kserve-5b69657f49-2d9mv" Jul 21 21:44:10.137592 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:44:10.137492 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-24ec74/sub-test-model-24ec74-kserve-5b69657f49-2d9mv"] Jul 21 21:44:25.157451 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:44:25.157404 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-24ec74/sub-test-model-24ec74-kserve-5b69657f49-2d9mv" podUID="50719c99-058b-4c65-81b9-b869332c1819" containerName="main" containerID="cri-o://bd81bc9bf399bc9be95d1726ed211f7a6944db9fa0b1fdf8f3f8488d3d9fccfe" gracePeriod=45 Jul 21 21:44:25.405388 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:44:25.405362 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-24ec74/sub-test-model-24ec74-kserve-5b69657f49-2d9mv" Jul 21 21:44:25.454300 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:44:25.454199 2575 generic.go:356] "Generic (PLEG): container finished" podID="50719c99-058b-4c65-81b9-b869332c1819" containerID="bd81bc9bf399bc9be95d1726ed211f7a6944db9fa0b1fdf8f3f8488d3d9fccfe" exitCode=0 Jul 21 21:44:25.454300 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:44:25.454255 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-24ec74/sub-test-model-24ec74-kserve-5b69657f49-2d9mv" event={"ID":"50719c99-058b-4c65-81b9-b869332c1819","Type":"ContainerDied","Data":"bd81bc9bf399bc9be95d1726ed211f7a6944db9fa0b1fdf8f3f8488d3d9fccfe"} Jul 21 21:44:25.454300 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:44:25.454277 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-24ec74/sub-test-model-24ec74-kserve-5b69657f49-2d9mv" Jul 21 21:44:25.454300 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:44:25.454299 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-24ec74/sub-test-model-24ec74-kserve-5b69657f49-2d9mv" event={"ID":"50719c99-058b-4c65-81b9-b869332c1819","Type":"ContainerDied","Data":"0d6bd7e6706bd67277d3ead3cc15534d203e15ad5964bb38afee53292fd62fb0"} Jul 21 21:44:25.454626 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:44:25.454321 2575 scope.go:117] "RemoveContainer" containerID="bd81bc9bf399bc9be95d1726ed211f7a6944db9fa0b1fdf8f3f8488d3d9fccfe" Jul 21 21:44:25.462540 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:44:25.462513 2575 scope.go:117] "RemoveContainer" containerID="bd81bc9bf399bc9be95d1726ed211f7a6944db9fa0b1fdf8f3f8488d3d9fccfe" Jul 21 21:44:25.462875 ip-10-0-140-97 kubenswrapper[2575]: E0721 21:44:25.462856 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bd81bc9bf399bc9be95d1726ed211f7a6944db9fa0b1fdf8f3f8488d3d9fccfe\": container with ID starting with bd81bc9bf399bc9be95d1726ed211f7a6944db9fa0b1fdf8f3f8488d3d9fccfe not found: ID does not exist" containerID="bd81bc9bf399bc9be95d1726ed211f7a6944db9fa0b1fdf8f3f8488d3d9fccfe" Jul 21 21:44:25.462939 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:44:25.462883 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bd81bc9bf399bc9be95d1726ed211f7a6944db9fa0b1fdf8f3f8488d3d9fccfe"} err="failed to get container status \"bd81bc9bf399bc9be95d1726ed211f7a6944db9fa0b1fdf8f3f8488d3d9fccfe\": rpc error: code = NotFound desc = could not find container \"bd81bc9bf399bc9be95d1726ed211f7a6944db9fa0b1fdf8f3f8488d3d9fccfe\": container with ID starting with bd81bc9bf399bc9be95d1726ed211f7a6944db9fa0b1fdf8f3f8488d3d9fccfe not found: ID does not exist" Jul 21 21:44:25.532070 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:44:25.532035 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/50719c99-058b-4c65-81b9-b869332c1819-tls-certs\") pod \"50719c99-058b-4c65-81b9-b869332c1819\" (UID: \"50719c99-058b-4c65-81b9-b869332c1819\") " Jul 21 21:44:25.532234 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:44:25.532077 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/50719c99-058b-4c65-81b9-b869332c1819-dshm\") pod \"50719c99-058b-4c65-81b9-b869332c1819\" (UID: \"50719c99-058b-4c65-81b9-b869332c1819\") " Jul 21 21:44:25.532234 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:44:25.532143 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/50719c99-058b-4c65-81b9-b869332c1819-home\") pod \"50719c99-058b-4c65-81b9-b869332c1819\" (UID: \"50719c99-058b-4c65-81b9-b869332c1819\") " Jul 21 21:44:25.532234 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:44:25.532178 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/50719c99-058b-4c65-81b9-b869332c1819-model-cache\") pod \"50719c99-058b-4c65-81b9-b869332c1819\" (UID: \"50719c99-058b-4c65-81b9-b869332c1819\") " Jul 21 21:44:25.532234 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:44:25.532202 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/50719c99-058b-4c65-81b9-b869332c1819-tmp-dir\") pod \"50719c99-058b-4c65-81b9-b869332c1819\" (UID: \"50719c99-058b-4c65-81b9-b869332c1819\") " Jul 21 21:44:25.532407 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:44:25.532246 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xrlbl\" (UniqueName: \"kubernetes.io/projected/50719c99-058b-4c65-81b9-b869332c1819-kube-api-access-xrlbl\") pod \"50719c99-058b-4c65-81b9-b869332c1819\" (UID: \"50719c99-058b-4c65-81b9-b869332c1819\") " Jul 21 21:44:25.532464 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:44:25.532430 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/50719c99-058b-4c65-81b9-b869332c1819-model-cache" (OuterVolumeSpecName: "model-cache") pod "50719c99-058b-4c65-81b9-b869332c1819" (UID: "50719c99-058b-4c65-81b9-b869332c1819"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 21:44:25.532521 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:44:25.532433 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/50719c99-058b-4c65-81b9-b869332c1819-home" (OuterVolumeSpecName: "home") pod "50719c99-058b-4c65-81b9-b869332c1819" (UID: "50719c99-058b-4c65-81b9-b869332c1819"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 21:44:25.532521 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:44:25.532472 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/50719c99-058b-4c65-81b9-b869332c1819-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "50719c99-058b-4c65-81b9-b869332c1819" (UID: "50719c99-058b-4c65-81b9-b869332c1819"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 21:44:25.532675 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:44:25.532631 2575 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/50719c99-058b-4c65-81b9-b869332c1819-home\") on node \"ip-10-0-140-97.ec2.internal\" DevicePath \"\"" Jul 21 21:44:25.532675 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:44:25.532644 2575 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/50719c99-058b-4c65-81b9-b869332c1819-model-cache\") on node \"ip-10-0-140-97.ec2.internal\" DevicePath \"\"" Jul 21 21:44:25.532675 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:44:25.532654 2575 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/50719c99-058b-4c65-81b9-b869332c1819-tmp-dir\") on node \"ip-10-0-140-97.ec2.internal\" DevicePath \"\"" Jul 21 21:44:25.534250 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:44:25.534208 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/50719c99-058b-4c65-81b9-b869332c1819-dshm" (OuterVolumeSpecName: "dshm") pod "50719c99-058b-4c65-81b9-b869332c1819" (UID: "50719c99-058b-4c65-81b9-b869332c1819"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 21:44:25.534354 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:44:25.534279 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/50719c99-058b-4c65-81b9-b869332c1819-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "50719c99-058b-4c65-81b9-b869332c1819" (UID: "50719c99-058b-4c65-81b9-b869332c1819"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 21 21:44:25.534354 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:44:25.534304 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/50719c99-058b-4c65-81b9-b869332c1819-kube-api-access-xrlbl" (OuterVolumeSpecName: "kube-api-access-xrlbl") pod "50719c99-058b-4c65-81b9-b869332c1819" (UID: "50719c99-058b-4c65-81b9-b869332c1819"). InnerVolumeSpecName "kube-api-access-xrlbl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 21 21:44:25.633997 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:44:25.633963 2575 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/50719c99-058b-4c65-81b9-b869332c1819-tls-certs\") on node \"ip-10-0-140-97.ec2.internal\" DevicePath \"\"" Jul 21 21:44:25.633997 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:44:25.633992 2575 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/50719c99-058b-4c65-81b9-b869332c1819-dshm\") on node \"ip-10-0-140-97.ec2.internal\" DevicePath \"\"" Jul 21 21:44:25.633997 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:44:25.634001 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xrlbl\" (UniqueName: \"kubernetes.io/projected/50719c99-058b-4c65-81b9-b869332c1819-kube-api-access-xrlbl\") on node \"ip-10-0-140-97.ec2.internal\" DevicePath \"\"" Jul 21 21:44:25.776033 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:44:25.776005 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-24ec74/sub-test-model-24ec74-kserve-5b69657f49-2d9mv"] Jul 21 21:44:25.782315 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:44:25.782293 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-24ec74/sub-test-model-24ec74-kserve-5b69657f49-2d9mv"] Jul 21 21:44:26.740550 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:44:26.740511 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="50719c99-058b-4c65-81b9-b869332c1819" path="/var/lib/kubelet/pods/50719c99-058b-4c65-81b9-b869332c1819/volumes" Jul 21 21:44:42.438222 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:44:42.438188 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-24ec74/rate-test-model-24ec74-kserve-59b89cdfff-j6n6l"] Jul 21 21:44:42.438672 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:44:42.438651 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="50719c99-058b-4c65-81b9-b869332c1819" containerName="main" Jul 21 21:44:42.438672 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:44:42.438673 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="50719c99-058b-4c65-81b9-b869332c1819" containerName="main" Jul 21 21:44:42.438843 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:44:42.438749 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="50719c99-058b-4c65-81b9-b869332c1819" containerName="main" Jul 21 21:44:42.443268 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:44:42.443245 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-24ec74/rate-test-model-24ec74-kserve-59b89cdfff-j6n6l" Jul 21 21:44:42.446180 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:44:42.446157 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-24ec74\"/\"kube-root-ca.crt\"" Jul 21 21:44:42.447052 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:44:42.447026 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-24ec74\"/\"rate-test-model-24ec74-kserve-self-signed-certs\"" Jul 21 21:44:42.447173 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:44:42.447064 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-24ec74\"/\"openshift-service-ca.crt\"" Jul 21 21:44:42.447173 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:44:42.447031 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-24ec74\"/\"default-dockercfg-4x6vd\"" Jul 21 21:44:42.452514 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:44:42.452491 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-24ec74/rate-test-model-24ec74-kserve-59b89cdfff-j6n6l"] Jul 21 21:44:42.488013 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:44:42.487986 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4ceeb2fd-b5af-4715-8c77-5c041b61a08b-tmp-dir\") pod \"rate-test-model-24ec74-kserve-59b89cdfff-j6n6l\" (UID: \"4ceeb2fd-b5af-4715-8c77-5c041b61a08b\") " pod="ai-tenant-e2e-shared-b-24ec74/rate-test-model-24ec74-kserve-59b89cdfff-j6n6l" Jul 21 21:44:42.488166 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:44:42.488038 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4ceeb2fd-b5af-4715-8c77-5c041b61a08b-dshm\") pod \"rate-test-model-24ec74-kserve-59b89cdfff-j6n6l\" (UID: \"4ceeb2fd-b5af-4715-8c77-5c041b61a08b\") " pod="ai-tenant-e2e-shared-b-24ec74/rate-test-model-24ec74-kserve-59b89cdfff-j6n6l" Jul 21 21:44:42.488166 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:44:42.488088 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4ceeb2fd-b5af-4715-8c77-5c041b61a08b-home\") pod \"rate-test-model-24ec74-kserve-59b89cdfff-j6n6l\" (UID: \"4ceeb2fd-b5af-4715-8c77-5c041b61a08b\") " pod="ai-tenant-e2e-shared-b-24ec74/rate-test-model-24ec74-kserve-59b89cdfff-j6n6l" Jul 21 21:44:42.488166 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:44:42.488143 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8c4b7\" (UniqueName: \"kubernetes.io/projected/4ceeb2fd-b5af-4715-8c77-5c041b61a08b-kube-api-access-8c4b7\") pod \"rate-test-model-24ec74-kserve-59b89cdfff-j6n6l\" (UID: \"4ceeb2fd-b5af-4715-8c77-5c041b61a08b\") " pod="ai-tenant-e2e-shared-b-24ec74/rate-test-model-24ec74-kserve-59b89cdfff-j6n6l" Jul 21 21:44:42.488314 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:44:42.488280 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4ceeb2fd-b5af-4715-8c77-5c041b61a08b-tls-certs\") pod \"rate-test-model-24ec74-kserve-59b89cdfff-j6n6l\" (UID: \"4ceeb2fd-b5af-4715-8c77-5c041b61a08b\") " pod="ai-tenant-e2e-shared-b-24ec74/rate-test-model-24ec74-kserve-59b89cdfff-j6n6l" Jul 21 21:44:42.488358 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:44:42.488313 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4ceeb2fd-b5af-4715-8c77-5c041b61a08b-model-cache\") pod \"rate-test-model-24ec74-kserve-59b89cdfff-j6n6l\" (UID: \"4ceeb2fd-b5af-4715-8c77-5c041b61a08b\") " pod="ai-tenant-e2e-shared-b-24ec74/rate-test-model-24ec74-kserve-59b89cdfff-j6n6l" Jul 21 21:44:42.588796 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:44:42.588764 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4ceeb2fd-b5af-4715-8c77-5c041b61a08b-tls-certs\") pod \"rate-test-model-24ec74-kserve-59b89cdfff-j6n6l\" (UID: \"4ceeb2fd-b5af-4715-8c77-5c041b61a08b\") " pod="ai-tenant-e2e-shared-b-24ec74/rate-test-model-24ec74-kserve-59b89cdfff-j6n6l" Jul 21 21:44:42.588796 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:44:42.588796 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4ceeb2fd-b5af-4715-8c77-5c041b61a08b-model-cache\") pod \"rate-test-model-24ec74-kserve-59b89cdfff-j6n6l\" (UID: \"4ceeb2fd-b5af-4715-8c77-5c041b61a08b\") " pod="ai-tenant-e2e-shared-b-24ec74/rate-test-model-24ec74-kserve-59b89cdfff-j6n6l" Jul 21 21:44:42.589025 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:44:42.588831 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4ceeb2fd-b5af-4715-8c77-5c041b61a08b-tmp-dir\") pod \"rate-test-model-24ec74-kserve-59b89cdfff-j6n6l\" (UID: \"4ceeb2fd-b5af-4715-8c77-5c041b61a08b\") " pod="ai-tenant-e2e-shared-b-24ec74/rate-test-model-24ec74-kserve-59b89cdfff-j6n6l" Jul 21 21:44:42.589025 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:44:42.588937 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4ceeb2fd-b5af-4715-8c77-5c041b61a08b-dshm\") pod \"rate-test-model-24ec74-kserve-59b89cdfff-j6n6l\" (UID: \"4ceeb2fd-b5af-4715-8c77-5c041b61a08b\") " pod="ai-tenant-e2e-shared-b-24ec74/rate-test-model-24ec74-kserve-59b89cdfff-j6n6l" Jul 21 21:44:42.589025 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:44:42.588981 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4ceeb2fd-b5af-4715-8c77-5c041b61a08b-home\") pod \"rate-test-model-24ec74-kserve-59b89cdfff-j6n6l\" (UID: \"4ceeb2fd-b5af-4715-8c77-5c041b61a08b\") " pod="ai-tenant-e2e-shared-b-24ec74/rate-test-model-24ec74-kserve-59b89cdfff-j6n6l" Jul 21 21:44:42.589200 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:44:42.589051 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8c4b7\" (UniqueName: \"kubernetes.io/projected/4ceeb2fd-b5af-4715-8c77-5c041b61a08b-kube-api-access-8c4b7\") pod \"rate-test-model-24ec74-kserve-59b89cdfff-j6n6l\" (UID: \"4ceeb2fd-b5af-4715-8c77-5c041b61a08b\") " pod="ai-tenant-e2e-shared-b-24ec74/rate-test-model-24ec74-kserve-59b89cdfff-j6n6l" Jul 21 21:44:42.589200 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:44:42.589172 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4ceeb2fd-b5af-4715-8c77-5c041b61a08b-tmp-dir\") pod \"rate-test-model-24ec74-kserve-59b89cdfff-j6n6l\" (UID: \"4ceeb2fd-b5af-4715-8c77-5c041b61a08b\") " pod="ai-tenant-e2e-shared-b-24ec74/rate-test-model-24ec74-kserve-59b89cdfff-j6n6l" Jul 21 21:44:42.589314 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:44:42.589239 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4ceeb2fd-b5af-4715-8c77-5c041b61a08b-model-cache\") pod \"rate-test-model-24ec74-kserve-59b89cdfff-j6n6l\" (UID: \"4ceeb2fd-b5af-4715-8c77-5c041b61a08b\") " pod="ai-tenant-e2e-shared-b-24ec74/rate-test-model-24ec74-kserve-59b89cdfff-j6n6l" Jul 21 21:44:42.589391 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:44:42.589368 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4ceeb2fd-b5af-4715-8c77-5c041b61a08b-home\") pod \"rate-test-model-24ec74-kserve-59b89cdfff-j6n6l\" (UID: \"4ceeb2fd-b5af-4715-8c77-5c041b61a08b\") " pod="ai-tenant-e2e-shared-b-24ec74/rate-test-model-24ec74-kserve-59b89cdfff-j6n6l" Jul 21 21:44:42.591068 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:44:42.591041 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4ceeb2fd-b5af-4715-8c77-5c041b61a08b-dshm\") pod \"rate-test-model-24ec74-kserve-59b89cdfff-j6n6l\" (UID: \"4ceeb2fd-b5af-4715-8c77-5c041b61a08b\") " pod="ai-tenant-e2e-shared-b-24ec74/rate-test-model-24ec74-kserve-59b89cdfff-j6n6l" Jul 21 21:44:42.591324 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:44:42.591306 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4ceeb2fd-b5af-4715-8c77-5c041b61a08b-tls-certs\") pod \"rate-test-model-24ec74-kserve-59b89cdfff-j6n6l\" (UID: \"4ceeb2fd-b5af-4715-8c77-5c041b61a08b\") " pod="ai-tenant-e2e-shared-b-24ec74/rate-test-model-24ec74-kserve-59b89cdfff-j6n6l" Jul 21 21:44:42.597403 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:44:42.597385 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8c4b7\" (UniqueName: \"kubernetes.io/projected/4ceeb2fd-b5af-4715-8c77-5c041b61a08b-kube-api-access-8c4b7\") pod \"rate-test-model-24ec74-kserve-59b89cdfff-j6n6l\" (UID: \"4ceeb2fd-b5af-4715-8c77-5c041b61a08b\") " pod="ai-tenant-e2e-shared-b-24ec74/rate-test-model-24ec74-kserve-59b89cdfff-j6n6l" Jul 21 21:44:42.753919 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:44:42.753843 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-24ec74/rate-test-model-24ec74-kserve-59b89cdfff-j6n6l" Jul 21 21:44:42.879624 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:44:42.879595 2575 scope.go:117] "RemoveContainer" containerID="fdaeb42e6695f521567b2c92f2d680d23330075235d0eed54434b21b6d39886b" Jul 21 21:44:42.887360 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:44:42.887339 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-24ec74/rate-test-model-24ec74-kserve-59b89cdfff-j6n6l"] Jul 21 21:44:42.889602 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:44:42.889579 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4ceeb2fd_b5af_4715_8c77_5c041b61a08b.slice/crio-ba98d257fcd9671f20ec6066a001bf8db5a215c4fdc571367b979b1d9534f543 WatchSource:0}: Error finding container ba98d257fcd9671f20ec6066a001bf8db5a215c4fdc571367b979b1d9534f543: Status 404 returned error can't find the container with id ba98d257fcd9671f20ec6066a001bf8db5a215c4fdc571367b979b1d9534f543 Jul 21 21:44:43.547936 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:44:43.547892 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-24ec74/rate-test-model-24ec74-kserve-59b89cdfff-j6n6l" event={"ID":"4ceeb2fd-b5af-4715-8c77-5c041b61a08b","Type":"ContainerStarted","Data":"56f929d9da9f14077e9f94a5d38af109afde40251e3413220ec9ee154f674627"} Jul 21 21:44:43.547936 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:44:43.547935 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-24ec74/rate-test-model-24ec74-kserve-59b89cdfff-j6n6l" event={"ID":"4ceeb2fd-b5af-4715-8c77-5c041b61a08b","Type":"ContainerStarted","Data":"ba98d257fcd9671f20ec6066a001bf8db5a215c4fdc571367b979b1d9534f543"} Jul 21 21:44:43.569068 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:44:43.569003 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-24ec74/rate-test-model-24ec74-kserve-59b89cdfff-j6n6l" podStartSLOduration=1.568977953 podStartE2EDuration="1.568977953s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 21:44:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 21:44:43.568296922 +0000 UTC m=+2641.503046610" watchObservedRunningTime="2026-07-21 21:44:43.568977953 +0000 UTC m=+2641.503727617" Jul 21 21:44:52.754214 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:44:52.754176 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-24ec74/rate-test-model-24ec74-kserve-59b89cdfff-j6n6l" Jul 21 21:44:52.754637 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:44:52.754229 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-24ec74/rate-test-model-24ec74-kserve-59b89cdfff-j6n6l" Jul 21 21:44:52.766383 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:44:52.766362 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-24ec74/rate-test-model-24ec74-kserve-59b89cdfff-j6n6l" Jul 21 21:44:53.610911 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:44:53.610883 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-24ec74/rate-test-model-24ec74-kserve-59b89cdfff-j6n6l" Jul 21 21:45:00.135791 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:45:00.135760 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-e584ac-29744505-88qgh"] Jul 21 21:45:00.139496 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:45:00.139473 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-e584ac-29744505-88qgh" Jul 21 21:45:00.141998 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:45:00.141968 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9fxhr\"" Jul 21 21:45:00.148382 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:45:00.148360 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-e584ac-29744505-88qgh"] Jul 21 21:45:00.153923 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:45:00.153902 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-e584ac-29744505-88qgh" Jul 21 21:45:00.290610 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:45:00.290587 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-e584ac-29744505-88qgh"] Jul 21 21:45:00.292700 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:45:00.292673 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8b710e33_df66_4f10_8233_56fc35fcf4f6.slice/crio-4172fa05f54ccd3e13c26bfb8ead756ced2fa917bd73a6486d6baa5300a55f87 WatchSource:0}: Error finding container 4172fa05f54ccd3e13c26bfb8ead756ced2fa917bd73a6486d6baa5300a55f87: Status 404 returned error can't find the container with id 4172fa05f54ccd3e13c26bfb8ead756ced2fa917bd73a6486d6baa5300a55f87 Jul 21 21:45:00.635234 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:45:00.635205 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-e584ac-29744505-88qgh" event={"ID":"8b710e33-df66-4f10-8233-56fc35fcf4f6","Type":"ContainerStarted","Data":"26d2280c810a1e454d03348b9565b2e7ae4a372c59c68e7066342f9416658109"} Jul 21 21:45:00.635375 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:45:00.635257 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-e584ac-29744505-88qgh" event={"ID":"8b710e33-df66-4f10-8233-56fc35fcf4f6","Type":"ContainerStarted","Data":"4172fa05f54ccd3e13c26bfb8ead756ced2fa917bd73a6486d6baa5300a55f87"} Jul 21 21:45:00.651549 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:45:00.651485 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-e584ac-29744505-88qgh" podStartSLOduration=0.651471171 podStartE2EDuration="651.471171ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 21:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 21:45:00.649333599 +0000 UTC m=+2658.584083278" watchObservedRunningTime="2026-07-21 21:45:00.651471171 +0000 UTC m=+2658.586220865" Jul 21 21:45:01.643039 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:45:01.643006 2575 generic.go:356] "Generic (PLEG): container finished" podID="8b710e33-df66-4f10-8233-56fc35fcf4f6" containerID="26d2280c810a1e454d03348b9565b2e7ae4a372c59c68e7066342f9416658109" exitCode=0 Jul 21 21:45:01.643437 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:45:01.643092 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-e584ac-29744505-88qgh" event={"ID":"8b710e33-df66-4f10-8233-56fc35fcf4f6","Type":"ContainerDied","Data":"26d2280c810a1e454d03348b9565b2e7ae4a372c59c68e7066342f9416658109"} Jul 21 21:45:02.783765 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:45:02.783742 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-e584ac-29744505-88qgh" Jul 21 21:45:03.658306 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:45:03.658180 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-e584ac-29744505-88qgh" Jul 21 21:45:03.658481 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:45:03.658176 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-e584ac-29744505-88qgh" event={"ID":"8b710e33-df66-4f10-8233-56fc35fcf4f6","Type":"ContainerDied","Data":"4172fa05f54ccd3e13c26bfb8ead756ced2fa917bd73a6486d6baa5300a55f87"} Jul 21 21:45:03.658481 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:45:03.658424 2575 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4172fa05f54ccd3e13c26bfb8ead756ced2fa917bd73a6486d6baa5300a55f87" Jul 21 21:45:30.346430 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:45:30.346387 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-24ec74/rate-test-model-24ec74-kserve-59b89cdfff-j6n6l"] Jul 21 21:45:42.907904 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:45:42.907832 2575 scope.go:117] "RemoveContainer" containerID="f9b62bbc510310956c353686071201576e714df9a0cb546ea4c54da4ac468aee" Jul 21 21:45:42.915841 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:45:42.915822 2575 scope.go:117] "RemoveContainer" containerID="5e3ef35fc244a528f60136f25df645c2319164789191e03dc6acba6e3d135db8" Jul 21 21:45:45.366337 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:45:45.366290 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-24ec74/rate-test-model-24ec74-kserve-59b89cdfff-j6n6l" podUID="4ceeb2fd-b5af-4715-8c77-5c041b61a08b" containerName="main" containerID="cri-o://56f929d9da9f14077e9f94a5d38af109afde40251e3413220ec9ee154f674627" gracePeriod=45 Jul 21 21:45:45.612357 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:45:45.612336 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-24ec74/rate-test-model-24ec74-kserve-59b89cdfff-j6n6l" Jul 21 21:45:45.676347 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:45:45.676267 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8c4b7\" (UniqueName: \"kubernetes.io/projected/4ceeb2fd-b5af-4715-8c77-5c041b61a08b-kube-api-access-8c4b7\") pod \"4ceeb2fd-b5af-4715-8c77-5c041b61a08b\" (UID: \"4ceeb2fd-b5af-4715-8c77-5c041b61a08b\") " Jul 21 21:45:45.676347 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:45:45.676305 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4ceeb2fd-b5af-4715-8c77-5c041b61a08b-home\") pod \"4ceeb2fd-b5af-4715-8c77-5c041b61a08b\" (UID: \"4ceeb2fd-b5af-4715-8c77-5c041b61a08b\") " Jul 21 21:45:45.676347 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:45:45.676342 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4ceeb2fd-b5af-4715-8c77-5c041b61a08b-dshm\") pod \"4ceeb2fd-b5af-4715-8c77-5c041b61a08b\" (UID: \"4ceeb2fd-b5af-4715-8c77-5c041b61a08b\") " Jul 21 21:45:45.676622 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:45:45.676462 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4ceeb2fd-b5af-4715-8c77-5c041b61a08b-tls-certs\") pod \"4ceeb2fd-b5af-4715-8c77-5c041b61a08b\" (UID: \"4ceeb2fd-b5af-4715-8c77-5c041b61a08b\") " Jul 21 21:45:45.676622 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:45:45.676595 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4ceeb2fd-b5af-4715-8c77-5c041b61a08b-model-cache\") pod \"4ceeb2fd-b5af-4715-8c77-5c041b61a08b\" (UID: \"4ceeb2fd-b5af-4715-8c77-5c041b61a08b\") " Jul 21 21:45:45.676740 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:45:45.676622 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ceeb2fd-b5af-4715-8c77-5c041b61a08b-home" (OuterVolumeSpecName: "home") pod "4ceeb2fd-b5af-4715-8c77-5c041b61a08b" (UID: "4ceeb2fd-b5af-4715-8c77-5c041b61a08b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 21:45:45.676740 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:45:45.676693 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4ceeb2fd-b5af-4715-8c77-5c041b61a08b-tmp-dir\") pod \"4ceeb2fd-b5af-4715-8c77-5c041b61a08b\" (UID: \"4ceeb2fd-b5af-4715-8c77-5c041b61a08b\") " Jul 21 21:45:45.676836 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:45:45.676813 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ceeb2fd-b5af-4715-8c77-5c041b61a08b-model-cache" (OuterVolumeSpecName: "model-cache") pod "4ceeb2fd-b5af-4715-8c77-5c041b61a08b" (UID: "4ceeb2fd-b5af-4715-8c77-5c041b61a08b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 21:45:45.676963 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:45:45.676935 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ceeb2fd-b5af-4715-8c77-5c041b61a08b-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "4ceeb2fd-b5af-4715-8c77-5c041b61a08b" (UID: "4ceeb2fd-b5af-4715-8c77-5c041b61a08b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 21:45:45.677098 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:45:45.677081 2575 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4ceeb2fd-b5af-4715-8c77-5c041b61a08b-model-cache\") on node \"ip-10-0-140-97.ec2.internal\" DevicePath \"\"" Jul 21 21:45:45.677148 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:45:45.677104 2575 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4ceeb2fd-b5af-4715-8c77-5c041b61a08b-tmp-dir\") on node \"ip-10-0-140-97.ec2.internal\" DevicePath \"\"" Jul 21 21:45:45.677148 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:45:45.677118 2575 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4ceeb2fd-b5af-4715-8c77-5c041b61a08b-home\") on node \"ip-10-0-140-97.ec2.internal\" DevicePath \"\"" Jul 21 21:45:45.678651 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:45:45.678594 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4ceeb2fd-b5af-4715-8c77-5c041b61a08b-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "4ceeb2fd-b5af-4715-8c77-5c041b61a08b" (UID: "4ceeb2fd-b5af-4715-8c77-5c041b61a08b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 21 21:45:45.678764 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:45:45.678659 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4ceeb2fd-b5af-4715-8c77-5c041b61a08b-kube-api-access-8c4b7" (OuterVolumeSpecName: "kube-api-access-8c4b7") pod "4ceeb2fd-b5af-4715-8c77-5c041b61a08b" (UID: "4ceeb2fd-b5af-4715-8c77-5c041b61a08b"). InnerVolumeSpecName "kube-api-access-8c4b7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 21 21:45:45.678764 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:45:45.678693 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ceeb2fd-b5af-4715-8c77-5c041b61a08b-dshm" (OuterVolumeSpecName: "dshm") pod "4ceeb2fd-b5af-4715-8c77-5c041b61a08b" (UID: "4ceeb2fd-b5af-4715-8c77-5c041b61a08b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 21:45:45.778349 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:45:45.778311 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8c4b7\" (UniqueName: \"kubernetes.io/projected/4ceeb2fd-b5af-4715-8c77-5c041b61a08b-kube-api-access-8c4b7\") on node \"ip-10-0-140-97.ec2.internal\" DevicePath \"\"" Jul 21 21:45:45.778349 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:45:45.778342 2575 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4ceeb2fd-b5af-4715-8c77-5c041b61a08b-dshm\") on node \"ip-10-0-140-97.ec2.internal\" DevicePath \"\"" Jul 21 21:45:45.778349 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:45:45.778351 2575 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4ceeb2fd-b5af-4715-8c77-5c041b61a08b-tls-certs\") on node \"ip-10-0-140-97.ec2.internal\" DevicePath \"\"" Jul 21 21:45:45.879583 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:45:45.879512 2575 generic.go:356] "Generic (PLEG): container finished" podID="4ceeb2fd-b5af-4715-8c77-5c041b61a08b" containerID="56f929d9da9f14077e9f94a5d38af109afde40251e3413220ec9ee154f674627" exitCode=0 Jul 21 21:45:45.879583 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:45:45.879577 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-24ec74/rate-test-model-24ec74-kserve-59b89cdfff-j6n6l" Jul 21 21:45:45.879804 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:45:45.879601 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-24ec74/rate-test-model-24ec74-kserve-59b89cdfff-j6n6l" event={"ID":"4ceeb2fd-b5af-4715-8c77-5c041b61a08b","Type":"ContainerDied","Data":"56f929d9da9f14077e9f94a5d38af109afde40251e3413220ec9ee154f674627"} Jul 21 21:45:45.879804 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:45:45.879642 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-24ec74/rate-test-model-24ec74-kserve-59b89cdfff-j6n6l" event={"ID":"4ceeb2fd-b5af-4715-8c77-5c041b61a08b","Type":"ContainerDied","Data":"ba98d257fcd9671f20ec6066a001bf8db5a215c4fdc571367b979b1d9534f543"} Jul 21 21:45:45.879804 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:45:45.879660 2575 scope.go:117] "RemoveContainer" containerID="56f929d9da9f14077e9f94a5d38af109afde40251e3413220ec9ee154f674627" Jul 21 21:45:45.888906 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:45:45.888885 2575 scope.go:117] "RemoveContainer" containerID="56f929d9da9f14077e9f94a5d38af109afde40251e3413220ec9ee154f674627" Jul 21 21:45:45.889221 ip-10-0-140-97 kubenswrapper[2575]: E0721 21:45:45.889198 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"56f929d9da9f14077e9f94a5d38af109afde40251e3413220ec9ee154f674627\": container with ID starting with 56f929d9da9f14077e9f94a5d38af109afde40251e3413220ec9ee154f674627 not found: ID does not exist" containerID="56f929d9da9f14077e9f94a5d38af109afde40251e3413220ec9ee154f674627" Jul 21 21:45:45.889282 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:45:45.889230 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"56f929d9da9f14077e9f94a5d38af109afde40251e3413220ec9ee154f674627"} err="failed to get container status \"56f929d9da9f14077e9f94a5d38af109afde40251e3413220ec9ee154f674627\": rpc error: code = NotFound desc = could not find container \"56f929d9da9f14077e9f94a5d38af109afde40251e3413220ec9ee154f674627\": container with ID starting with 56f929d9da9f14077e9f94a5d38af109afde40251e3413220ec9ee154f674627 not found: ID does not exist" Jul 21 21:45:45.900421 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:45:45.900399 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-24ec74/rate-test-model-24ec74-kserve-59b89cdfff-j6n6l"] Jul 21 21:45:45.905124 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:45:45.905103 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-24ec74/rate-test-model-24ec74-kserve-59b89cdfff-j6n6l"] Jul 21 21:45:46.742607 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:45:46.742564 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4ceeb2fd-b5af-4715-8c77-5c041b61a08b" path="/var/lib/kubelet/pods/4ceeb2fd-b5af-4715-8c77-5c041b61a08b/volumes" Jul 21 21:46:00.527757 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:46:00.527719 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-e584ac-29744505-88qgh"] Jul 21 21:46:00.529953 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:46:00.529923 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-e584ac-29744505-88qgh"] Jul 21 21:46:00.741454 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:46:00.741418 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8b710e33-df66-4f10-8233-56fc35fcf4f6" path="/var/lib/kubelet/pods/8b710e33-df66-4f10-8233-56fc35fcf4f6/volumes" Jul 21 21:46:11.974884 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:46:11.974851 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-e584ac-openshift-default-5d8bd494c-sslp5"] Jul 21 21:46:11.975592 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:46:11.975092 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-e584ac-openshift-default-5d8bd494c-sslp5" podUID="4f4a0c63-45f1-4f45-b038-c2e2433900f6" containerName="istio-proxy" containerID="cri-o://fb6cad868ca2384d672a281e4adc5bdfbe09d88cf7147e7a595207177e4d18bf" gracePeriod=30 Jul 21 21:46:17.230610 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:46:17.230583 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-e584ac-openshift-default-5d8bd494c-sslp5" Jul 21 21:46:17.283896 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:46:17.283817 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f4a0c63-45f1-4f45-b038-c2e2433900f6-workload-socket\") pod \"4f4a0c63-45f1-4f45-b038-c2e2433900f6\" (UID: \"4f4a0c63-45f1-4f45-b038-c2e2433900f6\") " Jul 21 21:46:17.283896 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:46:17.283871 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4f4a0c63-45f1-4f45-b038-c2e2433900f6-istiod-ca-cert\") pod \"4f4a0c63-45f1-4f45-b038-c2e2433900f6\" (UID: \"4f4a0c63-45f1-4f45-b038-c2e2433900f6\") " Jul 21 21:46:17.284105 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:46:17.283903 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4f4a0c63-45f1-4f45-b038-c2e2433900f6-istio-token\") pod \"4f4a0c63-45f1-4f45-b038-c2e2433900f6\" (UID: \"4f4a0c63-45f1-4f45-b038-c2e2433900f6\") " Jul 21 21:46:17.284105 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:46:17.283931 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f4a0c63-45f1-4f45-b038-c2e2433900f6-credential-socket\") pod \"4f4a0c63-45f1-4f45-b038-c2e2433900f6\" (UID: \"4f4a0c63-45f1-4f45-b038-c2e2433900f6\") " Jul 21 21:46:17.284105 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:46:17.283959 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4f4a0c63-45f1-4f45-b038-c2e2433900f6-istio-podinfo\") pod \"4f4a0c63-45f1-4f45-b038-c2e2433900f6\" (UID: \"4f4a0c63-45f1-4f45-b038-c2e2433900f6\") " Jul 21 21:46:17.284105 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:46:17.284008 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4f4a0c63-45f1-4f45-b038-c2e2433900f6-workload-certs\") pod \"4f4a0c63-45f1-4f45-b038-c2e2433900f6\" (UID: \"4f4a0c63-45f1-4f45-b038-c2e2433900f6\") " Jul 21 21:46:17.284105 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:46:17.284074 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4f4a0c63-45f1-4f45-b038-c2e2433900f6-istio-data\") pod \"4f4a0c63-45f1-4f45-b038-c2e2433900f6\" (UID: \"4f4a0c63-45f1-4f45-b038-c2e2433900f6\") " Jul 21 21:46:17.284105 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:46:17.284101 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-h7llv\" (UniqueName: \"kubernetes.io/projected/4f4a0c63-45f1-4f45-b038-c2e2433900f6-kube-api-access-h7llv\") pod \"4f4a0c63-45f1-4f45-b038-c2e2433900f6\" (UID: \"4f4a0c63-45f1-4f45-b038-c2e2433900f6\") " Jul 21 21:46:17.284389 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:46:17.284134 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4f4a0c63-45f1-4f45-b038-c2e2433900f6-istio-envoy\") pod \"4f4a0c63-45f1-4f45-b038-c2e2433900f6\" (UID: \"4f4a0c63-45f1-4f45-b038-c2e2433900f6\") " Jul 21 21:46:17.284389 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:46:17.284139 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4f4a0c63-45f1-4f45-b038-c2e2433900f6-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "4f4a0c63-45f1-4f45-b038-c2e2433900f6" (UID: "4f4a0c63-45f1-4f45-b038-c2e2433900f6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 21:46:17.284389 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:46:17.284249 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4f4a0c63-45f1-4f45-b038-c2e2433900f6-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "4f4a0c63-45f1-4f45-b038-c2e2433900f6" (UID: "4f4a0c63-45f1-4f45-b038-c2e2433900f6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 21:46:17.284556 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:46:17.284441 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4f4a0c63-45f1-4f45-b038-c2e2433900f6-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "4f4a0c63-45f1-4f45-b038-c2e2433900f6" (UID: "4f4a0c63-45f1-4f45-b038-c2e2433900f6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 21 21:46:17.284556 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:46:17.284499 2575 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f4a0c63-45f1-4f45-b038-c2e2433900f6-workload-socket\") on node \"ip-10-0-140-97.ec2.internal\" DevicePath \"\"" Jul 21 21:46:17.284556 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:46:17.284517 2575 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f4a0c63-45f1-4f45-b038-c2e2433900f6-credential-socket\") on node \"ip-10-0-140-97.ec2.internal\" DevicePath \"\"" Jul 21 21:46:17.284680 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:46:17.284568 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4f4a0c63-45f1-4f45-b038-c2e2433900f6-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "4f4a0c63-45f1-4f45-b038-c2e2433900f6" (UID: "4f4a0c63-45f1-4f45-b038-c2e2433900f6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 21:46:17.285012 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:46:17.284977 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4f4a0c63-45f1-4f45-b038-c2e2433900f6-istio-data" (OuterVolumeSpecName: "istio-data") pod "4f4a0c63-45f1-4f45-b038-c2e2433900f6" (UID: "4f4a0c63-45f1-4f45-b038-c2e2433900f6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 21:46:17.286338 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:46:17.286306 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4f4a0c63-45f1-4f45-b038-c2e2433900f6-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "4f4a0c63-45f1-4f45-b038-c2e2433900f6" (UID: "4f4a0c63-45f1-4f45-b038-c2e2433900f6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 21:46:17.286338 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:46:17.286314 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4f4a0c63-45f1-4f45-b038-c2e2433900f6-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "4f4a0c63-45f1-4f45-b038-c2e2433900f6" (UID: "4f4a0c63-45f1-4f45-b038-c2e2433900f6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 21 21:46:17.286541 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:46:17.286512 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4f4a0c63-45f1-4f45-b038-c2e2433900f6-istio-token" (OuterVolumeSpecName: "istio-token") pod "4f4a0c63-45f1-4f45-b038-c2e2433900f6" (UID: "4f4a0c63-45f1-4f45-b038-c2e2433900f6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 21 21:46:17.286654 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:46:17.286635 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4f4a0c63-45f1-4f45-b038-c2e2433900f6-kube-api-access-h7llv" (OuterVolumeSpecName: "kube-api-access-h7llv") pod "4f4a0c63-45f1-4f45-b038-c2e2433900f6" (UID: "4f4a0c63-45f1-4f45-b038-c2e2433900f6"). InnerVolumeSpecName "kube-api-access-h7llv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 21 21:46:17.385383 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:46:17.385346 2575 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4f4a0c63-45f1-4f45-b038-c2e2433900f6-istiod-ca-cert\") on node \"ip-10-0-140-97.ec2.internal\" DevicePath \"\"" Jul 21 21:46:17.385383 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:46:17.385381 2575 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4f4a0c63-45f1-4f45-b038-c2e2433900f6-istio-token\") on node \"ip-10-0-140-97.ec2.internal\" DevicePath \"\"" Jul 21 21:46:17.385594 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:46:17.385394 2575 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4f4a0c63-45f1-4f45-b038-c2e2433900f6-istio-podinfo\") on node \"ip-10-0-140-97.ec2.internal\" DevicePath \"\"" Jul 21 21:46:17.385594 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:46:17.385406 2575 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4f4a0c63-45f1-4f45-b038-c2e2433900f6-workload-certs\") on node \"ip-10-0-140-97.ec2.internal\" DevicePath \"\"" Jul 21 21:46:17.385594 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:46:17.385417 2575 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4f4a0c63-45f1-4f45-b038-c2e2433900f6-istio-data\") on node \"ip-10-0-140-97.ec2.internal\" DevicePath \"\"" Jul 21 21:46:17.385594 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:46:17.385429 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h7llv\" (UniqueName: \"kubernetes.io/projected/4f4a0c63-45f1-4f45-b038-c2e2433900f6-kube-api-access-h7llv\") on node \"ip-10-0-140-97.ec2.internal\" DevicePath \"\"" Jul 21 21:46:17.385594 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:46:17.385442 2575 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4f4a0c63-45f1-4f45-b038-c2e2433900f6-istio-envoy\") on node \"ip-10-0-140-97.ec2.internal\" DevicePath \"\"" Jul 21 21:46:18.048180 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:46:18.048142 2575 generic.go:356] "Generic (PLEG): container finished" podID="4f4a0c63-45f1-4f45-b038-c2e2433900f6" containerID="fb6cad868ca2384d672a281e4adc5bdfbe09d88cf7147e7a595207177e4d18bf" exitCode=0 Jul 21 21:46:18.048349 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:46:18.048186 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-e584ac-openshift-default-5d8bd494c-sslp5" Jul 21 21:46:18.048349 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:46:18.048228 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-e584ac-openshift-default-5d8bd494c-sslp5" event={"ID":"4f4a0c63-45f1-4f45-b038-c2e2433900f6","Type":"ContainerDied","Data":"fb6cad868ca2384d672a281e4adc5bdfbe09d88cf7147e7a595207177e4d18bf"} Jul 21 21:46:18.048349 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:46:18.048292 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-e584ac-openshift-default-5d8bd494c-sslp5" event={"ID":"4f4a0c63-45f1-4f45-b038-c2e2433900f6","Type":"ContainerDied","Data":"bc9f4f4cdad840b7308c9b0b8d5f933e2a6d58870ae06909b6c6390b91e13e3d"} Jul 21 21:46:18.048349 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:46:18.048321 2575 scope.go:117] "RemoveContainer" containerID="fb6cad868ca2384d672a281e4adc5bdfbe09d88cf7147e7a595207177e4d18bf" Jul 21 21:46:18.057561 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:46:18.057545 2575 scope.go:117] "RemoveContainer" containerID="00e47cff51276e6433b46cce20e4f05f30947f56940c942df6f6792f5831402f" Jul 21 21:46:18.065627 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:46:18.065611 2575 scope.go:117] "RemoveContainer" containerID="fb6cad868ca2384d672a281e4adc5bdfbe09d88cf7147e7a595207177e4d18bf" Jul 21 21:46:18.065880 ip-10-0-140-97 kubenswrapper[2575]: E0721 21:46:18.065862 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fb6cad868ca2384d672a281e4adc5bdfbe09d88cf7147e7a595207177e4d18bf\": container with ID starting with fb6cad868ca2384d672a281e4adc5bdfbe09d88cf7147e7a595207177e4d18bf not found: ID does not exist" containerID="fb6cad868ca2384d672a281e4adc5bdfbe09d88cf7147e7a595207177e4d18bf" Jul 21 21:46:18.065956 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:46:18.065892 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fb6cad868ca2384d672a281e4adc5bdfbe09d88cf7147e7a595207177e4d18bf"} err="failed to get container status \"fb6cad868ca2384d672a281e4adc5bdfbe09d88cf7147e7a595207177e4d18bf\": rpc error: code = NotFound desc = could not find container \"fb6cad868ca2384d672a281e4adc5bdfbe09d88cf7147e7a595207177e4d18bf\": container with ID starting with fb6cad868ca2384d672a281e4adc5bdfbe09d88cf7147e7a595207177e4d18bf not found: ID does not exist" Jul 21 21:46:18.065956 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:46:18.065914 2575 scope.go:117] "RemoveContainer" containerID="00e47cff51276e6433b46cce20e4f05f30947f56940c942df6f6792f5831402f" Jul 21 21:46:18.066156 ip-10-0-140-97 kubenswrapper[2575]: E0721 21:46:18.066138 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"00e47cff51276e6433b46cce20e4f05f30947f56940c942df6f6792f5831402f\": container with ID starting with 00e47cff51276e6433b46cce20e4f05f30947f56940c942df6f6792f5831402f not found: ID does not exist" containerID="00e47cff51276e6433b46cce20e4f05f30947f56940c942df6f6792f5831402f" Jul 21 21:46:18.066195 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:46:18.066161 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"00e47cff51276e6433b46cce20e4f05f30947f56940c942df6f6792f5831402f"} err="failed to get container status \"00e47cff51276e6433b46cce20e4f05f30947f56940c942df6f6792f5831402f\": rpc error: code = NotFound desc = could not find container \"00e47cff51276e6433b46cce20e4f05f30947f56940c942df6f6792f5831402f\": container with ID starting with 00e47cff51276e6433b46cce20e4f05f30947f56940c942df6f6792f5831402f not found: ID does not exist" Jul 21 21:46:18.070975 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:46:18.070933 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-e584ac-openshift-default-5d8bd494c-sslp5"] Jul 21 21:46:18.073077 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:46:18.073058 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-e584ac-openshift-default-5d8bd494c-sslp5"] Jul 21 21:46:18.743552 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:46:18.743499 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4f4a0c63-45f1-4f45-b038-c2e2433900f6" path="/var/lib/kubelet/pods/4f4a0c63-45f1-4f45-b038-c2e2433900f6/volumes" Jul 21 21:46:33.929880 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:46:33.929846 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-24ec74-openshift-default-58d5566c95-chm82"] Jul 21 21:46:33.930313 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:46:33.930039 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-24ec74-openshift-default-58d5566c95-chm82" podUID="d6099e69-dcca-4458-a948-29493d41605f" containerName="istio-proxy" containerID="cri-o://73aaeebdf509e0530e7116e04e1ca9cf9b3597c17cc517918c8ccc27bc7a0551" gracePeriod=30 Jul 21 21:46:39.154320 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:46:39.154286 2575 generic.go:356] "Generic (PLEG): container finished" podID="d6099e69-dcca-4458-a948-29493d41605f" containerID="73aaeebdf509e0530e7116e04e1ca9cf9b3597c17cc517918c8ccc27bc7a0551" exitCode=0 Jul 21 21:46:39.154731 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:46:39.154327 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-24ec74-openshift-default-58d5566c95-chm82" event={"ID":"d6099e69-dcca-4458-a948-29493d41605f","Type":"ContainerDied","Data":"73aaeebdf509e0530e7116e04e1ca9cf9b3597c17cc517918c8ccc27bc7a0551"} Jul 21 21:46:39.154731 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:46:39.154385 2575 scope.go:117] "RemoveContainer" containerID="2c7a272169452bb4d238b2e59dd9df6a920e179a53c23109a606877d1736dfde" Jul 21 21:46:39.188633 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:46:39.188613 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-24ec74-openshift-default-58d5566c95-chm82" Jul 21 21:46:39.290628 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:46:39.290524 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d6099e69-dcca-4458-a948-29493d41605f-istio-podinfo\") pod \"d6099e69-dcca-4458-a948-29493d41605f\" (UID: \"d6099e69-dcca-4458-a948-29493d41605f\") " Jul 21 21:46:39.290628 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:46:39.290584 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d6099e69-dcca-4458-a948-29493d41605f-istio-envoy\") pod \"d6099e69-dcca-4458-a948-29493d41605f\" (UID: \"d6099e69-dcca-4458-a948-29493d41605f\") " Jul 21 21:46:39.290628 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:46:39.290613 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6099e69-dcca-4458-a948-29493d41605f-workload-socket\") pod \"d6099e69-dcca-4458-a948-29493d41605f\" (UID: \"d6099e69-dcca-4458-a948-29493d41605f\") " Jul 21 21:46:39.290904 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:46:39.290655 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d6099e69-dcca-4458-a948-29493d41605f-workload-certs\") pod \"d6099e69-dcca-4458-a948-29493d41605f\" (UID: \"d6099e69-dcca-4458-a948-29493d41605f\") " Jul 21 21:46:39.290904 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:46:39.290676 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d6099e69-dcca-4458-a948-29493d41605f-istiod-ca-cert\") pod \"d6099e69-dcca-4458-a948-29493d41605f\" (UID: \"d6099e69-dcca-4458-a948-29493d41605f\") " Jul 21 21:46:39.290904 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:46:39.290695 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dddwv\" (UniqueName: \"kubernetes.io/projected/d6099e69-dcca-4458-a948-29493d41605f-kube-api-access-dddwv\") pod \"d6099e69-dcca-4458-a948-29493d41605f\" (UID: \"d6099e69-dcca-4458-a948-29493d41605f\") " Jul 21 21:46:39.290904 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:46:39.290740 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6099e69-dcca-4458-a948-29493d41605f-credential-socket\") pod \"d6099e69-dcca-4458-a948-29493d41605f\" (UID: \"d6099e69-dcca-4458-a948-29493d41605f\") " Jul 21 21:46:39.290904 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:46:39.290815 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d6099e69-dcca-4458-a948-29493d41605f-istio-data\") pod \"d6099e69-dcca-4458-a948-29493d41605f\" (UID: \"d6099e69-dcca-4458-a948-29493d41605f\") " Jul 21 21:46:39.290904 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:46:39.290859 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d6099e69-dcca-4458-a948-29493d41605f-istio-token\") pod \"d6099e69-dcca-4458-a948-29493d41605f\" (UID: \"d6099e69-dcca-4458-a948-29493d41605f\") " Jul 21 21:46:39.291223 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:46:39.290942 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d6099e69-dcca-4458-a948-29493d41605f-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "d6099e69-dcca-4458-a948-29493d41605f" (UID: "d6099e69-dcca-4458-a948-29493d41605f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 21:46:39.291223 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:46:39.291122 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d6099e69-dcca-4458-a948-29493d41605f-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "d6099e69-dcca-4458-a948-29493d41605f" (UID: "d6099e69-dcca-4458-a948-29493d41605f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 21:46:39.291223 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:46:39.291144 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d6099e69-dcca-4458-a948-29493d41605f-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "d6099e69-dcca-4458-a948-29493d41605f" (UID: "d6099e69-dcca-4458-a948-29493d41605f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 21 21:46:39.291223 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:46:39.291220 2575 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d6099e69-dcca-4458-a948-29493d41605f-workload-certs\") on node \"ip-10-0-140-97.ec2.internal\" DevicePath \"\"" Jul 21 21:46:39.291432 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:46:39.291241 2575 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6099e69-dcca-4458-a948-29493d41605f-credential-socket\") on node \"ip-10-0-140-97.ec2.internal\" DevicePath \"\"" Jul 21 21:46:39.291432 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:46:39.291311 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d6099e69-dcca-4458-a948-29493d41605f-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "d6099e69-dcca-4458-a948-29493d41605f" (UID: "d6099e69-dcca-4458-a948-29493d41605f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 21:46:39.291845 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:46:39.291825 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d6099e69-dcca-4458-a948-29493d41605f-istio-data" (OuterVolumeSpecName: "istio-data") pod "d6099e69-dcca-4458-a948-29493d41605f" (UID: "d6099e69-dcca-4458-a948-29493d41605f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 21:46:39.292945 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:46:39.292920 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d6099e69-dcca-4458-a948-29493d41605f-kube-api-access-dddwv" (OuterVolumeSpecName: "kube-api-access-dddwv") pod "d6099e69-dcca-4458-a948-29493d41605f" (UID: "d6099e69-dcca-4458-a948-29493d41605f"). InnerVolumeSpecName "kube-api-access-dddwv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 21 21:46:39.293065 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:46:39.292941 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d6099e69-dcca-4458-a948-29493d41605f-istio-token" (OuterVolumeSpecName: "istio-token") pod "d6099e69-dcca-4458-a948-29493d41605f" (UID: "d6099e69-dcca-4458-a948-29493d41605f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 21 21:46:39.293225 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:46:39.293203 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d6099e69-dcca-4458-a948-29493d41605f-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "d6099e69-dcca-4458-a948-29493d41605f" (UID: "d6099e69-dcca-4458-a948-29493d41605f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 21 21:46:39.293225 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:46:39.293212 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d6099e69-dcca-4458-a948-29493d41605f-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "d6099e69-dcca-4458-a948-29493d41605f" (UID: "d6099e69-dcca-4458-a948-29493d41605f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 21:46:39.392226 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:46:39.392194 2575 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6099e69-dcca-4458-a948-29493d41605f-workload-socket\") on node \"ip-10-0-140-97.ec2.internal\" DevicePath \"\"" Jul 21 21:46:39.392226 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:46:39.392223 2575 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d6099e69-dcca-4458-a948-29493d41605f-istiod-ca-cert\") on node \"ip-10-0-140-97.ec2.internal\" DevicePath \"\"" Jul 21 21:46:39.392226 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:46:39.392232 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dddwv\" (UniqueName: \"kubernetes.io/projected/d6099e69-dcca-4458-a948-29493d41605f-kube-api-access-dddwv\") on node \"ip-10-0-140-97.ec2.internal\" DevicePath \"\"" Jul 21 21:46:39.392226 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:46:39.392243 2575 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d6099e69-dcca-4458-a948-29493d41605f-istio-data\") on node \"ip-10-0-140-97.ec2.internal\" DevicePath \"\"" Jul 21 21:46:39.392508 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:46:39.392254 2575 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d6099e69-dcca-4458-a948-29493d41605f-istio-token\") on node \"ip-10-0-140-97.ec2.internal\" DevicePath \"\"" Jul 21 21:46:39.392508 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:46:39.392262 2575 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d6099e69-dcca-4458-a948-29493d41605f-istio-podinfo\") on node \"ip-10-0-140-97.ec2.internal\" DevicePath \"\"" Jul 21 21:46:39.392508 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:46:39.392270 2575 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d6099e69-dcca-4458-a948-29493d41605f-istio-envoy\") on node \"ip-10-0-140-97.ec2.internal\" DevicePath \"\"" Jul 21 21:46:40.161104 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:46:40.161075 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-24ec74-openshift-default-58d5566c95-chm82" Jul 21 21:46:40.161104 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:46:40.161098 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-24ec74-openshift-default-58d5566c95-chm82" event={"ID":"d6099e69-dcca-4458-a948-29493d41605f","Type":"ContainerDied","Data":"02e3e63fb47253900348a79ab9251b8a9731405518429714964bb106925ed5bf"} Jul 21 21:46:40.161585 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:46:40.161153 2575 scope.go:117] "RemoveContainer" containerID="73aaeebdf509e0530e7116e04e1ca9cf9b3597c17cc517918c8ccc27bc7a0551" Jul 21 21:46:40.183185 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:46:40.183159 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-24ec74-openshift-default-58d5566c95-chm82"] Jul 21 21:46:40.186861 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:46:40.186838 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-24ec74-openshift-default-58d5566c95-chm82"] Jul 21 21:46:40.742145 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:46:40.742114 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d6099e69-dcca-4458-a948-29493d41605f" path="/var/lib/kubelet/pods/d6099e69-dcca-4458-a948-29493d41605f/volumes" Jul 21 21:46:41.221643 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:46:41.221616 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-694dcbb678-vqdp9_a2248fc1-a1c0-461a-87dd-8f6f9cb0d634/copy-manifests/0.log" Jul 21 21:46:41.229657 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:46:41.229637 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-694dcbb678-vqdp9_a2248fc1-a1c0-461a-87dd-8f6f9cb0d634/manager/0.log" Jul 21 21:46:41.705237 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:46:41.705214 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29744475-s4gdw_23c11dc8-340e-413d-9cfa-19db16bb4708/cleanup/0.log" Jul 21 21:46:41.822639 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:46:41.822608 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29744490-jrddn_41509374-45f4-438c-946a-3518b35ffae7/cleanup/0.log" Jul 21 21:46:43.563701 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:46:43.563666 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-75cc8d5cb7-b45rs_296877a3-26aa-437f-9970-c5d99bfdbd94/manager/0.log" Jul 21 21:46:43.931549 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:46:43.931497 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-catalog-d4ms7_a50b9c14-3c3f-47f9-bacd-3cbdc890fb93/registry-server/0.log" Jul 21 21:46:44.060002 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:46:44.059963 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-86878c769f-zl6lh_2e172821-8e5a-4f3d-87cc-8742d612b230/manager/0.log" Jul 21 21:46:44.720600 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:46:44.720568 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-6576859cf4v97t9_4573f5c2-4d6e-4c0d-a850-55b738a81de7/istio-proxy/0.log" Jul 21 21:46:45.221271 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:46:45.221242 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-2pnp6_34d429d3-1614-43f5-997f-492caa021783/istio-proxy/1.log" Jul 21 21:46:45.580258 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:46:45.580184 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-69864f84f6-qzxnd_2adb651d-e8aa-4e76-878c-e5cdafafbe5f/router/0.log" Jul 21 21:46:46.091736 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:46:46.091705 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-59975d69f9-bz46f_7f45b356-8eac-4587-9cf0-183e1a86555a/main/0.log" Jul 21 21:46:46.224434 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:46:46.224407 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-trlp-test-simulated-kserve-598466548d-tqkl4_87408656-7347-480e-9e6e-937e499c6f6a/main/0.log" Jul 21 21:46:52.877697 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:46:52.877662 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-b4hjj/must-gather-sbrkn"] Jul 21 21:46:52.878069 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:46:52.878000 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8b710e33-df66-4f10-8233-56fc35fcf4f6" containerName="cleanup" Jul 21 21:46:52.878069 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:46:52.878010 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="8b710e33-df66-4f10-8233-56fc35fcf4f6" containerName="cleanup" Jul 21 21:46:52.878069 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:46:52.878020 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d6099e69-dcca-4458-a948-29493d41605f" containerName="istio-proxy" Jul 21 21:46:52.878069 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:46:52.878026 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="d6099e69-dcca-4458-a948-29493d41605f" containerName="istio-proxy" Jul 21 21:46:52.878069 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:46:52.878033 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4ceeb2fd-b5af-4715-8c77-5c041b61a08b" containerName="main" Jul 21 21:46:52.878069 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:46:52.878038 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="4ceeb2fd-b5af-4715-8c77-5c041b61a08b" containerName="main" Jul 21 21:46:52.878069 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:46:52.878054 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4f4a0c63-45f1-4f45-b038-c2e2433900f6" containerName="istio-proxy" Jul 21 21:46:52.878069 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:46:52.878059 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="4f4a0c63-45f1-4f45-b038-c2e2433900f6" containerName="istio-proxy" Jul 21 21:46:52.878069 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:46:52.878065 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4f4a0c63-45f1-4f45-b038-c2e2433900f6" containerName="istio-proxy" Jul 21 21:46:52.878069 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:46:52.878070 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="4f4a0c63-45f1-4f45-b038-c2e2433900f6" containerName="istio-proxy" Jul 21 21:46:52.878390 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:46:52.878129 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="4f4a0c63-45f1-4f45-b038-c2e2433900f6" containerName="istio-proxy" Jul 21 21:46:52.878390 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:46:52.878136 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="d6099e69-dcca-4458-a948-29493d41605f" containerName="istio-proxy" Jul 21 21:46:52.878390 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:46:52.878143 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="4ceeb2fd-b5af-4715-8c77-5c041b61a08b" containerName="main" Jul 21 21:46:52.878390 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:46:52.878151 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="4f4a0c63-45f1-4f45-b038-c2e2433900f6" containerName="istio-proxy" Jul 21 21:46:52.878390 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:46:52.878157 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="8b710e33-df66-4f10-8233-56fc35fcf4f6" containerName="cleanup" Jul 21 21:46:52.878390 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:46:52.878210 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d6099e69-dcca-4458-a948-29493d41605f" containerName="istio-proxy" Jul 21 21:46:52.878390 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:46:52.878216 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="d6099e69-dcca-4458-a948-29493d41605f" containerName="istio-proxy" Jul 21 21:46:52.878390 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:46:52.878277 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="d6099e69-dcca-4458-a948-29493d41605f" containerName="istio-proxy" Jul 21 21:46:52.881207 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:46:52.881187 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-b4hjj/must-gather-sbrkn" Jul 21 21:46:52.883933 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:46:52.883908 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-b4hjj\"/\"kube-root-ca.crt\"" Jul 21 21:46:52.884052 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:46:52.883985 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-b4hjj\"/\"default-dockercfg-p8l7g\"" Jul 21 21:46:52.884422 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:46:52.884390 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-b4hjj\"/\"openshift-service-ca.crt\"" Jul 21 21:46:52.892109 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:46:52.892089 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-b4hjj/must-gather-sbrkn"] Jul 21 21:46:52.908996 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:46:52.908963 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5cvpt\" (UniqueName: \"kubernetes.io/projected/57160256-4976-4861-8adc-792bc18319c2-kube-api-access-5cvpt\") pod \"must-gather-sbrkn\" (UID: \"57160256-4976-4861-8adc-792bc18319c2\") " pod="openshift-must-gather-b4hjj/must-gather-sbrkn" Jul 21 21:46:52.909126 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:46:52.909072 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/57160256-4976-4861-8adc-792bc18319c2-must-gather-output\") pod \"must-gather-sbrkn\" (UID: \"57160256-4976-4861-8adc-792bc18319c2\") " pod="openshift-must-gather-b4hjj/must-gather-sbrkn" Jul 21 21:46:53.009800 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:46:53.009774 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/57160256-4976-4861-8adc-792bc18319c2-must-gather-output\") pod \"must-gather-sbrkn\" (UID: \"57160256-4976-4861-8adc-792bc18319c2\") " pod="openshift-must-gather-b4hjj/must-gather-sbrkn" Jul 21 21:46:53.009922 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:46:53.009873 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5cvpt\" (UniqueName: \"kubernetes.io/projected/57160256-4976-4861-8adc-792bc18319c2-kube-api-access-5cvpt\") pod \"must-gather-sbrkn\" (UID: \"57160256-4976-4861-8adc-792bc18319c2\") " pod="openshift-must-gather-b4hjj/must-gather-sbrkn" Jul 21 21:46:53.010173 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:46:53.010141 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/57160256-4976-4861-8adc-792bc18319c2-must-gather-output\") pod \"must-gather-sbrkn\" (UID: \"57160256-4976-4861-8adc-792bc18319c2\") " pod="openshift-must-gather-b4hjj/must-gather-sbrkn" Jul 21 21:46:53.019404 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:46:53.019373 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5cvpt\" (UniqueName: \"kubernetes.io/projected/57160256-4976-4861-8adc-792bc18319c2-kube-api-access-5cvpt\") pod \"must-gather-sbrkn\" (UID: \"57160256-4976-4861-8adc-792bc18319c2\") " pod="openshift-must-gather-b4hjj/must-gather-sbrkn" Jul 21 21:46:53.190298 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:46:53.190220 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-b4hjj/must-gather-sbrkn" Jul 21 21:46:53.326365 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:46:53.326334 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-b4hjj/must-gather-sbrkn"] Jul 21 21:46:53.329670 ip-10-0-140-97 kubenswrapper[2575]: W0721 21:46:53.329646 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod57160256_4976_4861_8adc_792bc18319c2.slice/crio-1c7cb3f305b0915a528c4b74ca4ec622cf1daf246cd4b2ff717dceb346c289ae WatchSource:0}: Error finding container 1c7cb3f305b0915a528c4b74ca4ec622cf1daf246cd4b2ff717dceb346c289ae: Status 404 returned error can't find the container with id 1c7cb3f305b0915a528c4b74ca4ec622cf1daf246cd4b2ff717dceb346c289ae Jul 21 21:46:53.331494 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:46:53.331477 2575 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 21 21:46:54.228330 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:46:54.228285 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-b4hjj/must-gather-sbrkn" event={"ID":"57160256-4976-4861-8adc-792bc18319c2","Type":"ContainerStarted","Data":"1c7cb3f305b0915a528c4b74ca4ec622cf1daf246cd4b2ff717dceb346c289ae"} Jul 21 21:46:55.236732 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:46:55.236693 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-b4hjj/must-gather-sbrkn" event={"ID":"57160256-4976-4861-8adc-792bc18319c2","Type":"ContainerStarted","Data":"18c65cd35fcef9bf2ce8a70bc75c361930770ab3541d8ca00877fe0a09757fb2"} Jul 21 21:46:55.236732 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:46:55.236739 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-b4hjj/must-gather-sbrkn" event={"ID":"57160256-4976-4861-8adc-792bc18319c2","Type":"ContainerStarted","Data":"5e1068ad60145c2a7e549aa79e954715f624dd46dedf1e8bb9487692c73ab9ef"} Jul 21 21:46:55.254692 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:46:55.254637 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-b4hjj/must-gather-sbrkn" podStartSLOduration=2.282407578 podStartE2EDuration="3.254618383s" totalImagesPullingTime="972.210805ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 21:46:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 21:46:55.253479504 +0000 UTC m=+2773.188229231" watchObservedRunningTime="2026-07-21 21:46:55.254618383 +0000 UTC m=+2773.189368054" Jul 21 21:46:55.861497 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:46:55.861466 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-5694f_e3dcf4f7-a30b-46f5-9c5d-a77f61d88c51/global-pull-secret-syncer/0.log" Jul 21 21:46:56.047084 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:46:56.047055 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-cq6dd_aaa2e752-150d-4b89-852d-64473f98f593/konnectivity-agent/0.log" Jul 21 21:46:56.168175 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:46:56.168098 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-140-97.ec2.internal_810679f65980473b59e3cca08ad4f0fc/haproxy/0.log" Jul 21 21:47:00.756313 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:47:00.756262 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-75cc8d5cb7-b45rs_296877a3-26aa-437f-9970-c5d99bfdbd94/manager/0.log" Jul 21 21:47:00.904215 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:47:00.904185 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-catalog-d4ms7_a50b9c14-3c3f-47f9-bacd-3cbdc890fb93/registry-server/0.log" Jul 21 21:47:01.135516 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:47:01.135371 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-86878c769f-zl6lh_2e172821-8e5a-4f3d-87cc-8742d612b230/manager/0.log" Jul 21 21:47:03.272373 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:47:03.272272 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-76cf8966f8-7px4w_76e94cf3-ab49-4698-8ac9-bf3ad1dcdf3a/cluster-monitoring-operator/0.log" Jul 21 21:47:03.301479 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:47:03.301439 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-56c5594674-229p4_083847dd-e890-49f1-8873-5da46f1417e3/kube-state-metrics/0.log" Jul 21 21:47:03.323609 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:47:03.323556 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-56c5594674-229p4_083847dd-e890-49f1-8873-5da46f1417e3/kube-rbac-proxy-main/0.log" Jul 21 21:47:03.350252 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:47:03.350213 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-56c5594674-229p4_083847dd-e890-49f1-8873-5da46f1417e3/kube-rbac-proxy-self/0.log" Jul 21 21:47:03.528388 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:47:03.528265 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-cc8rp_90e9470b-56e7-4b66-9e86-622407c3b7c1/node-exporter/0.log" Jul 21 21:47:03.555825 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:47:03.555735 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-cc8rp_90e9470b-56e7-4b66-9e86-622407c3b7c1/kube-rbac-proxy/0.log" Jul 21 21:47:03.582678 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:47:03.582590 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-cc8rp_90e9470b-56e7-4b66-9e86-622407c3b7c1/init-textfile/0.log" Jul 21 21:47:03.807772 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:47:03.807686 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_24fd0b69-36fa-4f00-89ab-77b624152a32/prometheus/0.log" Jul 21 21:47:03.836372 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:47:03.836334 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_24fd0b69-36fa-4f00-89ab-77b624152a32/config-reloader/0.log" Jul 21 21:47:03.863695 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:47:03.863663 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_24fd0b69-36fa-4f00-89ab-77b624152a32/thanos-sidecar/0.log" Jul 21 21:47:03.887215 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:47:03.887185 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_24fd0b69-36fa-4f00-89ab-77b624152a32/kube-rbac-proxy-web/0.log" Jul 21 21:47:03.912071 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:47:03.912044 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_24fd0b69-36fa-4f00-89ab-77b624152a32/kube-rbac-proxy/0.log" Jul 21 21:47:03.939783 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:47:03.939756 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_24fd0b69-36fa-4f00-89ab-77b624152a32/kube-rbac-proxy-thanos/0.log" Jul 21 21:47:03.967060 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:47:03.967037 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_24fd0b69-36fa-4f00-89ab-77b624152a32/init-config-reloader/0.log" Jul 21 21:47:04.000038 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:47:04.000013 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-5945b5cfd-lfqxs_705f56be-95fb-478b-90ff-f89ebac1baa7/prometheus-operator/0.log" Jul 21 21:47:04.021620 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:47:04.021589 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-5945b5cfd-lfqxs_705f56be-95fb-478b-90ff-f89ebac1baa7/kube-rbac-proxy/0.log" Jul 21 21:47:04.050133 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:47:04.050094 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-85574b5f49-shswg_c922616d-f0b6-4513-a1d0-9cc5e2611708/prometheus-operator-admission-webhook/0.log" Jul 21 21:47:05.556618 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:47:05.556514 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-7bc7489cb5-6jvh2_c981da1f-67fe-49df-9230-91e4215eef08/networking-console-plugin/0.log" Jul 21 21:47:06.215825 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:47:06.215781 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-b4hjj/perf-node-gather-daemonset-28mhv"] Jul 21 21:47:06.221563 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:47:06.221513 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-b4hjj/perf-node-gather-daemonset-28mhv" Jul 21 21:47:06.227501 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:47:06.227467 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-b4hjj/perf-node-gather-daemonset-28mhv"] Jul 21 21:47:06.359712 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:47:06.359682 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5rq6p\" (UniqueName: \"kubernetes.io/projected/1cf41490-9322-4e3b-b14f-c2352887e90f-kube-api-access-5rq6p\") pod \"perf-node-gather-daemonset-28mhv\" (UID: \"1cf41490-9322-4e3b-b14f-c2352887e90f\") " pod="openshift-must-gather-b4hjj/perf-node-gather-daemonset-28mhv" Jul 21 21:47:06.359930 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:47:06.359915 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/1cf41490-9322-4e3b-b14f-c2352887e90f-proc\") pod \"perf-node-gather-daemonset-28mhv\" (UID: \"1cf41490-9322-4e3b-b14f-c2352887e90f\") " pod="openshift-must-gather-b4hjj/perf-node-gather-daemonset-28mhv" Jul 21 21:47:06.360025 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:47:06.360014 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/1cf41490-9322-4e3b-b14f-c2352887e90f-podres\") pod \"perf-node-gather-daemonset-28mhv\" (UID: \"1cf41490-9322-4e3b-b14f-c2352887e90f\") " pod="openshift-must-gather-b4hjj/perf-node-gather-daemonset-28mhv" Jul 21 21:47:06.360153 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:47:06.360135 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1cf41490-9322-4e3b-b14f-c2352887e90f-sys\") pod \"perf-node-gather-daemonset-28mhv\" (UID: \"1cf41490-9322-4e3b-b14f-c2352887e90f\") " pod="openshift-must-gather-b4hjj/perf-node-gather-daemonset-28mhv" Jul 21 21:47:06.360281 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:47:06.360265 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/1cf41490-9322-4e3b-b14f-c2352887e90f-lib-modules\") pod \"perf-node-gather-daemonset-28mhv\" (UID: \"1cf41490-9322-4e3b-b14f-c2352887e90f\") " pod="openshift-must-gather-b4hjj/perf-node-gather-daemonset-28mhv" Jul 21 21:47:06.461538 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:47:06.461500 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5rq6p\" (UniqueName: \"kubernetes.io/projected/1cf41490-9322-4e3b-b14f-c2352887e90f-kube-api-access-5rq6p\") pod \"perf-node-gather-daemonset-28mhv\" (UID: \"1cf41490-9322-4e3b-b14f-c2352887e90f\") " pod="openshift-must-gather-b4hjj/perf-node-gather-daemonset-28mhv" Jul 21 21:47:06.461744 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:47:06.461585 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/1cf41490-9322-4e3b-b14f-c2352887e90f-proc\") pod \"perf-node-gather-daemonset-28mhv\" (UID: \"1cf41490-9322-4e3b-b14f-c2352887e90f\") " pod="openshift-must-gather-b4hjj/perf-node-gather-daemonset-28mhv" Jul 21 21:47:06.461744 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:47:06.461624 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/1cf41490-9322-4e3b-b14f-c2352887e90f-podres\") pod \"perf-node-gather-daemonset-28mhv\" (UID: \"1cf41490-9322-4e3b-b14f-c2352887e90f\") " pod="openshift-must-gather-b4hjj/perf-node-gather-daemonset-28mhv" Jul 21 21:47:06.461744 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:47:06.461673 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1cf41490-9322-4e3b-b14f-c2352887e90f-sys\") pod \"perf-node-gather-daemonset-28mhv\" (UID: \"1cf41490-9322-4e3b-b14f-c2352887e90f\") " pod="openshift-must-gather-b4hjj/perf-node-gather-daemonset-28mhv" Jul 21 21:47:06.461744 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:47:06.461701 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/1cf41490-9322-4e3b-b14f-c2352887e90f-lib-modules\") pod \"perf-node-gather-daemonset-28mhv\" (UID: \"1cf41490-9322-4e3b-b14f-c2352887e90f\") " pod="openshift-must-gather-b4hjj/perf-node-gather-daemonset-28mhv" Jul 21 21:47:06.461994 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:47:06.461743 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/1cf41490-9322-4e3b-b14f-c2352887e90f-proc\") pod \"perf-node-gather-daemonset-28mhv\" (UID: \"1cf41490-9322-4e3b-b14f-c2352887e90f\") " pod="openshift-must-gather-b4hjj/perf-node-gather-daemonset-28mhv" Jul 21 21:47:06.461994 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:47:06.461887 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1cf41490-9322-4e3b-b14f-c2352887e90f-sys\") pod \"perf-node-gather-daemonset-28mhv\" (UID: \"1cf41490-9322-4e3b-b14f-c2352887e90f\") " pod="openshift-must-gather-b4hjj/perf-node-gather-daemonset-28mhv" Jul 21 21:47:06.461994 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:47:06.461899 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/1cf41490-9322-4e3b-b14f-c2352887e90f-lib-modules\") pod \"perf-node-gather-daemonset-28mhv\" (UID: \"1cf41490-9322-4e3b-b14f-c2352887e90f\") " pod="openshift-must-gather-b4hjj/perf-node-gather-daemonset-28mhv" Jul 21 21:47:06.461994 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:47:06.461913 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/1cf41490-9322-4e3b-b14f-c2352887e90f-podres\") pod \"perf-node-gather-daemonset-28mhv\" (UID: \"1cf41490-9322-4e3b-b14f-c2352887e90f\") " pod="openshift-must-gather-b4hjj/perf-node-gather-daemonset-28mhv" Jul 21 21:47:06.472801 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:47:06.472734 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5rq6p\" (UniqueName: \"kubernetes.io/projected/1cf41490-9322-4e3b-b14f-c2352887e90f-kube-api-access-5rq6p\") pod \"perf-node-gather-daemonset-28mhv\" (UID: \"1cf41490-9322-4e3b-b14f-c2352887e90f\") " pod="openshift-must-gather-b4hjj/perf-node-gather-daemonset-28mhv" Jul 21 21:47:06.537369 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:47:06.537048 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-b4hjj/perf-node-gather-daemonset-28mhv" Jul 21 21:47:06.715792 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:47:06.715747 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-b4hjj/perf-node-gather-daemonset-28mhv"] Jul 21 21:47:06.775561 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:47:06.775517 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-6675f745fd-fswbm_cef40b45-cb9f-4e56-adaa-f7be7e5f7c2d/console/0.log" Jul 21 21:47:06.816111 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:47:06.816082 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-5bc67dcd46-gqslk_e5751a54-5d9c-4f26-9007-74c141b8ace9/download-server/0.log" Jul 21 21:47:07.344330 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:47:07.344293 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-b4hjj/perf-node-gather-daemonset-28mhv" event={"ID":"1cf41490-9322-4e3b-b14f-c2352887e90f","Type":"ContainerStarted","Data":"dffb332f63fde9245a92aca4d30f261ddd22a11527a2b8c4f5b2d3b33bf563ae"} Jul 21 21:47:07.344509 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:47:07.344351 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-b4hjj/perf-node-gather-daemonset-28mhv" event={"ID":"1cf41490-9322-4e3b-b14f-c2352887e90f","Type":"ContainerStarted","Data":"3b6b39dec314d2cd692d5d2cd730cd6d79317225d9ce6400480075b37e55edbb"} Jul 21 21:47:07.344584 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:47:07.344504 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-b4hjj/perf-node-gather-daemonset-28mhv" Jul 21 21:47:07.362293 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:47:07.362229 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-b4hjj/perf-node-gather-daemonset-28mhv" podStartSLOduration=1.362207875 podStartE2EDuration="1.362207875s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-21 21:47:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 21:47:07.361783898 +0000 UTC m=+2785.296533581" watchObservedRunningTime="2026-07-21 21:47:07.362207875 +0000 UTC m=+2785.296957542" Jul 21 21:47:08.418418 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:47:08.418388 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-bfws8_e057811f-88be-47b7-9d41-4815de59f0df/dns/0.log" Jul 21 21:47:08.439832 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:47:08.439800 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-bfws8_e057811f-88be-47b7-9d41-4815de59f0df/kube-rbac-proxy/0.log" Jul 21 21:47:08.605136 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:47:08.605051 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-vdkfn_3877fdb0-b49b-444a-8f8c-597d3f935383/dns-node-resolver/0.log" Jul 21 21:47:09.233992 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:47:09.233962 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-jhz69_ce7f9a12-9e5b-4634-83d3-babc1e854a66/node-ca/0.log" Jul 21 21:47:10.269035 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:47:10.269001 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-6576859cf4v97t9_4573f5c2-4d6e-4c0d-a850-55b738a81de7/istio-proxy/0.log" Jul 21 21:47:10.487353 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:47:10.487324 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-2pnp6_34d429d3-1614-43f5-997f-492caa021783/istio-proxy/0.log" Jul 21 21:47:10.525222 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:47:10.525135 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-2pnp6_34d429d3-1614-43f5-997f-492caa021783/istio-proxy/1.log" Jul 21 21:47:10.612256 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:47:10.612220 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-69864f84f6-qzxnd_2adb651d-e8aa-4e76-878c-e5cdafafbe5f/router/0.log" Jul 21 21:47:11.352897 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:47:11.352861 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-hfmtx_4f86eef6-207f-4ad3-baa2-98cef60dbe52/serve-healthcheck-canary/0.log" Jul 21 21:47:12.058810 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:47:12.058784 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-wbwbl_4ebec69c-3c62-4d4b-bb73-3a170576ec58/kube-rbac-proxy/0.log" Jul 21 21:47:12.095989 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:47:12.095955 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-wbwbl_4ebec69c-3c62-4d4b-bb73-3a170576ec58/exporter/0.log" Jul 21 21:47:12.139408 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:47:12.139373 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-wbwbl_4ebec69c-3c62-4d4b-bb73-3a170576ec58/extractor/0.log" Jul 21 21:47:13.361932 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:47:13.361904 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-b4hjj/perf-node-gather-daemonset-28mhv" Jul 21 21:47:14.901262 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:47:14.901225 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-694dcbb678-vqdp9_a2248fc1-a1c0-461a-87dd-8f6f9cb0d634/manager/0.log" Jul 21 21:47:14.924692 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:47:14.924664 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-694dcbb678-vqdp9_a2248fc1-a1c0-461a-87dd-8f6f9cb0d634/copy-manifests/0.log" Jul 21 21:47:16.377269 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:47:16.377242 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-8f965b796-8wsc2_cbc0cc0e-a3cf-48d3-9417-64d1cda9f6ab/manager/0.log" Jul 21 21:47:16.453492 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:47:16.453451 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_openshift-lws-operator-fd8ccff4c-pxdxj_edaed366-c509-46cb-8d00-555f055259e2/openshift-lws-operator/0.log" Jul 21 21:47:16.529878 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:47:16.529842 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-6576859cf4v97t9_4573f5c2-4d6e-4c0d-a850-55b738a81de7/istio-proxy/0.log" Jul 21 21:47:16.699316 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:47:16.699078 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-2pnp6_34d429d3-1614-43f5-997f-492caa021783/istio-proxy/0.log" Jul 21 21:47:16.734131 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:47:16.734098 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-2pnp6_34d429d3-1614-43f5-997f-492caa021783/istio-proxy/1.log" Jul 21 21:47:16.797015 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:47:16.796972 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-69864f84f6-qzxnd_2adb651d-e8aa-4e76-878c-e5cdafafbe5f/router/0.log" Jul 21 21:47:18.054682 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:47:18.054656 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-6576859cf4v97t9_4573f5c2-4d6e-4c0d-a850-55b738a81de7/istio-proxy/0.log" Jul 21 21:47:18.256427 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:47:18.256401 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-2pnp6_34d429d3-1614-43f5-997f-492caa021783/istio-proxy/0.log" Jul 21 21:47:18.290390 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:47:18.290334 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-2pnp6_34d429d3-1614-43f5-997f-492caa021783/istio-proxy/1.log" Jul 21 21:47:18.360753 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:47:18.360717 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-69864f84f6-qzxnd_2adb651d-e8aa-4e76-878c-e5cdafafbe5f/router/0.log" Jul 21 21:47:22.188914 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:47:22.188838 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-7658d9b749-7zc94_ca2c8246-50a5-48fb-be89-a95b401523a7/kube-storage-version-migrator-operator/1.log" Jul 21 21:47:22.189812 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:47:22.189782 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-7658d9b749-7zc94_ca2c8246-50a5-48fb-be89-a95b401523a7/kube-storage-version-migrator-operator/0.log" Jul 21 21:47:23.181676 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:47:23.181648 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-952f6_d648b77f-43a8-4540-b59c-63768e78c245/kube-multus-additional-cni-plugins/0.log" Jul 21 21:47:23.205975 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:47:23.205948 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-952f6_d648b77f-43a8-4540-b59c-63768e78c245/egress-router-binary-copy/0.log" Jul 21 21:47:23.230170 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:47:23.230146 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-952f6_d648b77f-43a8-4540-b59c-63768e78c245/cni-plugins/0.log" Jul 21 21:47:23.257018 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:47:23.256984 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-952f6_d648b77f-43a8-4540-b59c-63768e78c245/bond-cni-plugin/0.log" Jul 21 21:47:23.282415 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:47:23.282394 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-952f6_d648b77f-43a8-4540-b59c-63768e78c245/routeoverride-cni/0.log" Jul 21 21:47:23.304976 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:47:23.304956 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-952f6_d648b77f-43a8-4540-b59c-63768e78c245/whereabouts-cni-bincopy/0.log" Jul 21 21:47:23.328759 ip-10-0-140-97 kubenswrapper[2575]: I0721 21:47:23.328741 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-952f6_d648b77f-43a8-4540-b59c-63768e78c245/whereabouts-cni/0.log"