Jul 18 11:48:51.658751 ip-10-0-136-240 systemd[1]: Starting Kubernetes Kubelet... Jul 18 11:48:52.106859 ip-10-0-136-240 kubenswrapper[2585]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 18 11:48:52.106859 ip-10-0-136-240 kubenswrapper[2585]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Jul 18 11:48:52.106859 ip-10-0-136-240 kubenswrapper[2585]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 18 11:48:52.106859 ip-10-0-136-240 kubenswrapper[2585]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Jul 18 11:48:52.106859 ip-10-0-136-240 kubenswrapper[2585]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 18 11:48:52.107699 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:52.107571 2585 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Jul 18 11:48:52.116968 ip-10-0-136-240 kubenswrapper[2585]: W0718 11:48:52.116950 2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 18 11:48:52.116968 ip-10-0-136-240 kubenswrapper[2585]: W0718 11:48:52.116965 2585 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 18 11:48:52.116968 ip-10-0-136-240 kubenswrapper[2585]: W0718 11:48:52.116968 2585 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 18 11:48:52.116968 ip-10-0-136-240 kubenswrapper[2585]: W0718 11:48:52.116972 2585 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 18 11:48:52.117094 ip-10-0-136-240 kubenswrapper[2585]: W0718 11:48:52.116975 2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 18 11:48:52.117094 ip-10-0-136-240 kubenswrapper[2585]: W0718 11:48:52.116978 2585 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 18 11:48:52.117094 ip-10-0-136-240 kubenswrapper[2585]: W0718 11:48:52.116981 2585 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 18 11:48:52.117094 ip-10-0-136-240 kubenswrapper[2585]: W0718 11:48:52.116984 2585 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 18 11:48:52.117094 ip-10-0-136-240 kubenswrapper[2585]: W0718 11:48:52.116988 2585 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 18 11:48:52.117094 ip-10-0-136-240 kubenswrapper[2585]: W0718 11:48:52.116990 2585 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 18 11:48:52.117094 ip-10-0-136-240 kubenswrapper[2585]: W0718 11:48:52.116993 2585 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 18 11:48:52.117094 ip-10-0-136-240 kubenswrapper[2585]: W0718 11:48:52.116996 2585 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 18 11:48:52.117094 ip-10-0-136-240 kubenswrapper[2585]: W0718 11:48:52.116998 2585 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 18 11:48:52.117094 ip-10-0-136-240 kubenswrapper[2585]: W0718 11:48:52.117002 2585 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 18 11:48:52.117094 ip-10-0-136-240 kubenswrapper[2585]: W0718 11:48:52.117005 2585 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 18 11:48:52.117094 ip-10-0-136-240 kubenswrapper[2585]: W0718 11:48:52.117008 2585 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 18 11:48:52.117094 ip-10-0-136-240 kubenswrapper[2585]: W0718 11:48:52.117010 2585 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 18 11:48:52.117094 ip-10-0-136-240 kubenswrapper[2585]: W0718 11:48:52.117013 2585 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 18 11:48:52.117094 ip-10-0-136-240 kubenswrapper[2585]: W0718 11:48:52.117015 2585 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 18 11:48:52.117094 ip-10-0-136-240 kubenswrapper[2585]: W0718 11:48:52.117018 2585 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 18 11:48:52.117094 ip-10-0-136-240 kubenswrapper[2585]: W0718 11:48:52.117020 2585 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 18 11:48:52.117094 ip-10-0-136-240 kubenswrapper[2585]: W0718 11:48:52.117023 2585 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 18 11:48:52.117094 ip-10-0-136-240 kubenswrapper[2585]: W0718 11:48:52.117025 2585 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 18 11:48:52.117604 ip-10-0-136-240 kubenswrapper[2585]: W0718 11:48:52.117030 2585 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 18 11:48:52.117604 ip-10-0-136-240 kubenswrapper[2585]: W0718 11:48:52.117034 2585 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 18 11:48:52.117604 ip-10-0-136-240 kubenswrapper[2585]: W0718 11:48:52.117038 2585 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 18 11:48:52.117604 ip-10-0-136-240 kubenswrapper[2585]: W0718 11:48:52.117041 2585 feature_gate.go:328] unrecognized feature gate: Example Jul 18 11:48:52.117604 ip-10-0-136-240 kubenswrapper[2585]: W0718 11:48:52.117044 2585 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 18 11:48:52.117604 ip-10-0-136-240 kubenswrapper[2585]: W0718 11:48:52.117046 2585 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 18 11:48:52.117604 ip-10-0-136-240 kubenswrapper[2585]: W0718 11:48:52.117049 2585 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 18 11:48:52.117604 ip-10-0-136-240 kubenswrapper[2585]: W0718 11:48:52.117051 2585 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 18 11:48:52.117604 ip-10-0-136-240 kubenswrapper[2585]: W0718 11:48:52.117054 2585 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 18 11:48:52.117604 ip-10-0-136-240 kubenswrapper[2585]: W0718 11:48:52.117056 2585 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 18 11:48:52.117604 ip-10-0-136-240 kubenswrapper[2585]: W0718 11:48:52.117059 2585 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 18 11:48:52.117604 ip-10-0-136-240 kubenswrapper[2585]: W0718 11:48:52.117061 2585 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 18 11:48:52.117604 ip-10-0-136-240 kubenswrapper[2585]: W0718 11:48:52.117064 2585 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 18 11:48:52.117604 ip-10-0-136-240 kubenswrapper[2585]: W0718 11:48:52.117066 2585 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 18 11:48:52.117604 ip-10-0-136-240 kubenswrapper[2585]: W0718 11:48:52.117070 2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 18 11:48:52.117604 ip-10-0-136-240 kubenswrapper[2585]: W0718 11:48:52.117073 2585 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 18 11:48:52.117604 ip-10-0-136-240 kubenswrapper[2585]: W0718 11:48:52.117076 2585 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 18 11:48:52.117604 ip-10-0-136-240 kubenswrapper[2585]: W0718 11:48:52.117080 2585 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 18 11:48:52.117604 ip-10-0-136-240 kubenswrapper[2585]: W0718 11:48:52.117083 2585 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 18 11:48:52.118142 ip-10-0-136-240 kubenswrapper[2585]: W0718 11:48:52.117086 2585 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 18 11:48:52.118142 ip-10-0-136-240 kubenswrapper[2585]: W0718 11:48:52.117089 2585 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 18 11:48:52.118142 ip-10-0-136-240 kubenswrapper[2585]: W0718 11:48:52.117091 2585 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 18 11:48:52.118142 ip-10-0-136-240 kubenswrapper[2585]: W0718 11:48:52.117094 2585 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 18 11:48:52.118142 ip-10-0-136-240 kubenswrapper[2585]: W0718 11:48:52.117096 2585 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 18 11:48:52.118142 ip-10-0-136-240 kubenswrapper[2585]: W0718 11:48:52.117099 2585 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 18 11:48:52.118142 ip-10-0-136-240 kubenswrapper[2585]: W0718 11:48:52.117103 2585 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 18 11:48:52.118142 ip-10-0-136-240 kubenswrapper[2585]: W0718 11:48:52.117106 2585 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 18 11:48:52.118142 ip-10-0-136-240 kubenswrapper[2585]: W0718 11:48:52.117110 2585 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 18 11:48:52.118142 ip-10-0-136-240 kubenswrapper[2585]: W0718 11:48:52.117113 2585 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 18 11:48:52.118142 ip-10-0-136-240 kubenswrapper[2585]: W0718 11:48:52.117116 2585 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 18 11:48:52.118142 ip-10-0-136-240 kubenswrapper[2585]: W0718 11:48:52.117119 2585 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 18 11:48:52.118142 ip-10-0-136-240 kubenswrapper[2585]: W0718 11:48:52.117121 2585 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 18 11:48:52.118142 ip-10-0-136-240 kubenswrapper[2585]: W0718 11:48:52.117124 2585 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 18 11:48:52.118142 ip-10-0-136-240 kubenswrapper[2585]: W0718 11:48:52.117127 2585 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 18 11:48:52.118142 ip-10-0-136-240 kubenswrapper[2585]: W0718 11:48:52.117129 2585 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 18 11:48:52.118142 ip-10-0-136-240 kubenswrapper[2585]: W0718 11:48:52.117132 2585 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 18 11:48:52.118142 ip-10-0-136-240 kubenswrapper[2585]: W0718 11:48:52.117135 2585 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 18 11:48:52.118142 ip-10-0-136-240 kubenswrapper[2585]: W0718 11:48:52.117137 2585 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 18 11:48:52.118655 ip-10-0-136-240 kubenswrapper[2585]: W0718 11:48:52.117140 2585 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 18 11:48:52.118655 ip-10-0-136-240 kubenswrapper[2585]: W0718 11:48:52.117142 2585 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 18 11:48:52.118655 ip-10-0-136-240 kubenswrapper[2585]: W0718 11:48:52.117145 2585 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 18 11:48:52.118655 ip-10-0-136-240 kubenswrapper[2585]: W0718 11:48:52.117147 2585 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 18 11:48:52.118655 ip-10-0-136-240 kubenswrapper[2585]: W0718 11:48:52.117150 2585 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 18 11:48:52.118655 ip-10-0-136-240 kubenswrapper[2585]: W0718 11:48:52.117153 2585 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 18 11:48:52.118655 ip-10-0-136-240 kubenswrapper[2585]: W0718 11:48:52.117155 2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 18 11:48:52.118655 ip-10-0-136-240 kubenswrapper[2585]: W0718 11:48:52.117158 2585 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 18 11:48:52.118655 ip-10-0-136-240 kubenswrapper[2585]: W0718 11:48:52.117161 2585 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 18 11:48:52.118655 ip-10-0-136-240 kubenswrapper[2585]: W0718 11:48:52.117164 2585 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 18 11:48:52.118655 ip-10-0-136-240 kubenswrapper[2585]: W0718 11:48:52.117167 2585 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 18 11:48:52.118655 ip-10-0-136-240 kubenswrapper[2585]: W0718 11:48:52.117169 2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 18 11:48:52.118655 ip-10-0-136-240 kubenswrapper[2585]: W0718 11:48:52.117173 2585 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 18 11:48:52.118655 ip-10-0-136-240 kubenswrapper[2585]: W0718 11:48:52.117176 2585 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 18 11:48:52.118655 ip-10-0-136-240 kubenswrapper[2585]: W0718 11:48:52.117178 2585 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 18 11:48:52.118655 ip-10-0-136-240 kubenswrapper[2585]: W0718 11:48:52.117180 2585 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 18 11:48:52.118655 ip-10-0-136-240 kubenswrapper[2585]: W0718 11:48:52.117183 2585 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 18 11:48:52.118655 ip-10-0-136-240 kubenswrapper[2585]: W0718 11:48:52.117186 2585 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 18 11:48:52.118655 ip-10-0-136-240 kubenswrapper[2585]: W0718 11:48:52.117188 2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 18 11:48:52.118655 ip-10-0-136-240 kubenswrapper[2585]: W0718 11:48:52.117191 2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 18 11:48:52.119197 ip-10-0-136-240 kubenswrapper[2585]: W0718 11:48:52.117193 2585 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 18 11:48:52.119197 ip-10-0-136-240 kubenswrapper[2585]: W0718 11:48:52.117196 2585 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 18 11:48:52.119197 ip-10-0-136-240 kubenswrapper[2585]: W0718 11:48:52.117198 2585 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 18 11:48:52.119197 ip-10-0-136-240 kubenswrapper[2585]: W0718 11:48:52.117200 2585 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 18 11:48:52.119197 ip-10-0-136-240 kubenswrapper[2585]: W0718 11:48:52.117203 2585 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 18 11:48:52.119197 ip-10-0-136-240 kubenswrapper[2585]: W0718 11:48:52.117205 2585 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 18 11:48:52.119197 ip-10-0-136-240 kubenswrapper[2585]: W0718 11:48:52.117208 2585 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 18 11:48:52.119197 ip-10-0-136-240 kubenswrapper[2585]: W0718 11:48:52.117210 2585 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 18 11:48:52.119197 ip-10-0-136-240 kubenswrapper[2585]: W0718 11:48:52.117212 2585 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 18 11:48:52.119197 ip-10-0-136-240 kubenswrapper[2585]: W0718 11:48:52.117215 2585 feature_gate.go:328] unrecognized feature gate: Example2 Jul 18 11:48:52.119197 ip-10-0-136-240 kubenswrapper[2585]: W0718 11:48:52.117217 2585 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 18 11:48:52.119197 ip-10-0-136-240 kubenswrapper[2585]: W0718 11:48:52.117220 2585 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 18 11:48:52.119197 ip-10-0-136-240 kubenswrapper[2585]: W0718 11:48:52.117222 2585 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 18 11:48:52.119197 ip-10-0-136-240 kubenswrapper[2585]: W0718 11:48:52.117225 2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 18 11:48:52.119197 ip-10-0-136-240 kubenswrapper[2585]: W0718 11:48:52.117228 2585 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 18 11:48:52.119197 ip-10-0-136-240 kubenswrapper[2585]: W0718 11:48:52.117230 2585 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 18 11:48:52.119197 ip-10-0-136-240 kubenswrapper[2585]: W0718 11:48:52.117232 2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 18 11:48:52.119197 ip-10-0-136-240 kubenswrapper[2585]: W0718 11:48:52.117235 2585 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 18 11:48:52.119197 ip-10-0-136-240 kubenswrapper[2585]: W0718 11:48:52.117928 2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 18 11:48:52.119197 ip-10-0-136-240 kubenswrapper[2585]: W0718 11:48:52.117934 2585 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 18 11:48:52.119769 ip-10-0-136-240 kubenswrapper[2585]: W0718 11:48:52.117938 2585 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 18 11:48:52.119769 ip-10-0-136-240 kubenswrapper[2585]: W0718 11:48:52.117942 2585 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 18 11:48:52.119769 ip-10-0-136-240 kubenswrapper[2585]: W0718 11:48:52.117944 2585 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 18 11:48:52.119769 ip-10-0-136-240 kubenswrapper[2585]: W0718 11:48:52.117947 2585 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 18 11:48:52.119769 ip-10-0-136-240 kubenswrapper[2585]: W0718 11:48:52.117950 2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 18 11:48:52.119769 ip-10-0-136-240 kubenswrapper[2585]: W0718 11:48:52.117953 2585 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 18 11:48:52.119769 ip-10-0-136-240 kubenswrapper[2585]: W0718 11:48:52.117957 2585 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 18 11:48:52.119769 ip-10-0-136-240 kubenswrapper[2585]: W0718 11:48:52.117959 2585 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 18 11:48:52.119769 ip-10-0-136-240 kubenswrapper[2585]: W0718 11:48:52.117962 2585 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 18 11:48:52.119769 ip-10-0-136-240 kubenswrapper[2585]: W0718 11:48:52.117966 2585 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 18 11:48:52.119769 ip-10-0-136-240 kubenswrapper[2585]: W0718 11:48:52.117970 2585 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 18 11:48:52.119769 ip-10-0-136-240 kubenswrapper[2585]: W0718 11:48:52.117973 2585 feature_gate.go:328] unrecognized feature gate: Example Jul 18 11:48:52.119769 ip-10-0-136-240 kubenswrapper[2585]: W0718 11:48:52.117976 2585 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 18 11:48:52.119769 ip-10-0-136-240 kubenswrapper[2585]: W0718 11:48:52.117979 2585 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 18 11:48:52.119769 ip-10-0-136-240 kubenswrapper[2585]: W0718 11:48:52.117982 2585 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 18 11:48:52.119769 ip-10-0-136-240 kubenswrapper[2585]: W0718 11:48:52.117984 2585 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 18 11:48:52.119769 ip-10-0-136-240 kubenswrapper[2585]: W0718 11:48:52.117987 2585 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 18 11:48:52.119769 ip-10-0-136-240 kubenswrapper[2585]: W0718 11:48:52.117991 2585 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 18 11:48:52.119769 ip-10-0-136-240 kubenswrapper[2585]: W0718 11:48:52.117994 2585 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 18 11:48:52.120279 ip-10-0-136-240 kubenswrapper[2585]: W0718 11:48:52.117997 2585 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 18 11:48:52.120279 ip-10-0-136-240 kubenswrapper[2585]: W0718 11:48:52.118000 2585 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 18 11:48:52.120279 ip-10-0-136-240 kubenswrapper[2585]: W0718 11:48:52.118003 2585 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 18 11:48:52.120279 ip-10-0-136-240 kubenswrapper[2585]: W0718 11:48:52.118005 2585 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 18 11:48:52.120279 ip-10-0-136-240 kubenswrapper[2585]: W0718 11:48:52.118007 2585 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 18 11:48:52.120279 ip-10-0-136-240 kubenswrapper[2585]: W0718 11:48:52.118010 2585 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 18 11:48:52.120279 ip-10-0-136-240 kubenswrapper[2585]: W0718 11:48:52.118012 2585 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 18 11:48:52.120279 ip-10-0-136-240 kubenswrapper[2585]: W0718 11:48:52.118015 2585 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 18 11:48:52.120279 ip-10-0-136-240 kubenswrapper[2585]: W0718 11:48:52.118019 2585 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 18 11:48:52.120279 ip-10-0-136-240 kubenswrapper[2585]: W0718 11:48:52.118021 2585 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 18 11:48:52.120279 ip-10-0-136-240 kubenswrapper[2585]: W0718 11:48:52.118024 2585 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 18 11:48:52.120279 ip-10-0-136-240 kubenswrapper[2585]: W0718 11:48:52.118026 2585 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 18 11:48:52.120279 ip-10-0-136-240 kubenswrapper[2585]: W0718 11:48:52.118028 2585 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 18 11:48:52.120279 ip-10-0-136-240 kubenswrapper[2585]: W0718 11:48:52.118032 2585 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 18 11:48:52.120279 ip-10-0-136-240 kubenswrapper[2585]: W0718 11:48:52.118035 2585 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 18 11:48:52.120279 ip-10-0-136-240 kubenswrapper[2585]: W0718 11:48:52.118039 2585 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 18 11:48:52.120279 ip-10-0-136-240 kubenswrapper[2585]: W0718 11:48:52.118042 2585 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 18 11:48:52.120279 ip-10-0-136-240 kubenswrapper[2585]: W0718 11:48:52.118045 2585 feature_gate.go:328] unrecognized feature gate: Example2 Jul 18 11:48:52.120279 ip-10-0-136-240 kubenswrapper[2585]: W0718 11:48:52.118048 2585 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 18 11:48:52.120785 ip-10-0-136-240 kubenswrapper[2585]: W0718 11:48:52.118050 2585 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 18 11:48:52.120785 ip-10-0-136-240 kubenswrapper[2585]: W0718 11:48:52.118053 2585 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 18 11:48:52.120785 ip-10-0-136-240 kubenswrapper[2585]: W0718 11:48:52.118055 2585 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 18 11:48:52.120785 ip-10-0-136-240 kubenswrapper[2585]: W0718 11:48:52.118058 2585 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 18 11:48:52.120785 ip-10-0-136-240 kubenswrapper[2585]: W0718 11:48:52.118060 2585 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 18 11:48:52.120785 ip-10-0-136-240 kubenswrapper[2585]: W0718 11:48:52.118063 2585 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 18 11:48:52.120785 ip-10-0-136-240 kubenswrapper[2585]: W0718 11:48:52.118065 2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 18 11:48:52.120785 ip-10-0-136-240 kubenswrapper[2585]: W0718 11:48:52.118067 2585 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 18 11:48:52.120785 ip-10-0-136-240 kubenswrapper[2585]: W0718 11:48:52.118070 2585 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 18 11:48:52.120785 ip-10-0-136-240 kubenswrapper[2585]: W0718 11:48:52.118072 2585 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 18 11:48:52.120785 ip-10-0-136-240 kubenswrapper[2585]: W0718 11:48:52.118075 2585 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 18 11:48:52.120785 ip-10-0-136-240 kubenswrapper[2585]: W0718 11:48:52.118077 2585 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 18 11:48:52.120785 ip-10-0-136-240 kubenswrapper[2585]: W0718 11:48:52.118080 2585 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 18 11:48:52.120785 ip-10-0-136-240 kubenswrapper[2585]: W0718 11:48:52.118082 2585 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 18 11:48:52.120785 ip-10-0-136-240 kubenswrapper[2585]: W0718 11:48:52.118085 2585 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 18 11:48:52.120785 ip-10-0-136-240 kubenswrapper[2585]: W0718 11:48:52.118088 2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 18 11:48:52.120785 ip-10-0-136-240 kubenswrapper[2585]: W0718 11:48:52.118090 2585 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 18 11:48:52.120785 ip-10-0-136-240 kubenswrapper[2585]: W0718 11:48:52.118093 2585 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 18 11:48:52.120785 ip-10-0-136-240 kubenswrapper[2585]: W0718 11:48:52.118095 2585 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 18 11:48:52.120785 ip-10-0-136-240 kubenswrapper[2585]: W0718 11:48:52.118098 2585 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 18 11:48:52.121303 ip-10-0-136-240 kubenswrapper[2585]: W0718 11:48:52.118100 2585 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 18 11:48:52.121303 ip-10-0-136-240 kubenswrapper[2585]: W0718 11:48:52.118103 2585 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 18 11:48:52.121303 ip-10-0-136-240 kubenswrapper[2585]: W0718 11:48:52.118105 2585 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 18 11:48:52.121303 ip-10-0-136-240 kubenswrapper[2585]: W0718 11:48:52.118107 2585 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 18 11:48:52.121303 ip-10-0-136-240 kubenswrapper[2585]: W0718 11:48:52.118111 2585 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 18 11:48:52.121303 ip-10-0-136-240 kubenswrapper[2585]: W0718 11:48:52.118113 2585 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 18 11:48:52.121303 ip-10-0-136-240 kubenswrapper[2585]: W0718 11:48:52.118116 2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 18 11:48:52.121303 ip-10-0-136-240 kubenswrapper[2585]: W0718 11:48:52.118119 2585 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 18 11:48:52.121303 ip-10-0-136-240 kubenswrapper[2585]: W0718 11:48:52.118121 2585 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 18 11:48:52.121303 ip-10-0-136-240 kubenswrapper[2585]: W0718 11:48:52.118124 2585 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 18 11:48:52.121303 ip-10-0-136-240 kubenswrapper[2585]: W0718 11:48:52.118126 2585 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 18 11:48:52.121303 ip-10-0-136-240 kubenswrapper[2585]: W0718 11:48:52.118130 2585 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 18 11:48:52.121303 ip-10-0-136-240 kubenswrapper[2585]: W0718 11:48:52.118132 2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 18 11:48:52.121303 ip-10-0-136-240 kubenswrapper[2585]: W0718 11:48:52.118134 2585 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 18 11:48:52.121303 ip-10-0-136-240 kubenswrapper[2585]: W0718 11:48:52.118137 2585 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 18 11:48:52.121303 ip-10-0-136-240 kubenswrapper[2585]: W0718 11:48:52.118139 2585 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 18 11:48:52.121303 ip-10-0-136-240 kubenswrapper[2585]: W0718 11:48:52.118142 2585 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 18 11:48:52.121303 ip-10-0-136-240 kubenswrapper[2585]: W0718 11:48:52.118144 2585 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 18 11:48:52.121303 ip-10-0-136-240 kubenswrapper[2585]: W0718 11:48:52.118147 2585 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 18 11:48:52.121303 ip-10-0-136-240 kubenswrapper[2585]: W0718 11:48:52.118149 2585 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 18 11:48:52.121870 ip-10-0-136-240 kubenswrapper[2585]: W0718 11:48:52.118152 2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 18 11:48:52.121870 ip-10-0-136-240 kubenswrapper[2585]: W0718 11:48:52.118154 2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 18 11:48:52.121870 ip-10-0-136-240 kubenswrapper[2585]: W0718 11:48:52.118157 2585 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 18 11:48:52.121870 ip-10-0-136-240 kubenswrapper[2585]: W0718 11:48:52.118159 2585 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 18 11:48:52.121870 ip-10-0-136-240 kubenswrapper[2585]: W0718 11:48:52.118162 2585 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 18 11:48:52.121870 ip-10-0-136-240 kubenswrapper[2585]: W0718 11:48:52.118164 2585 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 18 11:48:52.121870 ip-10-0-136-240 kubenswrapper[2585]: W0718 11:48:52.118167 2585 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 18 11:48:52.121870 ip-10-0-136-240 kubenswrapper[2585]: W0718 11:48:52.118169 2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 18 11:48:52.121870 ip-10-0-136-240 kubenswrapper[2585]: W0718 11:48:52.118171 2585 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 18 11:48:52.121870 ip-10-0-136-240 kubenswrapper[2585]: W0718 11:48:52.118174 2585 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 18 11:48:52.121870 ip-10-0-136-240 kubenswrapper[2585]: W0718 11:48:52.118176 2585 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 18 11:48:52.121870 ip-10-0-136-240 kubenswrapper[2585]: W0718 11:48:52.118179 2585 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 18 11:48:52.121870 ip-10-0-136-240 kubenswrapper[2585]: W0718 11:48:52.118181 2585 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 18 11:48:52.121870 ip-10-0-136-240 kubenswrapper[2585]: W0718 11:48:52.118184 2585 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 18 11:48:52.121870 ip-10-0-136-240 kubenswrapper[2585]: W0718 11:48:52.118186 2585 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 18 11:48:52.121870 ip-10-0-136-240 kubenswrapper[2585]: W0718 11:48:52.118189 2585 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 18 11:48:52.121870 ip-10-0-136-240 kubenswrapper[2585]: W0718 11:48:52.118191 2585 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 18 11:48:52.121870 ip-10-0-136-240 kubenswrapper[2585]: W0718 11:48:52.118194 2585 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 18 11:48:52.121870 ip-10-0-136-240 kubenswrapper[2585]: W0718 11:48:52.118197 2585 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 18 11:48:52.121870 ip-10-0-136-240 kubenswrapper[2585]: W0718 11:48:52.118603 2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 18 11:48:52.122385 ip-10-0-136-240 kubenswrapper[2585]: W0718 11:48:52.118608 2585 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 18 11:48:52.122385 ip-10-0-136-240 kubenswrapper[2585]: W0718 11:48:52.118611 2585 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 18 11:48:52.122385 ip-10-0-136-240 kubenswrapper[2585]: W0718 11:48:52.118615 2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 18 11:48:52.122385 ip-10-0-136-240 kubenswrapper[2585]: W0718 11:48:52.118618 2585 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 18 11:48:52.122385 ip-10-0-136-240 kubenswrapper[2585]: W0718 11:48:52.118621 2585 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 18 11:48:52.122385 ip-10-0-136-240 kubenswrapper[2585]: W0718 11:48:52.118624 2585 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 18 11:48:52.122385 ip-10-0-136-240 kubenswrapper[2585]: W0718 11:48:52.118627 2585 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 18 11:48:52.122385 ip-10-0-136-240 kubenswrapper[2585]: W0718 11:48:52.118629 2585 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 18 11:48:52.122385 ip-10-0-136-240 kubenswrapper[2585]: W0718 11:48:52.118632 2585 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 18 11:48:52.122385 ip-10-0-136-240 kubenswrapper[2585]: W0718 11:48:52.118634 2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 18 11:48:52.122385 ip-10-0-136-240 kubenswrapper[2585]: W0718 11:48:52.118637 2585 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 18 11:48:52.122385 ip-10-0-136-240 kubenswrapper[2585]: W0718 11:48:52.118639 2585 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 18 11:48:52.122385 ip-10-0-136-240 kubenswrapper[2585]: W0718 11:48:52.118642 2585 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 18 11:48:52.122385 ip-10-0-136-240 kubenswrapper[2585]: W0718 11:48:52.118644 2585 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 18 11:48:52.122385 ip-10-0-136-240 kubenswrapper[2585]: W0718 11:48:52.118647 2585 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 18 11:48:52.122385 ip-10-0-136-240 kubenswrapper[2585]: W0718 11:48:52.118649 2585 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 18 11:48:52.122385 ip-10-0-136-240 kubenswrapper[2585]: W0718 11:48:52.118651 2585 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 18 11:48:52.122385 ip-10-0-136-240 kubenswrapper[2585]: W0718 11:48:52.118654 2585 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 18 11:48:52.122385 ip-10-0-136-240 kubenswrapper[2585]: W0718 11:48:52.118658 2585 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 18 11:48:52.122909 ip-10-0-136-240 kubenswrapper[2585]: W0718 11:48:52.118662 2585 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 18 11:48:52.122909 ip-10-0-136-240 kubenswrapper[2585]: W0718 11:48:52.118665 2585 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 18 11:48:52.122909 ip-10-0-136-240 kubenswrapper[2585]: W0718 11:48:52.118668 2585 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 18 11:48:52.122909 ip-10-0-136-240 kubenswrapper[2585]: W0718 11:48:52.118671 2585 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 18 11:48:52.122909 ip-10-0-136-240 kubenswrapper[2585]: W0718 11:48:52.118674 2585 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 18 11:48:52.122909 ip-10-0-136-240 kubenswrapper[2585]: W0718 11:48:52.118676 2585 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 18 11:48:52.122909 ip-10-0-136-240 kubenswrapper[2585]: W0718 11:48:52.118679 2585 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 18 11:48:52.122909 ip-10-0-136-240 kubenswrapper[2585]: W0718 11:48:52.118681 2585 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 18 11:48:52.122909 ip-10-0-136-240 kubenswrapper[2585]: W0718 11:48:52.118684 2585 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 18 11:48:52.122909 ip-10-0-136-240 kubenswrapper[2585]: W0718 11:48:52.118686 2585 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 18 11:48:52.122909 ip-10-0-136-240 kubenswrapper[2585]: W0718 11:48:52.118689 2585 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 18 11:48:52.122909 ip-10-0-136-240 kubenswrapper[2585]: W0718 11:48:52.118693 2585 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 18 11:48:52.122909 ip-10-0-136-240 kubenswrapper[2585]: W0718 11:48:52.118695 2585 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 18 11:48:52.122909 ip-10-0-136-240 kubenswrapper[2585]: W0718 11:48:52.118698 2585 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 18 11:48:52.122909 ip-10-0-136-240 kubenswrapper[2585]: W0718 11:48:52.118700 2585 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 18 11:48:52.122909 ip-10-0-136-240 kubenswrapper[2585]: W0718 11:48:52.118703 2585 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 18 11:48:52.122909 ip-10-0-136-240 kubenswrapper[2585]: W0718 11:48:52.118706 2585 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 18 11:48:52.122909 ip-10-0-136-240 kubenswrapper[2585]: W0718 11:48:52.118708 2585 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 18 11:48:52.122909 ip-10-0-136-240 kubenswrapper[2585]: W0718 11:48:52.118711 2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 18 11:48:52.122909 ip-10-0-136-240 kubenswrapper[2585]: W0718 11:48:52.118713 2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 18 11:48:52.123468 ip-10-0-136-240 kubenswrapper[2585]: W0718 11:48:52.118716 2585 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 18 11:48:52.123468 ip-10-0-136-240 kubenswrapper[2585]: W0718 11:48:52.118718 2585 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 18 11:48:52.123468 ip-10-0-136-240 kubenswrapper[2585]: W0718 11:48:52.118721 2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 18 11:48:52.123468 ip-10-0-136-240 kubenswrapper[2585]: W0718 11:48:52.118723 2585 feature_gate.go:328] unrecognized feature gate: Example Jul 18 11:48:52.123468 ip-10-0-136-240 kubenswrapper[2585]: W0718 11:48:52.118726 2585 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 18 11:48:52.123468 ip-10-0-136-240 kubenswrapper[2585]: W0718 11:48:52.118728 2585 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 18 11:48:52.123468 ip-10-0-136-240 kubenswrapper[2585]: W0718 11:48:52.118731 2585 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 18 11:48:52.123468 ip-10-0-136-240 kubenswrapper[2585]: W0718 11:48:52.118733 2585 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 18 11:48:52.123468 ip-10-0-136-240 kubenswrapper[2585]: W0718 11:48:52.118736 2585 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 18 11:48:52.123468 ip-10-0-136-240 kubenswrapper[2585]: W0718 11:48:52.118738 2585 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 18 11:48:52.123468 ip-10-0-136-240 kubenswrapper[2585]: W0718 11:48:52.118740 2585 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 18 11:48:52.123468 ip-10-0-136-240 kubenswrapper[2585]: W0718 11:48:52.118744 2585 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 18 11:48:52.123468 ip-10-0-136-240 kubenswrapper[2585]: W0718 11:48:52.118746 2585 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 18 11:48:52.123468 ip-10-0-136-240 kubenswrapper[2585]: W0718 11:48:52.118749 2585 feature_gate.go:328] unrecognized feature gate: Example2 Jul 18 11:48:52.123468 ip-10-0-136-240 kubenswrapper[2585]: W0718 11:48:52.118751 2585 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 18 11:48:52.123468 ip-10-0-136-240 kubenswrapper[2585]: W0718 11:48:52.118754 2585 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 18 11:48:52.123468 ip-10-0-136-240 kubenswrapper[2585]: W0718 11:48:52.118756 2585 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 18 11:48:52.123468 ip-10-0-136-240 kubenswrapper[2585]: W0718 11:48:52.118759 2585 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 18 11:48:52.123468 ip-10-0-136-240 kubenswrapper[2585]: W0718 11:48:52.118761 2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 18 11:48:52.123468 ip-10-0-136-240 kubenswrapper[2585]: W0718 11:48:52.118763 2585 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 18 11:48:52.124012 ip-10-0-136-240 kubenswrapper[2585]: W0718 11:48:52.118766 2585 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 18 11:48:52.124012 ip-10-0-136-240 kubenswrapper[2585]: W0718 11:48:52.118768 2585 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 18 11:48:52.124012 ip-10-0-136-240 kubenswrapper[2585]: W0718 11:48:52.118771 2585 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 18 11:48:52.124012 ip-10-0-136-240 kubenswrapper[2585]: W0718 11:48:52.118774 2585 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 18 11:48:52.124012 ip-10-0-136-240 kubenswrapper[2585]: W0718 11:48:52.118777 2585 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 18 11:48:52.124012 ip-10-0-136-240 kubenswrapper[2585]: W0718 11:48:52.118779 2585 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 18 11:48:52.124012 ip-10-0-136-240 kubenswrapper[2585]: W0718 11:48:52.118782 2585 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 18 11:48:52.124012 ip-10-0-136-240 kubenswrapper[2585]: W0718 11:48:52.118785 2585 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 18 11:48:52.124012 ip-10-0-136-240 kubenswrapper[2585]: W0718 11:48:52.118789 2585 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 18 11:48:52.124012 ip-10-0-136-240 kubenswrapper[2585]: W0718 11:48:52.118792 2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 18 11:48:52.124012 ip-10-0-136-240 kubenswrapper[2585]: W0718 11:48:52.118795 2585 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 18 11:48:52.124012 ip-10-0-136-240 kubenswrapper[2585]: W0718 11:48:52.118798 2585 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 18 11:48:52.124012 ip-10-0-136-240 kubenswrapper[2585]: W0718 11:48:52.118801 2585 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 18 11:48:52.124012 ip-10-0-136-240 kubenswrapper[2585]: W0718 11:48:52.118803 2585 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 18 11:48:52.124012 ip-10-0-136-240 kubenswrapper[2585]: W0718 11:48:52.118806 2585 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 18 11:48:52.124012 ip-10-0-136-240 kubenswrapper[2585]: W0718 11:48:52.118808 2585 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 18 11:48:52.124012 ip-10-0-136-240 kubenswrapper[2585]: W0718 11:48:52.118810 2585 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 18 11:48:52.124012 ip-10-0-136-240 kubenswrapper[2585]: W0718 11:48:52.118813 2585 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 18 11:48:52.124012 ip-10-0-136-240 kubenswrapper[2585]: W0718 11:48:52.118815 2585 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 18 11:48:52.124012 ip-10-0-136-240 kubenswrapper[2585]: W0718 11:48:52.118818 2585 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 18 11:48:52.124544 ip-10-0-136-240 kubenswrapper[2585]: W0718 11:48:52.118821 2585 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 18 11:48:52.124544 ip-10-0-136-240 kubenswrapper[2585]: W0718 11:48:52.118825 2585 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 18 11:48:52.124544 ip-10-0-136-240 kubenswrapper[2585]: W0718 11:48:52.118828 2585 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 18 11:48:52.124544 ip-10-0-136-240 kubenswrapper[2585]: W0718 11:48:52.118831 2585 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 18 11:48:52.124544 ip-10-0-136-240 kubenswrapper[2585]: W0718 11:48:52.118835 2585 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 18 11:48:52.124544 ip-10-0-136-240 kubenswrapper[2585]: W0718 11:48:52.118838 2585 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 18 11:48:52.124544 ip-10-0-136-240 kubenswrapper[2585]: W0718 11:48:52.118841 2585 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 18 11:48:52.124544 ip-10-0-136-240 kubenswrapper[2585]: W0718 11:48:52.118844 2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 18 11:48:52.124544 ip-10-0-136-240 kubenswrapper[2585]: W0718 11:48:52.118846 2585 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 18 11:48:52.124544 ip-10-0-136-240 kubenswrapper[2585]: W0718 11:48:52.118849 2585 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 18 11:48:52.124544 ip-10-0-136-240 kubenswrapper[2585]: W0718 11:48:52.118851 2585 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 18 11:48:52.124544 ip-10-0-136-240 kubenswrapper[2585]: W0718 11:48:52.118853 2585 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 18 11:48:52.124544 ip-10-0-136-240 kubenswrapper[2585]: W0718 11:48:52.118856 2585 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 18 11:48:52.124544 ip-10-0-136-240 kubenswrapper[2585]: W0718 11:48:52.118859 2585 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 18 11:48:52.124544 ip-10-0-136-240 kubenswrapper[2585]: W0718 11:48:52.118862 2585 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 18 11:48:52.124544 ip-10-0-136-240 kubenswrapper[2585]: W0718 11:48:52.118864 2585 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 18 11:48:52.124544 ip-10-0-136-240 kubenswrapper[2585]: W0718 11:48:52.118867 2585 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 18 11:48:52.124544 ip-10-0-136-240 kubenswrapper[2585]: W0718 11:48:52.118869 2585 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 18 11:48:52.124544 ip-10-0-136-240 kubenswrapper[2585]: W0718 11:48:52.118872 2585 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 18 11:48:52.124544 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:52.118934 2585 flags.go:64] FLAG: --address="0.0.0.0" Jul 18 11:48:52.125096 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:52.118941 2585 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Jul 18 11:48:52.125096 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:52.118948 2585 flags.go:64] FLAG: --anonymous-auth="true" Jul 18 11:48:52.125096 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:52.118952 2585 flags.go:64] FLAG: --application-metrics-count-limit="100" Jul 18 11:48:52.125096 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:52.118956 2585 flags.go:64] FLAG: --authentication-token-webhook="false" Jul 18 11:48:52.125096 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:52.118959 2585 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Jul 18 11:48:52.125096 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:52.118963 2585 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Jul 18 11:48:52.125096 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:52.118967 2585 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Jul 18 11:48:52.125096 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:52.118970 2585 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Jul 18 11:48:52.125096 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:52.118973 2585 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Jul 18 11:48:52.125096 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:52.118976 2585 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Jul 18 11:48:52.125096 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:52.118979 2585 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Jul 18 11:48:52.125096 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:52.118982 2585 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Jul 18 11:48:52.125096 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:52.118985 2585 flags.go:64] FLAG: --cgroup-root="" Jul 18 11:48:52.125096 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:52.118988 2585 flags.go:64] FLAG: --cgroups-per-qos="true" Jul 18 11:48:52.125096 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:52.118991 2585 flags.go:64] FLAG: --client-ca-file="" Jul 18 11:48:52.125096 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:52.118994 2585 flags.go:64] FLAG: --cloud-provider="external" Jul 18 11:48:52.125096 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:52.118997 2585 flags.go:64] FLAG: --cluster-dns="[]" Jul 18 11:48:52.125096 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:52.119000 2585 flags.go:64] FLAG: --cluster-domain="" Jul 18 11:48:52.125096 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:52.119003 2585 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Jul 18 11:48:52.125096 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:52.119006 2585 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Jul 18 11:48:52.125096 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:52.119009 2585 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Jul 18 11:48:52.125096 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:52.119012 2585 flags.go:64] FLAG: --container-log-max-files="5" Jul 18 11:48:52.125096 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:52.119015 2585 flags.go:64] FLAG: --container-log-max-size="10Mi" Jul 18 11:48:52.125096 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:52.119018 2585 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Jul 18 11:48:52.125735 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:52.119021 2585 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Jul 18 11:48:52.125735 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:52.119024 2585 flags.go:64] FLAG: --containerd-namespace="k8s.io" Jul 18 11:48:52.125735 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:52.119027 2585 flags.go:64] FLAG: --contention-profiling="false" Jul 18 11:48:52.125735 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:52.119030 2585 flags.go:64] FLAG: --cpu-cfs-quota="true" Jul 18 11:48:52.125735 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:52.119032 2585 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Jul 18 11:48:52.125735 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:52.119035 2585 flags.go:64] FLAG: --cpu-manager-policy="none" Jul 18 11:48:52.125735 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:52.119038 2585 flags.go:64] FLAG: --cpu-manager-policy-options="" Jul 18 11:48:52.125735 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:52.119042 2585 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Jul 18 11:48:52.125735 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:52.119044 2585 flags.go:64] FLAG: --enable-controller-attach-detach="true" Jul 18 11:48:52.125735 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:52.119047 2585 flags.go:64] FLAG: --enable-debugging-handlers="true" Jul 18 11:48:52.125735 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:52.119050 2585 flags.go:64] FLAG: --enable-load-reader="false" Jul 18 11:48:52.125735 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:52.119052 2585 flags.go:64] FLAG: --enable-server="true" Jul 18 11:48:52.125735 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:52.119056 2585 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Jul 18 11:48:52.125735 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:52.119060 2585 flags.go:64] FLAG: --event-burst="100" Jul 18 11:48:52.125735 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:52.119063 2585 flags.go:64] FLAG: --event-qps="50" Jul 18 11:48:52.125735 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:52.119065 2585 flags.go:64] FLAG: --event-storage-age-limit="default=0" Jul 18 11:48:52.125735 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:52.119068 2585 flags.go:64] FLAG: --event-storage-event-limit="default=0" Jul 18 11:48:52.125735 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:52.119071 2585 flags.go:64] FLAG: --eviction-hard="" Jul 18 11:48:52.125735 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:52.119074 2585 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Jul 18 11:48:52.125735 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:52.119077 2585 flags.go:64] FLAG: --eviction-minimum-reclaim="" Jul 18 11:48:52.125735 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:52.119080 2585 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Jul 18 11:48:52.125735 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:52.119082 2585 flags.go:64] FLAG: --eviction-soft="" Jul 18 11:48:52.125735 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:52.119085 2585 flags.go:64] FLAG: --eviction-soft-grace-period="" Jul 18 11:48:52.125735 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:52.119088 2585 flags.go:64] FLAG: --exit-on-lock-contention="false" Jul 18 11:48:52.125735 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:52.119090 2585 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Jul 18 11:48:52.126396 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:52.119093 2585 flags.go:64] FLAG: --experimental-mounter-path="" Jul 18 11:48:52.126396 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:52.119097 2585 flags.go:64] FLAG: --fail-cgroupv1="false" Jul 18 11:48:52.126396 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:52.119100 2585 flags.go:64] FLAG: --fail-swap-on="true" Jul 18 11:48:52.126396 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:52.119102 2585 flags.go:64] FLAG: --feature-gates="" Jul 18 11:48:52.126396 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:52.119106 2585 flags.go:64] FLAG: --file-check-frequency="20s" Jul 18 11:48:52.126396 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:52.119109 2585 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Jul 18 11:48:52.126396 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:52.119112 2585 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Jul 18 11:48:52.126396 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:52.119115 2585 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Jul 18 11:48:52.126396 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:52.119118 2585 flags.go:64] FLAG: --healthz-port="10248" Jul 18 11:48:52.126396 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:52.119120 2585 flags.go:64] FLAG: --help="false" Jul 18 11:48:52.126396 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:52.119123 2585 flags.go:64] FLAG: --hostname-override="ip-10-0-136-240.ec2.internal" Jul 18 11:48:52.126396 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:52.119126 2585 flags.go:64] FLAG: --housekeeping-interval="10s" Jul 18 11:48:52.126396 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:52.119129 2585 flags.go:64] FLAG: --http-check-frequency="20s" Jul 18 11:48:52.126396 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:52.119131 2585 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Jul 18 11:48:52.126396 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:52.119135 2585 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Jul 18 11:48:52.126396 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:52.119138 2585 flags.go:64] FLAG: --image-gc-high-threshold="85" Jul 18 11:48:52.126396 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:52.119140 2585 flags.go:64] FLAG: --image-gc-low-threshold="80" Jul 18 11:48:52.126396 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:52.119143 2585 flags.go:64] FLAG: --image-service-endpoint="" Jul 18 11:48:52.126396 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:52.119146 2585 flags.go:64] FLAG: --kernel-memcg-notification="false" Jul 18 11:48:52.126396 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:52.119148 2585 flags.go:64] FLAG: --kube-api-burst="100" Jul 18 11:48:52.126396 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:52.119151 2585 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Jul 18 11:48:52.126396 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:52.119154 2585 flags.go:64] FLAG: --kube-api-qps="50" Jul 18 11:48:52.126396 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:52.119157 2585 flags.go:64] FLAG: --kube-reserved="" Jul 18 11:48:52.126396 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:52.119160 2585 flags.go:64] FLAG: --kube-reserved-cgroup="" Jul 18 11:48:52.127041 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:52.119162 2585 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Jul 18 11:48:52.127041 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:52.119165 2585 flags.go:64] FLAG: --kubelet-cgroups="" Jul 18 11:48:52.127041 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:52.119168 2585 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Jul 18 11:48:52.127041 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:52.119171 2585 flags.go:64] FLAG: --lock-file="" Jul 18 11:48:52.127041 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:52.119174 2585 flags.go:64] FLAG: --log-cadvisor-usage="false" Jul 18 11:48:52.127041 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:52.119177 2585 flags.go:64] FLAG: --log-flush-frequency="5s" Jul 18 11:48:52.127041 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:52.119179 2585 flags.go:64] FLAG: --log-json-info-buffer-size="0" Jul 18 11:48:52.127041 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:52.119185 2585 flags.go:64] FLAG: --log-json-split-stream="false" Jul 18 11:48:52.127041 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:52.119188 2585 flags.go:64] FLAG: --log-text-info-buffer-size="0" Jul 18 11:48:52.127041 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:52.119190 2585 flags.go:64] FLAG: --log-text-split-stream="false" Jul 18 11:48:52.127041 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:52.119193 2585 flags.go:64] FLAG: --logging-format="text" Jul 18 11:48:52.127041 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:52.119196 2585 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Jul 18 11:48:52.127041 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:52.119199 2585 flags.go:64] FLAG: --make-iptables-util-chains="true" Jul 18 11:48:52.127041 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:52.119202 2585 flags.go:64] FLAG: --manifest-url="" Jul 18 11:48:52.127041 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:52.119205 2585 flags.go:64] FLAG: --manifest-url-header="" Jul 18 11:48:52.127041 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:52.119209 2585 flags.go:64] FLAG: --max-housekeeping-interval="15s" Jul 18 11:48:52.127041 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:52.119212 2585 flags.go:64] FLAG: --max-open-files="1000000" Jul 18 11:48:52.127041 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:52.119216 2585 flags.go:64] FLAG: --max-pods="110" Jul 18 11:48:52.127041 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:52.119221 2585 flags.go:64] FLAG: --maximum-dead-containers="-1" Jul 18 11:48:52.127041 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:52.119224 2585 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Jul 18 11:48:52.127041 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:52.119227 2585 flags.go:64] FLAG: --memory-manager-policy="None" Jul 18 11:48:52.127041 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:52.119229 2585 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Jul 18 11:48:52.127041 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:52.119232 2585 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Jul 18 11:48:52.127041 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:52.119235 2585 flags.go:64] FLAG: --node-ip="0.0.0.0" Jul 18 11:48:52.127041 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:52.119238 2585 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Jul 18 11:48:52.127726 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:52.119245 2585 flags.go:64] FLAG: --node-status-max-images="50" Jul 18 11:48:52.127726 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:52.119248 2585 flags.go:64] FLAG: --node-status-update-frequency="10s" Jul 18 11:48:52.127726 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:52.119250 2585 flags.go:64] FLAG: --oom-score-adj="-999" Jul 18 11:48:52.127726 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:52.119253 2585 flags.go:64] FLAG: --pod-cidr="" Jul 18 11:48:52.127726 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:52.119257 2585 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:4017e99d76d7b91598d3a3430cd0eba6eda8b33653addd3838fdfe3cca69c5a0" Jul 18 11:48:52.127726 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:52.119262 2585 flags.go:64] FLAG: --pod-manifest-path="" Jul 18 11:48:52.127726 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:52.119264 2585 flags.go:64] FLAG: --pod-max-pids="-1" Jul 18 11:48:52.127726 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:52.119267 2585 flags.go:64] FLAG: --pods-per-core="0" Jul 18 11:48:52.127726 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:52.119270 2585 flags.go:64] FLAG: --port="10250" Jul 18 11:48:52.127726 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:52.119273 2585 flags.go:64] FLAG: --protect-kernel-defaults="false" Jul 18 11:48:52.127726 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:52.119276 2585 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0fc9c954028d13366" Jul 18 11:48:52.127726 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:52.119280 2585 flags.go:64] FLAG: --qos-reserved="" Jul 18 11:48:52.127726 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:52.119283 2585 flags.go:64] FLAG: --read-only-port="10255" Jul 18 11:48:52.127726 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:52.119286 2585 flags.go:64] FLAG: --register-node="true" Jul 18 11:48:52.127726 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:52.119289 2585 flags.go:64] FLAG: --register-with-taints="" Jul 18 11:48:52.127726 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:52.119292 2585 flags.go:64] FLAG: --registry-burst="10" Jul 18 11:48:52.127726 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:52.119295 2585 flags.go:64] FLAG: --registry-qps="5" Jul 18 11:48:52.127726 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:52.119297 2585 flags.go:64] FLAG: --reserved-cpus="" Jul 18 11:48:52.127726 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:52.119300 2585 flags.go:64] FLAG: --reserved-memory="" Jul 18 11:48:52.127726 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:52.119304 2585 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Jul 18 11:48:52.127726 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:52.119306 2585 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Jul 18 11:48:52.127726 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:52.119310 2585 flags.go:64] FLAG: --rotate-certificates="false" Jul 18 11:48:52.127726 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:52.119312 2585 flags.go:64] FLAG: --rotate-server-certificates="false" Jul 18 11:48:52.127726 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:52.119315 2585 flags.go:64] FLAG: --runonce="false" Jul 18 11:48:52.127726 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:52.119318 2585 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Jul 18 11:48:52.128374 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:52.119323 2585 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Jul 18 11:48:52.128374 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:52.119326 2585 flags.go:64] FLAG: --seccomp-default="false" Jul 18 11:48:52.128374 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:52.119328 2585 flags.go:64] FLAG: --serialize-image-pulls="true" Jul 18 11:48:52.128374 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:52.119331 2585 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Jul 18 11:48:52.128374 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:52.119334 2585 flags.go:64] FLAG: --storage-driver-db="cadvisor" Jul 18 11:48:52.128374 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:52.119337 2585 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Jul 18 11:48:52.128374 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:52.119340 2585 flags.go:64] FLAG: --storage-driver-password="root" Jul 18 11:48:52.128374 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:52.119346 2585 flags.go:64] FLAG: --storage-driver-secure="false" Jul 18 11:48:52.128374 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:52.119349 2585 flags.go:64] FLAG: --storage-driver-table="stats" Jul 18 11:48:52.128374 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:52.119351 2585 flags.go:64] FLAG: --storage-driver-user="root" Jul 18 11:48:52.128374 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:52.119354 2585 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Jul 18 11:48:52.128374 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:52.119357 2585 flags.go:64] FLAG: --sync-frequency="1m0s" Jul 18 11:48:52.128374 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:52.119360 2585 flags.go:64] FLAG: --system-cgroups="" Jul 18 11:48:52.128374 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:52.119363 2585 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Jul 18 11:48:52.128374 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:52.119368 2585 flags.go:64] FLAG: --system-reserved-cgroup="" Jul 18 11:48:52.128374 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:52.119371 2585 flags.go:64] FLAG: --tls-cert-file="" Jul 18 11:48:52.128374 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:52.119373 2585 flags.go:64] FLAG: --tls-cipher-suites="[]" Jul 18 11:48:52.128374 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:52.119377 2585 flags.go:64] FLAG: --tls-min-version="" Jul 18 11:48:52.128374 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:52.119381 2585 flags.go:64] FLAG: --tls-private-key-file="" Jul 18 11:48:52.128374 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:52.119384 2585 flags.go:64] FLAG: --topology-manager-policy="none" Jul 18 11:48:52.128374 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:52.119387 2585 flags.go:64] FLAG: --topology-manager-policy-options="" Jul 18 11:48:52.128374 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:52.119389 2585 flags.go:64] FLAG: --topology-manager-scope="container" Jul 18 11:48:52.128374 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:52.119392 2585 flags.go:64] FLAG: --v="2" Jul 18 11:48:52.128374 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:52.119396 2585 flags.go:64] FLAG: --version="false" Jul 18 11:48:52.128374 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:52.119400 2585 flags.go:64] FLAG: --vmodule="" Jul 18 11:48:52.129080 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:52.119408 2585 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Jul 18 11:48:52.129080 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:52.119411 2585 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Jul 18 11:48:52.129080 ip-10-0-136-240 kubenswrapper[2585]: W0718 11:48:52.119547 2585 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 18 11:48:52.129080 ip-10-0-136-240 kubenswrapper[2585]: W0718 11:48:52.119552 2585 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 18 11:48:52.129080 ip-10-0-136-240 kubenswrapper[2585]: W0718 11:48:52.119555 2585 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 18 11:48:52.129080 ip-10-0-136-240 kubenswrapper[2585]: W0718 11:48:52.119557 2585 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 18 11:48:52.129080 ip-10-0-136-240 kubenswrapper[2585]: W0718 11:48:52.119560 2585 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 18 11:48:52.129080 ip-10-0-136-240 kubenswrapper[2585]: W0718 11:48:52.119564 2585 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 18 11:48:52.129080 ip-10-0-136-240 kubenswrapper[2585]: W0718 11:48:52.119566 2585 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 18 11:48:52.129080 ip-10-0-136-240 kubenswrapper[2585]: W0718 11:48:52.119569 2585 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 18 11:48:52.129080 ip-10-0-136-240 kubenswrapper[2585]: W0718 11:48:52.119572 2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 18 11:48:52.129080 ip-10-0-136-240 kubenswrapper[2585]: W0718 11:48:52.119574 2585 feature_gate.go:328] unrecognized feature gate: Example Jul 18 11:48:52.129080 ip-10-0-136-240 kubenswrapper[2585]: W0718 11:48:52.119577 2585 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 18 11:48:52.129080 ip-10-0-136-240 kubenswrapper[2585]: W0718 11:48:52.119580 2585 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 18 11:48:52.129080 ip-10-0-136-240 kubenswrapper[2585]: W0718 11:48:52.119582 2585 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 18 11:48:52.129080 ip-10-0-136-240 kubenswrapper[2585]: W0718 11:48:52.119585 2585 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 18 11:48:52.129080 ip-10-0-136-240 kubenswrapper[2585]: W0718 11:48:52.119589 2585 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 18 11:48:52.129080 ip-10-0-136-240 kubenswrapper[2585]: W0718 11:48:52.119592 2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 18 11:48:52.129080 ip-10-0-136-240 kubenswrapper[2585]: W0718 11:48:52.119596 2585 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 18 11:48:52.129080 ip-10-0-136-240 kubenswrapper[2585]: W0718 11:48:52.119598 2585 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 18 11:48:52.129648 ip-10-0-136-240 kubenswrapper[2585]: W0718 11:48:52.119602 2585 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 18 11:48:52.129648 ip-10-0-136-240 kubenswrapper[2585]: W0718 11:48:52.119604 2585 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 18 11:48:52.129648 ip-10-0-136-240 kubenswrapper[2585]: W0718 11:48:52.119607 2585 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 18 11:48:52.129648 ip-10-0-136-240 kubenswrapper[2585]: W0718 11:48:52.119609 2585 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 18 11:48:52.129648 ip-10-0-136-240 kubenswrapper[2585]: W0718 11:48:52.119612 2585 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 18 11:48:52.129648 ip-10-0-136-240 kubenswrapper[2585]: W0718 11:48:52.119616 2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 18 11:48:52.129648 ip-10-0-136-240 kubenswrapper[2585]: W0718 11:48:52.119618 2585 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 18 11:48:52.129648 ip-10-0-136-240 kubenswrapper[2585]: W0718 11:48:52.119621 2585 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 18 11:48:52.129648 ip-10-0-136-240 kubenswrapper[2585]: W0718 11:48:52.119623 2585 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 18 11:48:52.129648 ip-10-0-136-240 kubenswrapper[2585]: W0718 11:48:52.119626 2585 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 18 11:48:52.129648 ip-10-0-136-240 kubenswrapper[2585]: W0718 11:48:52.119628 2585 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 18 11:48:52.129648 ip-10-0-136-240 kubenswrapper[2585]: W0718 11:48:52.119631 2585 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 18 11:48:52.129648 ip-10-0-136-240 kubenswrapper[2585]: W0718 11:48:52.119633 2585 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 18 11:48:52.129648 ip-10-0-136-240 kubenswrapper[2585]: W0718 11:48:52.119636 2585 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 18 11:48:52.129648 ip-10-0-136-240 kubenswrapper[2585]: W0718 11:48:52.119638 2585 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 18 11:48:52.129648 ip-10-0-136-240 kubenswrapper[2585]: W0718 11:48:52.119641 2585 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 18 11:48:52.129648 ip-10-0-136-240 kubenswrapper[2585]: W0718 11:48:52.119643 2585 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 18 11:48:52.129648 ip-10-0-136-240 kubenswrapper[2585]: W0718 11:48:52.119646 2585 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 18 11:48:52.129648 ip-10-0-136-240 kubenswrapper[2585]: W0718 11:48:52.119648 2585 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 18 11:48:52.129648 ip-10-0-136-240 kubenswrapper[2585]: W0718 11:48:52.119651 2585 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 18 11:48:52.130158 ip-10-0-136-240 kubenswrapper[2585]: W0718 11:48:52.119654 2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 18 11:48:52.130158 ip-10-0-136-240 kubenswrapper[2585]: W0718 11:48:52.119657 2585 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 18 11:48:52.130158 ip-10-0-136-240 kubenswrapper[2585]: W0718 11:48:52.119659 2585 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 18 11:48:52.130158 ip-10-0-136-240 kubenswrapper[2585]: W0718 11:48:52.119662 2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 18 11:48:52.130158 ip-10-0-136-240 kubenswrapper[2585]: W0718 11:48:52.119664 2585 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 18 11:48:52.130158 ip-10-0-136-240 kubenswrapper[2585]: W0718 11:48:52.119666 2585 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 18 11:48:52.130158 ip-10-0-136-240 kubenswrapper[2585]: W0718 11:48:52.119669 2585 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 18 11:48:52.130158 ip-10-0-136-240 kubenswrapper[2585]: W0718 11:48:52.119671 2585 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 18 11:48:52.130158 ip-10-0-136-240 kubenswrapper[2585]: W0718 11:48:52.119674 2585 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 18 11:48:52.130158 ip-10-0-136-240 kubenswrapper[2585]: W0718 11:48:52.119676 2585 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 18 11:48:52.130158 ip-10-0-136-240 kubenswrapper[2585]: W0718 11:48:52.119693 2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 18 11:48:52.130158 ip-10-0-136-240 kubenswrapper[2585]: W0718 11:48:52.119697 2585 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 18 11:48:52.130158 ip-10-0-136-240 kubenswrapper[2585]: W0718 11:48:52.119700 2585 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 18 11:48:52.130158 ip-10-0-136-240 kubenswrapper[2585]: W0718 11:48:52.119703 2585 feature_gate.go:328] unrecognized feature gate: Example2 Jul 18 11:48:52.130158 ip-10-0-136-240 kubenswrapper[2585]: W0718 11:48:52.119706 2585 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 18 11:48:52.130158 ip-10-0-136-240 kubenswrapper[2585]: W0718 11:48:52.119709 2585 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 18 11:48:52.130158 ip-10-0-136-240 kubenswrapper[2585]: W0718 11:48:52.119712 2585 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 18 11:48:52.130158 ip-10-0-136-240 kubenswrapper[2585]: W0718 11:48:52.119716 2585 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 18 11:48:52.130158 ip-10-0-136-240 kubenswrapper[2585]: W0718 11:48:52.119719 2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 18 11:48:52.130158 ip-10-0-136-240 kubenswrapper[2585]: W0718 11:48:52.119721 2585 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 18 11:48:52.130735 ip-10-0-136-240 kubenswrapper[2585]: W0718 11:48:52.119723 2585 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 18 11:48:52.130735 ip-10-0-136-240 kubenswrapper[2585]: W0718 11:48:52.119726 2585 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 18 11:48:52.130735 ip-10-0-136-240 kubenswrapper[2585]: W0718 11:48:52.119728 2585 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 18 11:48:52.130735 ip-10-0-136-240 kubenswrapper[2585]: W0718 11:48:52.119731 2585 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 18 11:48:52.130735 ip-10-0-136-240 kubenswrapper[2585]: W0718 11:48:52.119733 2585 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 18 11:48:52.130735 ip-10-0-136-240 kubenswrapper[2585]: W0718 11:48:52.119736 2585 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 18 11:48:52.130735 ip-10-0-136-240 kubenswrapper[2585]: W0718 11:48:52.119739 2585 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 18 11:48:52.130735 ip-10-0-136-240 kubenswrapper[2585]: W0718 11:48:52.119741 2585 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 18 11:48:52.130735 ip-10-0-136-240 kubenswrapper[2585]: W0718 11:48:52.119743 2585 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 18 11:48:52.130735 ip-10-0-136-240 kubenswrapper[2585]: W0718 11:48:52.119746 2585 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 18 11:48:52.130735 ip-10-0-136-240 kubenswrapper[2585]: W0718 11:48:52.119748 2585 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 18 11:48:52.130735 ip-10-0-136-240 kubenswrapper[2585]: W0718 11:48:52.119752 2585 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 18 11:48:52.130735 ip-10-0-136-240 kubenswrapper[2585]: W0718 11:48:52.119755 2585 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 18 11:48:52.130735 ip-10-0-136-240 kubenswrapper[2585]: W0718 11:48:52.119757 2585 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 18 11:48:52.130735 ip-10-0-136-240 kubenswrapper[2585]: W0718 11:48:52.119761 2585 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 18 11:48:52.130735 ip-10-0-136-240 kubenswrapper[2585]: W0718 11:48:52.119765 2585 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 18 11:48:52.130735 ip-10-0-136-240 kubenswrapper[2585]: W0718 11:48:52.119768 2585 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 18 11:48:52.130735 ip-10-0-136-240 kubenswrapper[2585]: W0718 11:48:52.119771 2585 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 18 11:48:52.130735 ip-10-0-136-240 kubenswrapper[2585]: W0718 11:48:52.119773 2585 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 18 11:48:52.131228 ip-10-0-136-240 kubenswrapper[2585]: W0718 11:48:52.119776 2585 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 18 11:48:52.131228 ip-10-0-136-240 kubenswrapper[2585]: W0718 11:48:52.119778 2585 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 18 11:48:52.131228 ip-10-0-136-240 kubenswrapper[2585]: W0718 11:48:52.119781 2585 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 18 11:48:52.131228 ip-10-0-136-240 kubenswrapper[2585]: W0718 11:48:52.119784 2585 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 18 11:48:52.131228 ip-10-0-136-240 kubenswrapper[2585]: W0718 11:48:52.119786 2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 18 11:48:52.131228 ip-10-0-136-240 kubenswrapper[2585]: W0718 11:48:52.119789 2585 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 18 11:48:52.131228 ip-10-0-136-240 kubenswrapper[2585]: W0718 11:48:52.119791 2585 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 18 11:48:52.131228 ip-10-0-136-240 kubenswrapper[2585]: W0718 11:48:52.119794 2585 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 18 11:48:52.131228 ip-10-0-136-240 kubenswrapper[2585]: W0718 11:48:52.119797 2585 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 18 11:48:52.131228 ip-10-0-136-240 kubenswrapper[2585]: W0718 11:48:52.119799 2585 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 18 11:48:52.131228 ip-10-0-136-240 kubenswrapper[2585]: W0718 11:48:52.119803 2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 18 11:48:52.131228 ip-10-0-136-240 kubenswrapper[2585]: W0718 11:48:52.119805 2585 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 18 11:48:52.131228 ip-10-0-136-240 kubenswrapper[2585]: W0718 11:48:52.119808 2585 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 18 11:48:52.131228 ip-10-0-136-240 kubenswrapper[2585]: W0718 11:48:52.119810 2585 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 18 11:48:52.131228 ip-10-0-136-240 kubenswrapper[2585]: W0718 11:48:52.119813 2585 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 18 11:48:52.131228 ip-10-0-136-240 kubenswrapper[2585]: W0718 11:48:52.119815 2585 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 18 11:48:52.131228 ip-10-0-136-240 kubenswrapper[2585]: W0718 11:48:52.119817 2585 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 18 11:48:52.131228 ip-10-0-136-240 kubenswrapper[2585]: W0718 11:48:52.119820 2585 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 18 11:48:52.131228 ip-10-0-136-240 kubenswrapper[2585]: W0718 11:48:52.119822 2585 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 18 11:48:52.131228 ip-10-0-136-240 kubenswrapper[2585]: W0718 11:48:52.119825 2585 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 18 11:48:52.131850 ip-10-0-136-240 kubenswrapper[2585]: W0718 11:48:52.119829 2585 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 18 11:48:52.131850 ip-10-0-136-240 kubenswrapper[2585]: W0718 11:48:52.119832 2585 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 18 11:48:52.131850 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:52.121697 2585 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 18 11:48:52.131850 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:52.129157 2585 server.go:529] "Kubelet version" kubeletVersion="v1.34.9" Jul 18 11:48:52.131850 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:52.129172 2585 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Jul 18 11:48:52.131850 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:52.129191 2585 watchdog_linux.go:95] "Systemd watchdog is not enabled" Jul 18 11:48:52.131850 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:52.129197 2585 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Jul 18 11:48:52.131850 ip-10-0-136-240 kubenswrapper[2585]: W0718 11:48:52.129253 2585 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 18 11:48:52.131850 ip-10-0-136-240 kubenswrapper[2585]: W0718 11:48:52.129258 2585 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 18 11:48:52.131850 ip-10-0-136-240 kubenswrapper[2585]: W0718 11:48:52.129262 2585 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 18 11:48:52.131850 ip-10-0-136-240 kubenswrapper[2585]: W0718 11:48:52.129265 2585 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 18 11:48:52.131850 ip-10-0-136-240 kubenswrapper[2585]: W0718 11:48:52.129267 2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 18 11:48:52.131850 ip-10-0-136-240 kubenswrapper[2585]: W0718 11:48:52.129270 2585 feature_gate.go:328] unrecognized feature gate: Example Jul 18 11:48:52.131850 ip-10-0-136-240 kubenswrapper[2585]: W0718 11:48:52.129272 2585 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 18 11:48:52.132381 ip-10-0-136-240 kubenswrapper[2585]: W0718 11:48:52.129275 2585 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 18 11:48:52.132381 ip-10-0-136-240 kubenswrapper[2585]: W0718 11:48:52.129278 2585 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 18 11:48:52.132381 ip-10-0-136-240 kubenswrapper[2585]: W0718 11:48:52.129280 2585 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 18 11:48:52.132381 ip-10-0-136-240 kubenswrapper[2585]: W0718 11:48:52.129283 2585 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 18 11:48:52.132381 ip-10-0-136-240 kubenswrapper[2585]: W0718 11:48:52.129285 2585 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 18 11:48:52.132381 ip-10-0-136-240 kubenswrapper[2585]: W0718 11:48:52.129288 2585 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 18 11:48:52.132381 ip-10-0-136-240 kubenswrapper[2585]: W0718 11:48:52.129290 2585 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 18 11:48:52.132381 ip-10-0-136-240 kubenswrapper[2585]: W0718 11:48:52.129293 2585 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 18 11:48:52.132381 ip-10-0-136-240 kubenswrapper[2585]: W0718 11:48:52.129295 2585 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 18 11:48:52.132381 ip-10-0-136-240 kubenswrapper[2585]: W0718 11:48:52.129298 2585 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 18 11:48:52.132381 ip-10-0-136-240 kubenswrapper[2585]: W0718 11:48:52.129300 2585 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 18 11:48:52.132381 ip-10-0-136-240 kubenswrapper[2585]: W0718 11:48:52.129303 2585 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 18 11:48:52.132381 ip-10-0-136-240 kubenswrapper[2585]: W0718 11:48:52.129305 2585 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 18 11:48:52.132381 ip-10-0-136-240 kubenswrapper[2585]: W0718 11:48:52.129308 2585 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 18 11:48:52.132381 ip-10-0-136-240 kubenswrapper[2585]: W0718 11:48:52.129310 2585 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 18 11:48:52.132381 ip-10-0-136-240 kubenswrapper[2585]: W0718 11:48:52.129312 2585 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 18 11:48:52.132381 ip-10-0-136-240 kubenswrapper[2585]: W0718 11:48:52.129316 2585 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 18 11:48:52.132381 ip-10-0-136-240 kubenswrapper[2585]: W0718 11:48:52.129320 2585 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 18 11:48:52.132381 ip-10-0-136-240 kubenswrapper[2585]: W0718 11:48:52.129323 2585 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 18 11:48:52.133035 ip-10-0-136-240 kubenswrapper[2585]: W0718 11:48:52.129326 2585 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 18 11:48:52.133035 ip-10-0-136-240 kubenswrapper[2585]: W0718 11:48:52.129329 2585 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 18 11:48:52.133035 ip-10-0-136-240 kubenswrapper[2585]: W0718 11:48:52.129332 2585 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 18 11:48:52.133035 ip-10-0-136-240 kubenswrapper[2585]: W0718 11:48:52.129336 2585 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 18 11:48:52.133035 ip-10-0-136-240 kubenswrapper[2585]: W0718 11:48:52.129338 2585 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 18 11:48:52.133035 ip-10-0-136-240 kubenswrapper[2585]: W0718 11:48:52.129341 2585 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 18 11:48:52.133035 ip-10-0-136-240 kubenswrapper[2585]: W0718 11:48:52.129344 2585 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 18 11:48:52.133035 ip-10-0-136-240 kubenswrapper[2585]: W0718 11:48:52.129346 2585 feature_gate.go:328] unrecognized feature gate: Example2 Jul 18 11:48:52.133035 ip-10-0-136-240 kubenswrapper[2585]: W0718 11:48:52.129349 2585 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 18 11:48:52.133035 ip-10-0-136-240 kubenswrapper[2585]: W0718 11:48:52.129351 2585 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 18 11:48:52.133035 ip-10-0-136-240 kubenswrapper[2585]: W0718 11:48:52.129354 2585 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 18 11:48:52.133035 ip-10-0-136-240 kubenswrapper[2585]: W0718 11:48:52.129356 2585 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 18 11:48:52.133035 ip-10-0-136-240 kubenswrapper[2585]: W0718 11:48:52.129359 2585 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 18 11:48:52.133035 ip-10-0-136-240 kubenswrapper[2585]: W0718 11:48:52.129361 2585 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 18 11:48:52.133035 ip-10-0-136-240 kubenswrapper[2585]: W0718 11:48:52.129364 2585 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 18 11:48:52.133035 ip-10-0-136-240 kubenswrapper[2585]: W0718 11:48:52.129366 2585 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 18 11:48:52.133035 ip-10-0-136-240 kubenswrapper[2585]: W0718 11:48:52.129369 2585 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 18 11:48:52.133035 ip-10-0-136-240 kubenswrapper[2585]: W0718 11:48:52.129371 2585 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 18 11:48:52.133035 ip-10-0-136-240 kubenswrapper[2585]: W0718 11:48:52.129374 2585 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 18 11:48:52.133035 ip-10-0-136-240 kubenswrapper[2585]: W0718 11:48:52.129376 2585 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 18 11:48:52.133589 ip-10-0-136-240 kubenswrapper[2585]: W0718 11:48:52.129379 2585 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 18 11:48:52.133589 ip-10-0-136-240 kubenswrapper[2585]: W0718 11:48:52.129381 2585 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 18 11:48:52.133589 ip-10-0-136-240 kubenswrapper[2585]: W0718 11:48:52.129384 2585 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 18 11:48:52.133589 ip-10-0-136-240 kubenswrapper[2585]: W0718 11:48:52.129386 2585 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 18 11:48:52.133589 ip-10-0-136-240 kubenswrapper[2585]: W0718 11:48:52.129388 2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 18 11:48:52.133589 ip-10-0-136-240 kubenswrapper[2585]: W0718 11:48:52.129391 2585 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 18 11:48:52.133589 ip-10-0-136-240 kubenswrapper[2585]: W0718 11:48:52.129393 2585 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 18 11:48:52.133589 ip-10-0-136-240 kubenswrapper[2585]: W0718 11:48:52.129395 2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 18 11:48:52.133589 ip-10-0-136-240 kubenswrapper[2585]: W0718 11:48:52.129398 2585 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 18 11:48:52.133589 ip-10-0-136-240 kubenswrapper[2585]: W0718 11:48:52.129400 2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 18 11:48:52.133589 ip-10-0-136-240 kubenswrapper[2585]: W0718 11:48:52.129403 2585 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 18 11:48:52.133589 ip-10-0-136-240 kubenswrapper[2585]: W0718 11:48:52.129405 2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 18 11:48:52.133589 ip-10-0-136-240 kubenswrapper[2585]: W0718 11:48:52.129408 2585 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 18 11:48:52.133589 ip-10-0-136-240 kubenswrapper[2585]: W0718 11:48:52.129410 2585 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 18 11:48:52.133589 ip-10-0-136-240 kubenswrapper[2585]: W0718 11:48:52.129413 2585 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 18 11:48:52.133589 ip-10-0-136-240 kubenswrapper[2585]: W0718 11:48:52.129416 2585 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 18 11:48:52.133589 ip-10-0-136-240 kubenswrapper[2585]: W0718 11:48:52.129419 2585 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 18 11:48:52.133589 ip-10-0-136-240 kubenswrapper[2585]: W0718 11:48:52.129422 2585 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 18 11:48:52.133589 ip-10-0-136-240 kubenswrapper[2585]: W0718 11:48:52.129425 2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 18 11:48:52.133589 ip-10-0-136-240 kubenswrapper[2585]: W0718 11:48:52.129427 2585 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 18 11:48:52.134110 ip-10-0-136-240 kubenswrapper[2585]: W0718 11:48:52.129430 2585 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 18 11:48:52.134110 ip-10-0-136-240 kubenswrapper[2585]: W0718 11:48:52.129432 2585 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 18 11:48:52.134110 ip-10-0-136-240 kubenswrapper[2585]: W0718 11:48:52.129434 2585 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 18 11:48:52.134110 ip-10-0-136-240 kubenswrapper[2585]: W0718 11:48:52.129437 2585 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 18 11:48:52.134110 ip-10-0-136-240 kubenswrapper[2585]: W0718 11:48:52.129439 2585 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 18 11:48:52.134110 ip-10-0-136-240 kubenswrapper[2585]: W0718 11:48:52.129442 2585 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 18 11:48:52.134110 ip-10-0-136-240 kubenswrapper[2585]: W0718 11:48:52.129444 2585 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 18 11:48:52.134110 ip-10-0-136-240 kubenswrapper[2585]: W0718 11:48:52.129446 2585 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 18 11:48:52.134110 ip-10-0-136-240 kubenswrapper[2585]: W0718 11:48:52.129449 2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 18 11:48:52.134110 ip-10-0-136-240 kubenswrapper[2585]: W0718 11:48:52.129451 2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 18 11:48:52.134110 ip-10-0-136-240 kubenswrapper[2585]: W0718 11:48:52.129454 2585 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 18 11:48:52.134110 ip-10-0-136-240 kubenswrapper[2585]: W0718 11:48:52.129456 2585 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 18 11:48:52.134110 ip-10-0-136-240 kubenswrapper[2585]: W0718 11:48:52.129458 2585 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 18 11:48:52.134110 ip-10-0-136-240 kubenswrapper[2585]: W0718 11:48:52.129461 2585 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 18 11:48:52.134110 ip-10-0-136-240 kubenswrapper[2585]: W0718 11:48:52.129463 2585 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 18 11:48:52.134110 ip-10-0-136-240 kubenswrapper[2585]: W0718 11:48:52.129465 2585 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 18 11:48:52.134110 ip-10-0-136-240 kubenswrapper[2585]: W0718 11:48:52.129468 2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 18 11:48:52.134110 ip-10-0-136-240 kubenswrapper[2585]: W0718 11:48:52.129470 2585 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 18 11:48:52.134110 ip-10-0-136-240 kubenswrapper[2585]: W0718 11:48:52.129488 2585 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 18 11:48:52.134110 ip-10-0-136-240 kubenswrapper[2585]: W0718 11:48:52.129492 2585 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 18 11:48:52.134646 ip-10-0-136-240 kubenswrapper[2585]: W0718 11:48:52.129498 2585 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 18 11:48:52.134646 ip-10-0-136-240 kubenswrapper[2585]: W0718 11:48:52.129503 2585 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 18 11:48:52.134646 ip-10-0-136-240 kubenswrapper[2585]: W0718 11:48:52.129506 2585 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 18 11:48:52.134646 ip-10-0-136-240 kubenswrapper[2585]: W0718 11:48:52.129509 2585 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 18 11:48:52.134646 ip-10-0-136-240 kubenswrapper[2585]: W0718 11:48:52.129512 2585 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 18 11:48:52.134646 ip-10-0-136-240 kubenswrapper[2585]: W0718 11:48:52.129515 2585 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 18 11:48:52.134646 ip-10-0-136-240 kubenswrapper[2585]: W0718 11:48:52.129519 2585 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 18 11:48:52.134646 ip-10-0-136-240 kubenswrapper[2585]: W0718 11:48:52.129521 2585 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 18 11:48:52.134646 ip-10-0-136-240 kubenswrapper[2585]: W0718 11:48:52.129524 2585 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 18 11:48:52.134646 ip-10-0-136-240 kubenswrapper[2585]: W0718 11:48:52.129530 2585 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 18 11:48:52.134646 ip-10-0-136-240 kubenswrapper[2585]: W0718 11:48:52.129533 2585 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 18 11:48:52.134646 ip-10-0-136-240 kubenswrapper[2585]: W0718 11:48:52.129535 2585 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 18 11:48:52.134646 ip-10-0-136-240 kubenswrapper[2585]: W0718 11:48:52.129538 2585 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 18 11:48:52.134646 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:52.129543 2585 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 18 11:48:52.135023 ip-10-0-136-240 kubenswrapper[2585]: W0718 11:48:52.129652 2585 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 18 11:48:52.135023 ip-10-0-136-240 kubenswrapper[2585]: W0718 11:48:52.129657 2585 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 18 11:48:52.135023 ip-10-0-136-240 kubenswrapper[2585]: W0718 11:48:52.129660 2585 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 18 11:48:52.135023 ip-10-0-136-240 kubenswrapper[2585]: W0718 11:48:52.129663 2585 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 18 11:48:52.135023 ip-10-0-136-240 kubenswrapper[2585]: W0718 11:48:52.129666 2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 18 11:48:52.135023 ip-10-0-136-240 kubenswrapper[2585]: W0718 11:48:52.129669 2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 18 11:48:52.135023 ip-10-0-136-240 kubenswrapper[2585]: W0718 11:48:52.129671 2585 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 18 11:48:52.135023 ip-10-0-136-240 kubenswrapper[2585]: W0718 11:48:52.129674 2585 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 18 11:48:52.135023 ip-10-0-136-240 kubenswrapper[2585]: W0718 11:48:52.129677 2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 18 11:48:52.135023 ip-10-0-136-240 kubenswrapper[2585]: W0718 11:48:52.129679 2585 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 18 11:48:52.135023 ip-10-0-136-240 kubenswrapper[2585]: W0718 11:48:52.129681 2585 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 18 11:48:52.135023 ip-10-0-136-240 kubenswrapper[2585]: W0718 11:48:52.129684 2585 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 18 11:48:52.135023 ip-10-0-136-240 kubenswrapper[2585]: W0718 11:48:52.129686 2585 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 18 11:48:52.135023 ip-10-0-136-240 kubenswrapper[2585]: W0718 11:48:52.129689 2585 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 18 11:48:52.135023 ip-10-0-136-240 kubenswrapper[2585]: W0718 11:48:52.129691 2585 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 18 11:48:52.135023 ip-10-0-136-240 kubenswrapper[2585]: W0718 11:48:52.129694 2585 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 18 11:48:52.135023 ip-10-0-136-240 kubenswrapper[2585]: W0718 11:48:52.129696 2585 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 18 11:48:52.135023 ip-10-0-136-240 kubenswrapper[2585]: W0718 11:48:52.129699 2585 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 18 11:48:52.135023 ip-10-0-136-240 kubenswrapper[2585]: W0718 11:48:52.129702 2585 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 18 11:48:52.135619 ip-10-0-136-240 kubenswrapper[2585]: W0718 11:48:52.129706 2585 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 18 11:48:52.135619 ip-10-0-136-240 kubenswrapper[2585]: W0718 11:48:52.129708 2585 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 18 11:48:52.135619 ip-10-0-136-240 kubenswrapper[2585]: W0718 11:48:52.129711 2585 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 18 11:48:52.135619 ip-10-0-136-240 kubenswrapper[2585]: W0718 11:48:52.129714 2585 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 18 11:48:52.135619 ip-10-0-136-240 kubenswrapper[2585]: W0718 11:48:52.129716 2585 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 18 11:48:52.135619 ip-10-0-136-240 kubenswrapper[2585]: W0718 11:48:52.129719 2585 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 18 11:48:52.135619 ip-10-0-136-240 kubenswrapper[2585]: W0718 11:48:52.129722 2585 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 18 11:48:52.135619 ip-10-0-136-240 kubenswrapper[2585]: W0718 11:48:52.129724 2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 18 11:48:52.135619 ip-10-0-136-240 kubenswrapper[2585]: W0718 11:48:52.129727 2585 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 18 11:48:52.135619 ip-10-0-136-240 kubenswrapper[2585]: W0718 11:48:52.129730 2585 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 18 11:48:52.135619 ip-10-0-136-240 kubenswrapper[2585]: W0718 11:48:52.129732 2585 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 18 11:48:52.135619 ip-10-0-136-240 kubenswrapper[2585]: W0718 11:48:52.129735 2585 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 18 11:48:52.135619 ip-10-0-136-240 kubenswrapper[2585]: W0718 11:48:52.129737 2585 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 18 11:48:52.135619 ip-10-0-136-240 kubenswrapper[2585]: W0718 11:48:52.129739 2585 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 18 11:48:52.135619 ip-10-0-136-240 kubenswrapper[2585]: W0718 11:48:52.129742 2585 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 18 11:48:52.135619 ip-10-0-136-240 kubenswrapper[2585]: W0718 11:48:52.129744 2585 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 18 11:48:52.135619 ip-10-0-136-240 kubenswrapper[2585]: W0718 11:48:52.129747 2585 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 18 11:48:52.135619 ip-10-0-136-240 kubenswrapper[2585]: W0718 11:48:52.129749 2585 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 18 11:48:52.135619 ip-10-0-136-240 kubenswrapper[2585]: W0718 11:48:52.129752 2585 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 18 11:48:52.135619 ip-10-0-136-240 kubenswrapper[2585]: W0718 11:48:52.129754 2585 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 18 11:48:52.136127 ip-10-0-136-240 kubenswrapper[2585]: W0718 11:48:52.129757 2585 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 18 11:48:52.136127 ip-10-0-136-240 kubenswrapper[2585]: W0718 11:48:52.129759 2585 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 18 11:48:52.136127 ip-10-0-136-240 kubenswrapper[2585]: W0718 11:48:52.129761 2585 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 18 11:48:52.136127 ip-10-0-136-240 kubenswrapper[2585]: W0718 11:48:52.129764 2585 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 18 11:48:52.136127 ip-10-0-136-240 kubenswrapper[2585]: W0718 11:48:52.129766 2585 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 18 11:48:52.136127 ip-10-0-136-240 kubenswrapper[2585]: W0718 11:48:52.129768 2585 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 18 11:48:52.136127 ip-10-0-136-240 kubenswrapper[2585]: W0718 11:48:52.129771 2585 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 18 11:48:52.136127 ip-10-0-136-240 kubenswrapper[2585]: W0718 11:48:52.129773 2585 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 18 11:48:52.136127 ip-10-0-136-240 kubenswrapper[2585]: W0718 11:48:52.129776 2585 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 18 11:48:52.136127 ip-10-0-136-240 kubenswrapper[2585]: W0718 11:48:52.129778 2585 feature_gate.go:328] unrecognized feature gate: Example2 Jul 18 11:48:52.136127 ip-10-0-136-240 kubenswrapper[2585]: W0718 11:48:52.129782 2585 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 18 11:48:52.136127 ip-10-0-136-240 kubenswrapper[2585]: W0718 11:48:52.129784 2585 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 18 11:48:52.136127 ip-10-0-136-240 kubenswrapper[2585]: W0718 11:48:52.129788 2585 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 18 11:48:52.136127 ip-10-0-136-240 kubenswrapper[2585]: W0718 11:48:52.129792 2585 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 18 11:48:52.136127 ip-10-0-136-240 kubenswrapper[2585]: W0718 11:48:52.129795 2585 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 18 11:48:52.136127 ip-10-0-136-240 kubenswrapper[2585]: W0718 11:48:52.129798 2585 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 18 11:48:52.136127 ip-10-0-136-240 kubenswrapper[2585]: W0718 11:48:52.129801 2585 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 18 11:48:52.136127 ip-10-0-136-240 kubenswrapper[2585]: W0718 11:48:52.129804 2585 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 18 11:48:52.136127 ip-10-0-136-240 kubenswrapper[2585]: W0718 11:48:52.129807 2585 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 18 11:48:52.136645 ip-10-0-136-240 kubenswrapper[2585]: W0718 11:48:52.129810 2585 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 18 11:48:52.136645 ip-10-0-136-240 kubenswrapper[2585]: W0718 11:48:52.129812 2585 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 18 11:48:52.136645 ip-10-0-136-240 kubenswrapper[2585]: W0718 11:48:52.129815 2585 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 18 11:48:52.136645 ip-10-0-136-240 kubenswrapper[2585]: W0718 11:48:52.129818 2585 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 18 11:48:52.136645 ip-10-0-136-240 kubenswrapper[2585]: W0718 11:48:52.129821 2585 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 18 11:48:52.136645 ip-10-0-136-240 kubenswrapper[2585]: W0718 11:48:52.129823 2585 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 18 11:48:52.136645 ip-10-0-136-240 kubenswrapper[2585]: W0718 11:48:52.129825 2585 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 18 11:48:52.136645 ip-10-0-136-240 kubenswrapper[2585]: W0718 11:48:52.129828 2585 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 18 11:48:52.136645 ip-10-0-136-240 kubenswrapper[2585]: W0718 11:48:52.129830 2585 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 18 11:48:52.136645 ip-10-0-136-240 kubenswrapper[2585]: W0718 11:48:52.129832 2585 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 18 11:48:52.136645 ip-10-0-136-240 kubenswrapper[2585]: W0718 11:48:52.129835 2585 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 18 11:48:52.136645 ip-10-0-136-240 kubenswrapper[2585]: W0718 11:48:52.129837 2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 18 11:48:52.136645 ip-10-0-136-240 kubenswrapper[2585]: W0718 11:48:52.129840 2585 feature_gate.go:328] unrecognized feature gate: Example Jul 18 11:48:52.136645 ip-10-0-136-240 kubenswrapper[2585]: W0718 11:48:52.129843 2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 18 11:48:52.136645 ip-10-0-136-240 kubenswrapper[2585]: W0718 11:48:52.129845 2585 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 18 11:48:52.136645 ip-10-0-136-240 kubenswrapper[2585]: W0718 11:48:52.129848 2585 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 18 11:48:52.136645 ip-10-0-136-240 kubenswrapper[2585]: W0718 11:48:52.129850 2585 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 18 11:48:52.136645 ip-10-0-136-240 kubenswrapper[2585]: W0718 11:48:52.129852 2585 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 18 11:48:52.136645 ip-10-0-136-240 kubenswrapper[2585]: W0718 11:48:52.129855 2585 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 18 11:48:52.136645 ip-10-0-136-240 kubenswrapper[2585]: W0718 11:48:52.129857 2585 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 18 11:48:52.137152 ip-10-0-136-240 kubenswrapper[2585]: W0718 11:48:52.129859 2585 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 18 11:48:52.137152 ip-10-0-136-240 kubenswrapper[2585]: W0718 11:48:52.129862 2585 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 18 11:48:52.137152 ip-10-0-136-240 kubenswrapper[2585]: W0718 11:48:52.129864 2585 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 18 11:48:52.137152 ip-10-0-136-240 kubenswrapper[2585]: W0718 11:48:52.129866 2585 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 18 11:48:52.137152 ip-10-0-136-240 kubenswrapper[2585]: W0718 11:48:52.129869 2585 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 18 11:48:52.137152 ip-10-0-136-240 kubenswrapper[2585]: W0718 11:48:52.129871 2585 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 18 11:48:52.137152 ip-10-0-136-240 kubenswrapper[2585]: W0718 11:48:52.129873 2585 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 18 11:48:52.137152 ip-10-0-136-240 kubenswrapper[2585]: W0718 11:48:52.129876 2585 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 18 11:48:52.137152 ip-10-0-136-240 kubenswrapper[2585]: W0718 11:48:52.129878 2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 18 11:48:52.137152 ip-10-0-136-240 kubenswrapper[2585]: W0718 11:48:52.129880 2585 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 18 11:48:52.137152 ip-10-0-136-240 kubenswrapper[2585]: W0718 11:48:52.129883 2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 18 11:48:52.137152 ip-10-0-136-240 kubenswrapper[2585]: W0718 11:48:52.129886 2585 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 18 11:48:52.137152 ip-10-0-136-240 kubenswrapper[2585]: W0718 11:48:52.129889 2585 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 18 11:48:52.137152 ip-10-0-136-240 kubenswrapper[2585]: W0718 11:48:52.129891 2585 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 18 11:48:52.137152 ip-10-0-136-240 kubenswrapper[2585]: W0718 11:48:52.129893 2585 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 18 11:48:52.137152 ip-10-0-136-240 kubenswrapper[2585]: W0718 11:48:52.129896 2585 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 18 11:48:52.137152 ip-10-0-136-240 kubenswrapper[2585]: W0718 11:48:52.129899 2585 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 18 11:48:52.137152 ip-10-0-136-240 kubenswrapper[2585]: W0718 11:48:52.129901 2585 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 18 11:48:52.137152 ip-10-0-136-240 kubenswrapper[2585]: W0718 11:48:52.129904 2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 18 11:48:52.137152 ip-10-0-136-240 kubenswrapper[2585]: W0718 11:48:52.129906 2585 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 18 11:48:52.137687 ip-10-0-136-240 kubenswrapper[2585]: W0718 11:48:52.129908 2585 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 18 11:48:52.137687 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:52.129913 2585 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 18 11:48:52.137687 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:52.130012 2585 server.go:962] "Client rotation is on, will bootstrap in background" Jul 18 11:48:52.137687 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:52.133810 2585 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Jul 18 11:48:52.137687 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:52.135357 2585 server.go:1019] "Starting client certificate rotation" Jul 18 11:48:52.137687 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:52.135453 2585 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 18 11:48:52.137687 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:52.135502 2585 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 18 11:48:52.138101 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:52.138088 2585 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 18 11:48:52.139218 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:52.139202 2585 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 18 11:48:52.147319 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:52.147303 2585 log.go:25] "Validated CRI v1 runtime API" Jul 18 11:48:52.149700 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:52.149686 2585 log.go:25] "Validated CRI v1 image API" Jul 18 11:48:52.153467 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:52.153446 2585 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Jul 18 11:48:52.156031 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:52.156015 2585 fs.go:135] Filesystem UUIDs: map[1ff87156-6cbb-41cc-87bc-6dc1457952cb:/dev/nvme0n1p4 56d6a3b5-ad79-485d-bd3f-572ea8b6dfe1:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2] Jul 18 11:48:52.156086 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:52.156032 2585 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Jul 18 11:48:52.161251 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:52.161146 2585 manager.go:216] Machine: {Timestamp:2026-07-18 11:48:52.159761808 +0000 UTC m=+0.373042660 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3103523 MemoryCapacity:32812175360 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2c0f68076852824649d0e017d36946 SystemUUID:ec2c0f68-0768-5282-4649-d0e017d36946 BootID:4f0ca281-d56b-4752-af6c-9a55cc619949 Filesystems:[{Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16406089728 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16406085632 Type:vfs Inodes:4005392 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6562435072 Type:vfs Inodes:819200 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:9f:c7:98:f6:77 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:9f:c7:98:f6:77 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:ca:e6:02:17:f3:f8 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812175360 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Jul 18 11:48:52.161251 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:52.161243 2585 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Jul 18 11:48:52.161385 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:52.161314 2585 manager.go:232] Version: {KernelVersion:5.14.0-570.128.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260715-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Jul 18 11:48:52.164902 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:52.164886 2585 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 18 11:48:52.165348 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:52.165329 2585 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Jul 18 11:48:52.165495 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:52.165354 2585 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-136-240.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Jul 18 11:48:52.166153 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:52.166144 2585 topology_manager.go:138] "Creating topology manager with none policy" Jul 18 11:48:52.166183 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:52.166156 2585 container_manager_linux.go:309] "Creating device plugin manager" Jul 18 11:48:52.166183 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:52.166164 2585 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 18 11:48:52.166183 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:52.166176 2585 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 18 11:48:52.168022 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:52.168010 2585 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Jul 18 11:48:52.172471 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:52.172459 2585 state_mem.go:36] "Initialized new in-memory state store" Jul 18 11:48:52.172605 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:52.172596 2585 server.go:1240] "Using root directory" path="/var/lib/kubelet" Jul 18 11:48:52.172679 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:52.172626 2585 kubelet.go:486] "Attempting to sync node with API server" Jul 18 11:48:52.172679 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:52.172637 2585 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Jul 18 11:48:52.172679 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:52.172651 2585 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Jul 18 11:48:52.172679 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:52.172660 2585 kubelet.go:398] "Adding apiserver pod source" Jul 18 11:48:52.172679 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:52.172669 2585 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Jul 18 11:48:52.172993 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:52.172979 2585 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 18 11:48:52.173050 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:52.172998 2585 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 18 11:48:52.176210 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:52.176189 2585 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1" Jul 18 11:48:52.177910 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:52.177897 2585 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Jul 18 11:48:52.177965 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:52.177927 2585 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Jul 18 11:48:52.178730 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:52.178712 2585 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Jul 18 11:48:52.178763 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:52.178735 2585 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Jul 18 11:48:52.178763 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:52.178741 2585 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Jul 18 11:48:52.178763 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:52.178746 2585 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Jul 18 11:48:52.178763 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:52.178754 2585 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Jul 18 11:48:52.178763 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:52.178760 2585 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Jul 18 11:48:52.178897 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:52.178767 2585 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Jul 18 11:48:52.178897 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:52.178773 2585 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Jul 18 11:48:52.178897 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:52.178779 2585 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Jul 18 11:48:52.178897 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:52.178785 2585 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Jul 18 11:48:52.178897 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:52.178793 2585 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Jul 18 11:48:52.178897 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:52.178797 2585 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Jul 18 11:48:52.178897 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:52.178820 2585 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Jul 18 11:48:52.178897 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:52.178825 2585 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Jul 18 11:48:52.181778 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:52.181764 2585 server.go:1268] "Started kubelet" Jul 18 11:48:52.181970 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:52.181884 2585 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Jul 18 11:48:52.182045 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:52.181992 2585 server_v1.go:49] "podresources" method="list" useActivePods=true Jul 18 11:48:52.182100 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:52.182049 2585 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Jul 18 11:48:52.182595 ip-10-0-136-240 systemd[1]: Started Kubernetes Kubelet. Jul 18 11:48:52.183312 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:52.183296 2585 server.go:310] "Adding debug handlers to kubelet server" Jul 18 11:48:52.183386 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:52.183309 2585 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Jul 18 11:48:52.184290 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:52.184272 2585 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-rg8x7" Jul 18 11:48:52.184532 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:52.184500 2585 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-136-240.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Jul 18 11:48:52.184584 ip-10-0-136-240 kubenswrapper[2585]: E0718 11:48:52.184524 2585 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Jul 18 11:48:52.184584 ip-10-0-136-240 kubenswrapper[2585]: E0718 11:48:52.184549 2585 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-136-240.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Jul 18 11:48:52.187160 ip-10-0-136-240 kubenswrapper[2585]: E0718 11:48:52.187140 2585 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Jul 18 11:48:52.189612 ip-10-0-136-240 kubenswrapper[2585]: E0718 11:48:52.188572 2585 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-136-240.ec2.internal.18c35fe91ca984ec default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-136-240.ec2.internal,UID:ip-10-0-136-240.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-136-240.ec2.internal,},FirstTimestamp:2026-07-18 11:48:52.181738732 +0000 UTC m=+0.395019584,LastTimestamp:2026-07-18 11:48:52.181738732 +0000 UTC m=+0.395019584,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-136-240.ec2.internal,}" Jul 18 11:48:52.191823 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:52.191809 2585 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-rg8x7" Jul 18 11:48:52.192945 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:52.192925 2585 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Jul 18 11:48:52.192945 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:52.192935 2585 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Jul 18 11:48:52.193208 ip-10-0-136-240 kubenswrapper[2585]: E0718 11:48:52.193115 2585 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-240.ec2.internal\" not found" Jul 18 11:48:52.193736 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:52.193720 2585 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Jul 18 11:48:52.193736 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:52.193721 2585 volume_manager.go:311] "The desired_state_of_world populator starts" Jul 18 11:48:52.193873 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:52.193750 2585 factory.go:55] Registering systemd factory Jul 18 11:48:52.193873 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:52.193766 2585 factory.go:223] Registration of the systemd container factory successfully Jul 18 11:48:52.193873 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:52.193751 2585 volume_manager.go:313] "Starting Kubelet Volume Manager" Jul 18 11:48:52.193999 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:52.193896 2585 reconstruct.go:97] "Volume reconstruction finished" Jul 18 11:48:52.193999 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:52.193909 2585 reconciler.go:29] "Reconciler: start to sync state" Jul 18 11:48:52.193999 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:52.193953 2585 factory.go:153] Registering CRI-O factory Jul 18 11:48:52.193999 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:52.193966 2585 factory.go:223] Registration of the crio container factory successfully Jul 18 11:48:52.194161 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:52.194015 2585 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Jul 18 11:48:52.194161 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:52.194038 2585 factory.go:103] Registering Raw factory Jul 18 11:48:52.194161 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:52.194053 2585 manager.go:1196] Started watching for new ooms in manager Jul 18 11:48:52.194595 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:52.194580 2585 manager.go:349] Starting recovery of all containers Jul 18 11:48:52.194825 ip-10-0-136-240 kubenswrapper[2585]: E0718 11:48:52.194803 2585 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" Jul 18 11:48:52.194986 ip-10-0-136-240 kubenswrapper[2585]: E0718 11:48:52.194926 2585 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-136-240.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Jul 18 11:48:52.205641 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:52.205623 2585 manager.go:354] Recovery completed Jul 18 11:48:52.210671 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:52.210578 2585 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 18 11:48:52.212345 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:52.212330 2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-240.ec2.internal" event="NodeHasSufficientMemory" Jul 18 11:48:52.212412 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:52.212356 2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-240.ec2.internal" event="NodeHasNoDiskPressure" Jul 18 11:48:52.212412 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:52.212366 2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-240.ec2.internal" event="NodeHasSufficientPID" Jul 18 11:48:52.212980 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:52.212957 2585 cpu_manager.go:222] "Starting CPU manager" policy="none" Jul 18 11:48:52.212980 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:52.212971 2585 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Jul 18 11:48:52.213092 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:52.212984 2585 state_mem.go:36] "Initialized new in-memory state store" Jul 18 11:48:52.213092 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:52.213017 2585 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 18 11:48:52.213679 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:52.213663 2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-240.ec2.internal" event="NodeHasSufficientMemory" Jul 18 11:48:52.213735 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:52.213691 2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-240.ec2.internal" event="NodeHasNoDiskPressure" Jul 18 11:48:52.213735 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:52.213703 2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-240.ec2.internal" event="NodeHasSufficientPID" Jul 18 11:48:52.214993 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:52.214981 2585 policy_none.go:49] "None policy: Start" Jul 18 11:48:52.215040 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:52.214997 2585 memory_manager.go:187] "Starting memorymanager" policy="None" Jul 18 11:48:52.215040 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:52.215007 2585 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Jul 18 11:48:52.216953 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:52.216942 2585 policy_none.go:47] "Start" Jul 18 11:48:52.258133 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:52.258113 2585 manager.go:337] "Starting Device Plugin manager" Jul 18 11:48:52.279864 ip-10-0-136-240 kubenswrapper[2585]: E0718 11:48:52.258161 2585 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Jul 18 11:48:52.279864 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:52.258174 2585 server.go:87] "Starting device plugin registration server" Jul 18 11:48:52.279864 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:52.258410 2585 eviction_manager.go:189] "Eviction manager: starting control loop" Jul 18 11:48:52.279864 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:52.258425 2585 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Jul 18 11:48:52.279864 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:52.258536 2585 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Jul 18 11:48:52.279864 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:52.258623 2585 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Jul 18 11:48:52.279864 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:52.258632 2585 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Jul 18 11:48:52.279864 ip-10-0-136-240 kubenswrapper[2585]: E0718 11:48:52.259170 2585 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Jul 18 11:48:52.279864 ip-10-0-136-240 kubenswrapper[2585]: E0718 11:48:52.259212 2585 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-136-240.ec2.internal\" not found" Jul 18 11:48:52.327967 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:52.327938 2585 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Jul 18 11:48:52.329272 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:52.329254 2585 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Jul 18 11:48:52.329354 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:52.329291 2585 status_manager.go:244] "Starting to sync pod status with apiserver" Jul 18 11:48:52.329354 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:52.329309 2585 kubelet.go:2448] "Starting kubelet main sync loop" Jul 18 11:48:52.329354 ip-10-0-136-240 kubenswrapper[2585]: E0718 11:48:52.329339 2585 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Jul 18 11:48:52.331423 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:52.331409 2585 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 18 11:48:52.359316 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:52.359271 2585 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 18 11:48:52.360047 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:52.360034 2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-240.ec2.internal" event="NodeHasSufficientMemory" Jul 18 11:48:52.360097 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:52.360062 2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-240.ec2.internal" event="NodeHasNoDiskPressure" Jul 18 11:48:52.360097 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:52.360072 2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-240.ec2.internal" event="NodeHasSufficientPID" Jul 18 11:48:52.360097 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:52.360096 2585 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-136-240.ec2.internal" Jul 18 11:48:52.370718 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:52.370701 2585 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-136-240.ec2.internal" Jul 18 11:48:52.370764 ip-10-0-136-240 kubenswrapper[2585]: E0718 11:48:52.370723 2585 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-136-240.ec2.internal\": node \"ip-10-0-136-240.ec2.internal\" not found" Jul 18 11:48:52.394842 ip-10-0-136-240 kubenswrapper[2585]: E0718 11:48:52.394822 2585 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-240.ec2.internal\" not found" Jul 18 11:48:52.429794 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:52.429772 2585 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["kube-system/kube-apiserver-proxy-ip-10-0-136-240.ec2.internal","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-240.ec2.internal"] Jul 18 11:48:52.429878 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:52.429842 2585 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 18 11:48:52.430693 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:52.430677 2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-240.ec2.internal" event="NodeHasSufficientMemory" Jul 18 11:48:52.430782 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:52.430711 2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-240.ec2.internal" event="NodeHasNoDiskPressure" Jul 18 11:48:52.430782 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:52.430727 2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-240.ec2.internal" event="NodeHasSufficientPID" Jul 18 11:48:52.433026 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:52.433011 2585 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 18 11:48:52.433144 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:52.433131 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-240.ec2.internal" Jul 18 11:48:52.433188 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:52.433160 2585 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 18 11:48:52.433729 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:52.433714 2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-240.ec2.internal" event="NodeHasSufficientMemory" Jul 18 11:48:52.433778 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:52.433714 2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-240.ec2.internal" event="NodeHasSufficientMemory" Jul 18 11:48:52.433778 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:52.433742 2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-240.ec2.internal" event="NodeHasNoDiskPressure" Jul 18 11:48:52.433778 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:52.433750 2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-240.ec2.internal" event="NodeHasNoDiskPressure" Jul 18 11:48:52.433778 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:52.433756 2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-240.ec2.internal" event="NodeHasSufficientPID" Jul 18 11:48:52.433778 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:52.433766 2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-240.ec2.internal" event="NodeHasSufficientPID" Jul 18 11:48:52.435975 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:52.435953 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-240.ec2.internal" Jul 18 11:48:52.436069 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:52.435981 2585 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 18 11:48:52.436750 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:52.436733 2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-240.ec2.internal" event="NodeHasSufficientMemory" Jul 18 11:48:52.436829 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:52.436763 2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-240.ec2.internal" event="NodeHasNoDiskPressure" Jul 18 11:48:52.436829 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:52.436774 2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-240.ec2.internal" event="NodeHasSufficientPID" Jul 18 11:48:52.472007 ip-10-0-136-240 kubenswrapper[2585]: E0718 11:48:52.471983 2585 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-136-240.ec2.internal\" not found" node="ip-10-0-136-240.ec2.internal" Jul 18 11:48:52.476192 ip-10-0-136-240 kubenswrapper[2585]: E0718 11:48:52.476176 2585 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-136-240.ec2.internal\" not found" node="ip-10-0-136-240.ec2.internal" Jul 18 11:48:52.495265 ip-10-0-136-240 kubenswrapper[2585]: E0718 11:48:52.495246 2585 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-240.ec2.internal\" not found" Jul 18 11:48:52.495318 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:52.495295 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/868b3db141f7bc43ba875f2543fbda86-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-136-240.ec2.internal\" (UID: \"868b3db141f7bc43ba875f2543fbda86\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-240.ec2.internal" Jul 18 11:48:52.495355 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:52.495315 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/868b3db141f7bc43ba875f2543fbda86-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-136-240.ec2.internal\" (UID: \"868b3db141f7bc43ba875f2543fbda86\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-240.ec2.internal" Jul 18 11:48:52.495355 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:52.495334 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/a5cb0d1fcbe931b511b49dbd31969429-config\") pod \"kube-apiserver-proxy-ip-10-0-136-240.ec2.internal\" (UID: \"a5cb0d1fcbe931b511b49dbd31969429\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-136-240.ec2.internal" Jul 18 11:48:52.595675 ip-10-0-136-240 kubenswrapper[2585]: E0718 11:48:52.595648 2585 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-240.ec2.internal\" not found" Jul 18 11:48:52.595771 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:52.595692 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/868b3db141f7bc43ba875f2543fbda86-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-136-240.ec2.internal\" (UID: \"868b3db141f7bc43ba875f2543fbda86\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-240.ec2.internal" Jul 18 11:48:52.595771 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:52.595748 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/868b3db141f7bc43ba875f2543fbda86-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-136-240.ec2.internal\" (UID: \"868b3db141f7bc43ba875f2543fbda86\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-240.ec2.internal" Jul 18 11:48:52.595839 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:52.595717 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/868b3db141f7bc43ba875f2543fbda86-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-136-240.ec2.internal\" (UID: \"868b3db141f7bc43ba875f2543fbda86\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-240.ec2.internal" Jul 18 11:48:52.595839 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:52.595786 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/868b3db141f7bc43ba875f2543fbda86-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-136-240.ec2.internal\" (UID: \"868b3db141f7bc43ba875f2543fbda86\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-240.ec2.internal" Jul 18 11:48:52.595839 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:52.595817 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/a5cb0d1fcbe931b511b49dbd31969429-config\") pod \"kube-apiserver-proxy-ip-10-0-136-240.ec2.internal\" (UID: \"a5cb0d1fcbe931b511b49dbd31969429\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-136-240.ec2.internal" Jul 18 11:48:52.595839 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:52.595832 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/a5cb0d1fcbe931b511b49dbd31969429-config\") pod \"kube-apiserver-proxy-ip-10-0-136-240.ec2.internal\" (UID: \"a5cb0d1fcbe931b511b49dbd31969429\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-136-240.ec2.internal" Jul 18 11:48:52.696326 ip-10-0-136-240 kubenswrapper[2585]: E0718 11:48:52.696260 2585 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-240.ec2.internal\" not found" Jul 18 11:48:52.773521 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:52.773497 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-240.ec2.internal" Jul 18 11:48:52.779412 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:52.779393 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-240.ec2.internal" Jul 18 11:48:52.797144 ip-10-0-136-240 kubenswrapper[2585]: E0718 11:48:52.797120 2585 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-240.ec2.internal\" not found" Jul 18 11:48:52.897587 ip-10-0-136-240 kubenswrapper[2585]: E0718 11:48:52.897544 2585 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-240.ec2.internal\" not found" Jul 18 11:48:52.997990 ip-10-0-136-240 kubenswrapper[2585]: E0718 11:48:52.997918 2585 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-240.ec2.internal\" not found" Jul 18 11:48:53.098325 ip-10-0-136-240 kubenswrapper[2585]: E0718 11:48:53.098295 2585 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-240.ec2.internal\" not found" Jul 18 11:48:53.126544 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:53.126523 2585 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Jul 18 11:48:53.134409 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:53.134389 2585 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Jul 18 11:48:53.134566 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:53.134538 2585 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 18 11:48:53.134566 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:53.134542 2585 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 18 11:48:53.173688 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:53.173671 2585 apiserver.go:52] "Watching apiserver" Jul 18 11:48:53.193113 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:53.193088 2585 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Jul 18 11:48:53.193510 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:53.193494 2585 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-240.ec2.internal" Jul 18 11:48:53.194749 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:53.194720 2585 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Jul 18 11:48:53.195080 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:53.195060 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-multus/multus-xlwqp","openshift-multus/network-metrics-daemon-hppk2","kube-system/konnectivity-agent-844xq","openshift-network-diagnostics/network-check-target-vd8wb","openshift-network-operator/iptables-alerter-vrm8v","openshift-ovn-kubernetes/ovnkube-node-wq4lg","kube-system/global-pull-secret-syncer-sp74l","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fjjlw","openshift-cluster-node-tuning-operator/tuned-c6jpl","openshift-dns/node-resolver-hv6g8","openshift-image-registry/node-ca-nzpfs","openshift-multus/multus-additional-cni-plugins-9xp94"] Jul 18 11:48:53.197511 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:53.197491 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-xlwqp" Jul 18 11:48:53.198420 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:53.198381 2585 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-17 11:43:53 +0000 UTC" deadline="2028-01-04 03:49:16.982210088 +0000 UTC" Jul 18 11:48:53.198420 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:53.198416 2585 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="12832h0m23.78379685s" Jul 18 11:48:53.200378 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:53.199932 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Jul 18 11:48:53.200378 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:53.199934 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Jul 18 11:48:53.200378 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:53.200164 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Jul 18 11:48:53.200378 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:53.200276 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-kdktk\"" Jul 18 11:48:53.200378 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:53.200303 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Jul 18 11:48:53.200378 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:53.200324 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hppk2" Jul 18 11:48:53.200687 ip-10-0-136-240 kubenswrapper[2585]: E0718 11:48:53.200401 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hppk2" podUID="4731170e-9d96-4544-a641-5fc8e1659d8e" Jul 18 11:48:53.202288 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:53.202272 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-844xq" Jul 18 11:48:53.204709 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:53.204511 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vd8wb" Jul 18 11:48:53.204709 ip-10-0-136-240 kubenswrapper[2585]: E0718 11:48:53.204573 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vd8wb" podUID="c1cdc202-852a-4bc3-a68c-8f5bf5baab20" Jul 18 11:48:53.204709 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:53.204582 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-748rq\"" Jul 18 11:48:53.204709 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:53.204590 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Jul 18 11:48:53.204981 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:53.204965 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Jul 18 11:48:53.206606 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:53.206582 2585 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 18 11:48:53.206691 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:53.206644 2585 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-240.ec2.internal" Jul 18 11:48:53.206734 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:53.206724 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-vrm8v" Jul 18 11:48:53.209060 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:53.209037 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-26bjq\"" Jul 18 11:48:53.209268 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:53.209245 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Jul 18 11:48:53.209268 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:53.209259 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Jul 18 11:48:53.209406 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:53.209330 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-wq4lg" Jul 18 11:48:53.209406 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:53.209263 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Jul 18 11:48:53.209841 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:53.209822 2585 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 18 11:48:53.211747 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:53.211434 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sp74l" Jul 18 11:48:53.211747 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:53.211501 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Jul 18 11:48:53.211747 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:53.211559 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Jul 18 11:48:53.211747 ip-10-0-136-240 kubenswrapper[2585]: E0718 11:48:53.211555 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sp74l" podUID="62e02fd7-1e2e-4f92-9989-08bb7b819cfa" Jul 18 11:48:53.211747 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:53.211684 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Jul 18 11:48:53.212326 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:53.212308 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Jul 18 11:48:53.212394 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:53.212333 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Jul 18 11:48:53.212394 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:53.212342 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Jul 18 11:48:53.212468 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:53.212401 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-5pnxg\"" Jul 18 11:48:53.213809 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:53.213796 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fjjlw" Jul 18 11:48:53.215981 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:53.215960 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Jul 18 11:48:53.216066 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:53.216005 2585 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 18 11:48:53.216122 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:53.216092 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Jul 18 11:48:53.216177 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:53.216120 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-c6jpl" Jul 18 11:48:53.216231 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:53.216206 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Jul 18 11:48:53.216411 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:53.216393 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-g7rnt\"" Jul 18 11:48:53.218395 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:53.218162 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Jul 18 11:48:53.218867 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:53.218649 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-hv6g8" Jul 18 11:48:53.218998 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:53.218978 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-g6gg9\"" Jul 18 11:48:53.219075 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:53.219007 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Jul 18 11:48:53.221200 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:53.221169 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-nzpfs" Jul 18 11:48:53.221422 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:53.221402 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Jul 18 11:48:53.222541 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:53.222524 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Jul 18 11:48:53.222630 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:53.222547 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-m2rxv\"" Jul 18 11:48:53.223457 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:53.223440 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Jul 18 11:48:53.223836 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:53.223812 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Jul 18 11:48:53.224121 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:53.224103 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/kube-apiserver-proxy-ip-10-0-136-240.ec2.internal"] Jul 18 11:48:53.224436 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:53.224418 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-240.ec2.internal"] Jul 18 11:48:53.224529 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:53.224434 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-9xp94" Jul 18 11:48:53.224529 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:53.224500 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-l6jcv\"" Jul 18 11:48:53.224786 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:53.224771 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Jul 18 11:48:53.226520 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:53.226501 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Jul 18 11:48:53.226702 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:53.226598 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Jul 18 11:48:53.226770 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:53.226720 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-l6wq9\"" Jul 18 11:48:53.230020 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:53.230005 2585 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-s8xk9" Jul 18 11:48:53.235620 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:53.235603 2585 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-s8xk9" Jul 18 11:48:53.295341 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:53.295323 2585 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Jul 18 11:48:53.298983 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:53.298962 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/760609c0-c9c8-44d2-87f4-7fd81b6eb7da-system-cni-dir\") pod \"multus-additional-cni-plugins-9xp94\" (UID: \"760609c0-c9c8-44d2-87f4-7fd81b6eb7da\") " pod="openshift-multus/multus-additional-cni-plugins-9xp94" Jul 18 11:48:53.299072 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:53.298996 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/99561f5b-25bc-4089-b635-e8e896bfe43d-etc-modprobe-d\") pod \"tuned-c6jpl\" (UID: \"99561f5b-25bc-4089-b635-e8e896bfe43d\") " pod="openshift-cluster-node-tuning-operator/tuned-c6jpl" Jul 18 11:48:53.299072 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:53.299023 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/99561f5b-25bc-4089-b635-e8e896bfe43d-etc-systemd\") pod \"tuned-c6jpl\" (UID: \"99561f5b-25bc-4089-b635-e8e896bfe43d\") " pod="openshift-cluster-node-tuning-operator/tuned-c6jpl" Jul 18 11:48:53.299072 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:53.299045 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/99561f5b-25bc-4089-b635-e8e896bfe43d-var-lib-kubelet\") pod \"tuned-c6jpl\" (UID: \"99561f5b-25bc-4089-b635-e8e896bfe43d\") " pod="openshift-cluster-node-tuning-operator/tuned-c6jpl" Jul 18 11:48:53.299221 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:53.299092 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/760609c0-c9c8-44d2-87f4-7fd81b6eb7da-cnibin\") pod \"multus-additional-cni-plugins-9xp94\" (UID: \"760609c0-c9c8-44d2-87f4-7fd81b6eb7da\") " pod="openshift-multus/multus-additional-cni-plugins-9xp94" Jul 18 11:48:53.299221 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:53.299127 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/3072709c-b577-46c4-9616-f0ca460c4196-multus-socket-dir-parent\") pod \"multus-xlwqp\" (UID: \"3072709c-b577-46c4-9616-f0ca460c4196\") " pod="openshift-multus/multus-xlwqp" Jul 18 11:48:53.299221 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:53.299155 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/247c97c8-982b-4176-ae10-afc396745360-var-lib-openvswitch\") pod \"ovnkube-node-wq4lg\" (UID: \"247c97c8-982b-4176-ae10-afc396745360\") " pod="openshift-ovn-kubernetes/ovnkube-node-wq4lg" Jul 18 11:48:53.299221 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:53.299176 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/247c97c8-982b-4176-ae10-afc396745360-host-cni-netd\") pod \"ovnkube-node-wq4lg\" (UID: \"247c97c8-982b-4176-ae10-afc396745360\") " pod="openshift-ovn-kubernetes/ovnkube-node-wq4lg" Jul 18 11:48:53.299221 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:53.299201 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/247c97c8-982b-4176-ae10-afc396745360-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-wq4lg\" (UID: \"247c97c8-982b-4176-ae10-afc396745360\") " pod="openshift-ovn-kubernetes/ovnkube-node-wq4lg" Jul 18 11:48:53.299408 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:53.299224 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/99561f5b-25bc-4089-b635-e8e896bfe43d-etc-kubernetes\") pod \"tuned-c6jpl\" (UID: \"99561f5b-25bc-4089-b635-e8e896bfe43d\") " pod="openshift-cluster-node-tuning-operator/tuned-c6jpl" Jul 18 11:48:53.299408 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:53.299265 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/99561f5b-25bc-4089-b635-e8e896bfe43d-etc-sysctl-conf\") pod \"tuned-c6jpl\" (UID: \"99561f5b-25bc-4089-b635-e8e896bfe43d\") " pod="openshift-cluster-node-tuning-operator/tuned-c6jpl" Jul 18 11:48:53.299408 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:53.299287 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d12e3a2e-a05d-4659-b74e-c30e765448bf-host\") pod \"node-ca-nzpfs\" (UID: \"d12e3a2e-a05d-4659-b74e-c30e765448bf\") " pod="openshift-image-registry/node-ca-nzpfs" Jul 18 11:48:53.299408 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:53.299308 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c11da53c-730b-4a02-b2f6-fbf3ee1a6cbd-host-slash\") pod \"iptables-alerter-vrm8v\" (UID: \"c11da53c-730b-4a02-b2f6-fbf3ee1a6cbd\") " pod="openshift-network-operator/iptables-alerter-vrm8v" Jul 18 11:48:53.299408 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:53.299330 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/760609c0-c9c8-44d2-87f4-7fd81b6eb7da-os-release\") pod \"multus-additional-cni-plugins-9xp94\" (UID: \"760609c0-c9c8-44d2-87f4-7fd81b6eb7da\") " pod="openshift-multus/multus-additional-cni-plugins-9xp94" Jul 18 11:48:53.299408 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:53.299355 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/760609c0-c9c8-44d2-87f4-7fd81b6eb7da-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-9xp94\" (UID: \"760609c0-c9c8-44d2-87f4-7fd81b6eb7da\") " pod="openshift-multus/multus-additional-cni-plugins-9xp94" Jul 18 11:48:53.299408 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:53.299380 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/247c97c8-982b-4176-ae10-afc396745360-env-overrides\") pod \"ovnkube-node-wq4lg\" (UID: \"247c97c8-982b-4176-ae10-afc396745360\") " pod="openshift-ovn-kubernetes/ovnkube-node-wq4lg" Jul 18 11:48:53.299408 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:53.299406 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/c11da53c-730b-4a02-b2f6-fbf3ee1a6cbd-iptables-alerter-script\") pod \"iptables-alerter-vrm8v\" (UID: \"c11da53c-730b-4a02-b2f6-fbf3ee1a6cbd\") " pod="openshift-network-operator/iptables-alerter-vrm8v" Jul 18 11:48:53.299801 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:53.299461 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/247c97c8-982b-4176-ae10-afc396745360-etc-openvswitch\") pod \"ovnkube-node-wq4lg\" (UID: \"247c97c8-982b-4176-ae10-afc396745360\") " pod="openshift-ovn-kubernetes/ovnkube-node-wq4lg" Jul 18 11:48:53.299801 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:53.299536 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ldhwg\" (UniqueName: \"kubernetes.io/projected/247c97c8-982b-4176-ae10-afc396745360-kube-api-access-ldhwg\") pod \"ovnkube-node-wq4lg\" (UID: \"247c97c8-982b-4176-ae10-afc396745360\") " pod="openshift-ovn-kubernetes/ovnkube-node-wq4lg" Jul 18 11:48:53.299801 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:53.299564 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-frt9b\" (UniqueName: \"kubernetes.io/projected/c1cdc202-852a-4bc3-a68c-8f5bf5baab20-kube-api-access-frt9b\") pod \"network-check-target-vd8wb\" (UID: \"c1cdc202-852a-4bc3-a68c-8f5bf5baab20\") " pod="openshift-network-diagnostics/network-check-target-vd8wb" Jul 18 11:48:53.299801 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:53.299589 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/9e7851c1-2896-4b77-9f84-cb29edf29b87-device-dir\") pod \"aws-ebs-csi-driver-node-fjjlw\" (UID: \"9e7851c1-2896-4b77-9f84-cb29edf29b87\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fjjlw" Jul 18 11:48:53.299801 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:53.299615 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/760609c0-c9c8-44d2-87f4-7fd81b6eb7da-cni-binary-copy\") pod \"multus-additional-cni-plugins-9xp94\" (UID: \"760609c0-c9c8-44d2-87f4-7fd81b6eb7da\") " pod="openshift-multus/multus-additional-cni-plugins-9xp94" Jul 18 11:48:53.299801 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:53.299653 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q2pxv\" (UniqueName: \"kubernetes.io/projected/3072709c-b577-46c4-9616-f0ca460c4196-kube-api-access-q2pxv\") pod \"multus-xlwqp\" (UID: \"3072709c-b577-46c4-9616-f0ca460c4196\") " pod="openshift-multus/multus-xlwqp" Jul 18 11:48:53.299801 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:53.299678 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/247c97c8-982b-4176-ae10-afc396745360-host-kubelet\") pod \"ovnkube-node-wq4lg\" (UID: \"247c97c8-982b-4176-ae10-afc396745360\") " pod="openshift-ovn-kubernetes/ovnkube-node-wq4lg" Jul 18 11:48:53.299801 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:53.299702 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/247c97c8-982b-4176-ae10-afc396745360-run-openvswitch\") pod \"ovnkube-node-wq4lg\" (UID: \"247c97c8-982b-4176-ae10-afc396745360\") " pod="openshift-ovn-kubernetes/ovnkube-node-wq4lg" Jul 18 11:48:53.299801 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:53.299724 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/62e02fd7-1e2e-4f92-9989-08bb7b819cfa-dbus\") pod \"global-pull-secret-syncer-sp74l\" (UID: \"62e02fd7-1e2e-4f92-9989-08bb7b819cfa\") " pod="kube-system/global-pull-secret-syncer-sp74l" Jul 18 11:48:53.299801 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:53.299748 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/99561f5b-25bc-4089-b635-e8e896bfe43d-etc-sysctl-d\") pod \"tuned-c6jpl\" (UID: \"99561f5b-25bc-4089-b635-e8e896bfe43d\") " pod="openshift-cluster-node-tuning-operator/tuned-c6jpl" Jul 18 11:48:53.299801 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:53.299768 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/99561f5b-25bc-4089-b635-e8e896bfe43d-host\") pod \"tuned-c6jpl\" (UID: \"99561f5b-25bc-4089-b635-e8e896bfe43d\") " pod="openshift-cluster-node-tuning-operator/tuned-c6jpl" Jul 18 11:48:53.299801 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:53.299794 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4731170e-9d96-4544-a641-5fc8e1659d8e-metrics-certs\") pod \"network-metrics-daemon-hppk2\" (UID: \"4731170e-9d96-4544-a641-5fc8e1659d8e\") " pod="openshift-multus/network-metrics-daemon-hppk2" Jul 18 11:48:53.300207 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:53.299835 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/247c97c8-982b-4176-ae10-afc396745360-run-systemd\") pod \"ovnkube-node-wq4lg\" (UID: \"247c97c8-982b-4176-ae10-afc396745360\") " pod="openshift-ovn-kubernetes/ovnkube-node-wq4lg" Jul 18 11:48:53.300207 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:53.299871 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/99561f5b-25bc-4089-b635-e8e896bfe43d-etc-sysconfig\") pod \"tuned-c6jpl\" (UID: \"99561f5b-25bc-4089-b635-e8e896bfe43d\") " pod="openshift-cluster-node-tuning-operator/tuned-c6jpl" Jul 18 11:48:53.300207 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:53.299896 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/99561f5b-25bc-4089-b635-e8e896bfe43d-sys\") pod \"tuned-c6jpl\" (UID: \"99561f5b-25bc-4089-b635-e8e896bfe43d\") " pod="openshift-cluster-node-tuning-operator/tuned-c6jpl" Jul 18 11:48:53.300207 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:53.299920 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/d12e3a2e-a05d-4659-b74e-c30e765448bf-serviceca\") pod \"node-ca-nzpfs\" (UID: \"d12e3a2e-a05d-4659-b74e-c30e765448bf\") " pod="openshift-image-registry/node-ca-nzpfs" Jul 18 11:48:53.300207 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:53.299945 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/9e7851c1-2896-4b77-9f84-cb29edf29b87-sys-fs\") pod \"aws-ebs-csi-driver-node-fjjlw\" (UID: \"9e7851c1-2896-4b77-9f84-cb29edf29b87\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fjjlw" Jul 18 11:48:53.300207 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:53.299969 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/97228923-3172-4691-927f-723ef88035cf-hosts-file\") pod \"node-resolver-hv6g8\" (UID: \"97228923-3172-4691-927f-723ef88035cf\") " pod="openshift-dns/node-resolver-hv6g8" Jul 18 11:48:53.300207 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:53.299991 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2z5vr\" (UniqueName: \"kubernetes.io/projected/97228923-3172-4691-927f-723ef88035cf-kube-api-access-2z5vr\") pod \"node-resolver-hv6g8\" (UID: \"97228923-3172-4691-927f-723ef88035cf\") " pod="openshift-dns/node-resolver-hv6g8" Jul 18 11:48:53.300207 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:53.300011 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3072709c-b577-46c4-9616-f0ca460c4196-system-cni-dir\") pod \"multus-xlwqp\" (UID: \"3072709c-b577-46c4-9616-f0ca460c4196\") " pod="openshift-multus/multus-xlwqp" Jul 18 11:48:53.300207 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:53.300029 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/3072709c-b577-46c4-9616-f0ca460c4196-cnibin\") pod \"multus-xlwqp\" (UID: \"3072709c-b577-46c4-9616-f0ca460c4196\") " pod="openshift-multus/multus-xlwqp" Jul 18 11:48:53.300207 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:53.300047 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mlxwx\" (UniqueName: \"kubernetes.io/projected/c11da53c-730b-4a02-b2f6-fbf3ee1a6cbd-kube-api-access-mlxwx\") pod \"iptables-alerter-vrm8v\" (UID: \"c11da53c-730b-4a02-b2f6-fbf3ee1a6cbd\") " pod="openshift-network-operator/iptables-alerter-vrm8v" Jul 18 11:48:53.300207 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:53.300060 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/3072709c-b577-46c4-9616-f0ca460c4196-os-release\") pod \"multus-xlwqp\" (UID: \"3072709c-b577-46c4-9616-f0ca460c4196\") " pod="openshift-multus/multus-xlwqp" Jul 18 11:48:53.300207 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:53.300103 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/3072709c-b577-46c4-9616-f0ca460c4196-host-var-lib-cni-multus\") pod \"multus-xlwqp\" (UID: \"3072709c-b577-46c4-9616-f0ca460c4196\") " pod="openshift-multus/multus-xlwqp" Jul 18 11:48:53.300207 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:53.300167 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3072709c-b577-46c4-9616-f0ca460c4196-host-var-lib-kubelet\") pod \"multus-xlwqp\" (UID: \"3072709c-b577-46c4-9616-f0ca460c4196\") " pod="openshift-multus/multus-xlwqp" Jul 18 11:48:53.300748 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:53.300220 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/3072709c-b577-46c4-9616-f0ca460c4196-multus-conf-dir\") pod \"multus-xlwqp\" (UID: \"3072709c-b577-46c4-9616-f0ca460c4196\") " pod="openshift-multus/multus-xlwqp" Jul 18 11:48:53.300748 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:53.300235 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/247c97c8-982b-4176-ae10-afc396745360-log-socket\") pod \"ovnkube-node-wq4lg\" (UID: \"247c97c8-982b-4176-ae10-afc396745360\") " pod="openshift-ovn-kubernetes/ovnkube-node-wq4lg" Jul 18 11:48:53.300748 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:53.300273 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/247c97c8-982b-4176-ae10-afc396745360-ovn-node-metrics-cert\") pod \"ovnkube-node-wq4lg\" (UID: \"247c97c8-982b-4176-ae10-afc396745360\") " pod="openshift-ovn-kubernetes/ovnkube-node-wq4lg" Jul 18 11:48:53.300748 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:53.300298 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/99561f5b-25bc-4089-b635-e8e896bfe43d-lib-modules\") pod \"tuned-c6jpl\" (UID: \"99561f5b-25bc-4089-b635-e8e896bfe43d\") " pod="openshift-cluster-node-tuning-operator/tuned-c6jpl" Jul 18 11:48:53.300748 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:53.300314 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/3072709c-b577-46c4-9616-f0ca460c4196-host-var-lib-cni-bin\") pod \"multus-xlwqp\" (UID: \"3072709c-b577-46c4-9616-f0ca460c4196\") " pod="openshift-multus/multus-xlwqp" Jul 18 11:48:53.300748 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:53.300335 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/247c97c8-982b-4176-ae10-afc396745360-host-run-netns\") pod \"ovnkube-node-wq4lg\" (UID: \"247c97c8-982b-4176-ae10-afc396745360\") " pod="openshift-ovn-kubernetes/ovnkube-node-wq4lg" Jul 18 11:48:53.300748 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:53.300354 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/247c97c8-982b-4176-ae10-afc396745360-host-run-ovn-kubernetes\") pod \"ovnkube-node-wq4lg\" (UID: \"247c97c8-982b-4176-ae10-afc396745360\") " pod="openshift-ovn-kubernetes/ovnkube-node-wq4lg" Jul 18 11:48:53.300748 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:53.300383 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k5dxp\" (UniqueName: \"kubernetes.io/projected/99561f5b-25bc-4089-b635-e8e896bfe43d-kube-api-access-k5dxp\") pod \"tuned-c6jpl\" (UID: \"99561f5b-25bc-4089-b635-e8e896bfe43d\") " pod="openshift-cluster-node-tuning-operator/tuned-c6jpl" Jul 18 11:48:53.300748 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:53.300424 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q7lfh\" (UniqueName: \"kubernetes.io/projected/d12e3a2e-a05d-4659-b74e-c30e765448bf-kube-api-access-q7lfh\") pod \"node-ca-nzpfs\" (UID: \"d12e3a2e-a05d-4659-b74e-c30e765448bf\") " pod="openshift-image-registry/node-ca-nzpfs" Jul 18 11:48:53.300748 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:53.300456 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/99561f5b-25bc-4089-b635-e8e896bfe43d-etc-tuned\") pod \"tuned-c6jpl\" (UID: \"99561f5b-25bc-4089-b635-e8e896bfe43d\") " pod="openshift-cluster-node-tuning-operator/tuned-c6jpl" Jul 18 11:48:53.300748 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:53.300499 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/99561f5b-25bc-4089-b635-e8e896bfe43d-tmp\") pod \"tuned-c6jpl\" (UID: \"99561f5b-25bc-4089-b635-e8e896bfe43d\") " pod="openshift-cluster-node-tuning-operator/tuned-c6jpl" Jul 18 11:48:53.300748 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:53.300526 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3072709c-b577-46c4-9616-f0ca460c4196-etc-kubernetes\") pod \"multus-xlwqp\" (UID: \"3072709c-b577-46c4-9616-f0ca460c4196\") " pod="openshift-multus/multus-xlwqp" Jul 18 11:48:53.300748 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:53.300552 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/247c97c8-982b-4176-ae10-afc396745360-run-ovn\") pod \"ovnkube-node-wq4lg\" (UID: \"247c97c8-982b-4176-ae10-afc396745360\") " pod="openshift-ovn-kubernetes/ovnkube-node-wq4lg" Jul 18 11:48:53.300748 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:53.300591 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/247c97c8-982b-4176-ae10-afc396745360-ovnkube-script-lib\") pod \"ovnkube-node-wq4lg\" (UID: \"247c97c8-982b-4176-ae10-afc396745360\") " pod="openshift-ovn-kubernetes/ovnkube-node-wq4lg" Jul 18 11:48:53.300748 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:53.300624 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/97228923-3172-4691-927f-723ef88035cf-tmp-dir\") pod \"node-resolver-hv6g8\" (UID: \"97228923-3172-4691-927f-723ef88035cf\") " pod="openshift-dns/node-resolver-hv6g8" Jul 18 11:48:53.300748 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:53.300644 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/a2e064ca-7d18-4e4a-952c-c00aebf5deb4-konnectivity-ca\") pod \"konnectivity-agent-844xq\" (UID: \"a2e064ca-7d18-4e4a-952c-c00aebf5deb4\") " pod="kube-system/konnectivity-agent-844xq" Jul 18 11:48:53.300748 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:53.300676 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/760609c0-c9c8-44d2-87f4-7fd81b6eb7da-tuning-conf-dir\") pod \"multus-additional-cni-plugins-9xp94\" (UID: \"760609c0-c9c8-44d2-87f4-7fd81b6eb7da\") " pod="openshift-multus/multus-additional-cni-plugins-9xp94" Jul 18 11:48:53.301330 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:53.300692 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w2cf9\" (UniqueName: \"kubernetes.io/projected/760609c0-c9c8-44d2-87f4-7fd81b6eb7da-kube-api-access-w2cf9\") pod \"multus-additional-cni-plugins-9xp94\" (UID: \"760609c0-c9c8-44d2-87f4-7fd81b6eb7da\") " pod="openshift-multus/multus-additional-cni-plugins-9xp94" Jul 18 11:48:53.301330 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:53.300711 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3072709c-b577-46c4-9616-f0ca460c4196-multus-cni-dir\") pod \"multus-xlwqp\" (UID: \"3072709c-b577-46c4-9616-f0ca460c4196\") " pod="openshift-multus/multus-xlwqp" Jul 18 11:48:53.301330 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:53.300745 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/3072709c-b577-46c4-9616-f0ca460c4196-cni-binary-copy\") pod \"multus-xlwqp\" (UID: \"3072709c-b577-46c4-9616-f0ca460c4196\") " pod="openshift-multus/multus-xlwqp" Jul 18 11:48:53.301330 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:53.300770 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/3072709c-b577-46c4-9616-f0ca460c4196-host-run-k8s-cni-cncf-io\") pod \"multus-xlwqp\" (UID: \"3072709c-b577-46c4-9616-f0ca460c4196\") " pod="openshift-multus/multus-xlwqp" Jul 18 11:48:53.301330 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:53.300791 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/3072709c-b577-46c4-9616-f0ca460c4196-host-run-netns\") pod \"multus-xlwqp\" (UID: \"3072709c-b577-46c4-9616-f0ca460c4196\") " pod="openshift-multus/multus-xlwqp" Jul 18 11:48:53.301330 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:53.300817 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tgvmr\" (UniqueName: \"kubernetes.io/projected/4731170e-9d96-4544-a641-5fc8e1659d8e-kube-api-access-tgvmr\") pod \"network-metrics-daemon-hppk2\" (UID: \"4731170e-9d96-4544-a641-5fc8e1659d8e\") " pod="openshift-multus/network-metrics-daemon-hppk2" Jul 18 11:48:53.301330 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:53.300874 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/9e7851c1-2896-4b77-9f84-cb29edf29b87-socket-dir\") pod \"aws-ebs-csi-driver-node-fjjlw\" (UID: \"9e7851c1-2896-4b77-9f84-cb29edf29b87\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fjjlw" Jul 18 11:48:53.301330 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:53.300901 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/9e7851c1-2896-4b77-9f84-cb29edf29b87-registration-dir\") pod \"aws-ebs-csi-driver-node-fjjlw\" (UID: \"9e7851c1-2896-4b77-9f84-cb29edf29b87\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fjjlw" Jul 18 11:48:53.301330 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:53.300918 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/9e7851c1-2896-4b77-9f84-cb29edf29b87-etc-selinux\") pod \"aws-ebs-csi-driver-node-fjjlw\" (UID: \"9e7851c1-2896-4b77-9f84-cb29edf29b87\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fjjlw" Jul 18 11:48:53.301330 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:53.300941 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/247c97c8-982b-4176-ae10-afc396745360-host-cni-bin\") pod \"ovnkube-node-wq4lg\" (UID: \"247c97c8-982b-4176-ae10-afc396745360\") " pod="openshift-ovn-kubernetes/ovnkube-node-wq4lg" Jul 18 11:48:53.301330 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:53.300961 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/9e7851c1-2896-4b77-9f84-cb29edf29b87-kubelet-dir\") pod \"aws-ebs-csi-driver-node-fjjlw\" (UID: \"9e7851c1-2896-4b77-9f84-cb29edf29b87\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fjjlw" Jul 18 11:48:53.301330 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:53.300983 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v4gpm\" (UniqueName: \"kubernetes.io/projected/9e7851c1-2896-4b77-9f84-cb29edf29b87-kube-api-access-v4gpm\") pod \"aws-ebs-csi-driver-node-fjjlw\" (UID: \"9e7851c1-2896-4b77-9f84-cb29edf29b87\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fjjlw" Jul 18 11:48:53.301330 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:53.301003 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/3072709c-b577-46c4-9616-f0ca460c4196-multus-daemon-config\") pod \"multus-xlwqp\" (UID: \"3072709c-b577-46c4-9616-f0ca460c4196\") " pod="openshift-multus/multus-xlwqp" Jul 18 11:48:53.301330 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:53.301019 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/247c97c8-982b-4176-ae10-afc396745360-ovnkube-config\") pod \"ovnkube-node-wq4lg\" (UID: \"247c97c8-982b-4176-ae10-afc396745360\") " pod="openshift-ovn-kubernetes/ovnkube-node-wq4lg" Jul 18 11:48:53.301330 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:53.301036 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/62e02fd7-1e2e-4f92-9989-08bb7b819cfa-kubelet-config\") pod \"global-pull-secret-syncer-sp74l\" (UID: \"62e02fd7-1e2e-4f92-9989-08bb7b819cfa\") " pod="kube-system/global-pull-secret-syncer-sp74l" Jul 18 11:48:53.301330 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:53.301049 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/3072709c-b577-46c4-9616-f0ca460c4196-hostroot\") pod \"multus-xlwqp\" (UID: \"3072709c-b577-46c4-9616-f0ca460c4196\") " pod="openshift-multus/multus-xlwqp" Jul 18 11:48:53.301911 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:53.301066 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/3072709c-b577-46c4-9616-f0ca460c4196-host-run-multus-certs\") pod \"multus-xlwqp\" (UID: \"3072709c-b577-46c4-9616-f0ca460c4196\") " pod="openshift-multus/multus-xlwqp" Jul 18 11:48:53.301911 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:53.301079 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/a2e064ca-7d18-4e4a-952c-c00aebf5deb4-agent-certs\") pod \"konnectivity-agent-844xq\" (UID: \"a2e064ca-7d18-4e4a-952c-c00aebf5deb4\") " pod="kube-system/konnectivity-agent-844xq" Jul 18 11:48:53.301911 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:53.301094 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/247c97c8-982b-4176-ae10-afc396745360-systemd-units\") pod \"ovnkube-node-wq4lg\" (UID: \"247c97c8-982b-4176-ae10-afc396745360\") " pod="openshift-ovn-kubernetes/ovnkube-node-wq4lg" Jul 18 11:48:53.301911 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:53.301108 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/247c97c8-982b-4176-ae10-afc396745360-host-slash\") pod \"ovnkube-node-wq4lg\" (UID: \"247c97c8-982b-4176-ae10-afc396745360\") " pod="openshift-ovn-kubernetes/ovnkube-node-wq4lg" Jul 18 11:48:53.301911 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:53.301128 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/99561f5b-25bc-4089-b635-e8e896bfe43d-run\") pod \"tuned-c6jpl\" (UID: \"99561f5b-25bc-4089-b635-e8e896bfe43d\") " pod="openshift-cluster-node-tuning-operator/tuned-c6jpl" Jul 18 11:48:53.301911 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:53.301153 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/62e02fd7-1e2e-4f92-9989-08bb7b819cfa-original-pull-secret\") pod \"global-pull-secret-syncer-sp74l\" (UID: \"62e02fd7-1e2e-4f92-9989-08bb7b819cfa\") " pod="kube-system/global-pull-secret-syncer-sp74l" Jul 18 11:48:53.301911 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:53.301169 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/760609c0-c9c8-44d2-87f4-7fd81b6eb7da-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-9xp94\" (UID: \"760609c0-c9c8-44d2-87f4-7fd81b6eb7da\") " pod="openshift-multus/multus-additional-cni-plugins-9xp94" Jul 18 11:48:53.301911 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:53.301204 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/247c97c8-982b-4176-ae10-afc396745360-node-log\") pod \"ovnkube-node-wq4lg\" (UID: \"247c97c8-982b-4176-ae10-afc396745360\") " pod="openshift-ovn-kubernetes/ovnkube-node-wq4lg" Jul 18 11:48:53.309573 ip-10-0-136-240 kubenswrapper[2585]: W0718 11:48:53.309544 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod868b3db141f7bc43ba875f2543fbda86.slice/crio-28460c552e61c76301f3fe05316775ce40fda496179da09ab3977941faa8bd80 WatchSource:0}: Error finding container 28460c552e61c76301f3fe05316775ce40fda496179da09ab3977941faa8bd80: Status 404 returned error can't find the container with id 28460c552e61c76301f3fe05316775ce40fda496179da09ab3977941faa8bd80 Jul 18 11:48:53.310459 ip-10-0-136-240 kubenswrapper[2585]: W0718 11:48:53.310438 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5cb0d1fcbe931b511b49dbd31969429.slice/crio-f80696a8aeee82b799e51af265fd28d3d39404d5e331aaaab9c4217450d26985 WatchSource:0}: Error finding container f80696a8aeee82b799e51af265fd28d3d39404d5e331aaaab9c4217450d26985: Status 404 returned error can't find the container with id f80696a8aeee82b799e51af265fd28d3d39404d5e331aaaab9c4217450d26985 Jul 18 11:48:53.314015 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:53.314001 2585 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 18 11:48:53.332394 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:53.332355 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-240.ec2.internal" event={"ID":"a5cb0d1fcbe931b511b49dbd31969429","Type":"ContainerStarted","Data":"f80696a8aeee82b799e51af265fd28d3d39404d5e331aaaab9c4217450d26985"} Jul 18 11:48:53.333369 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:53.333350 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-240.ec2.internal" event={"ID":"868b3db141f7bc43ba875f2543fbda86","Type":"ContainerStarted","Data":"28460c552e61c76301f3fe05316775ce40fda496179da09ab3977941faa8bd80"} Jul 18 11:48:53.401404 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:53.401374 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/3072709c-b577-46c4-9616-f0ca460c4196-cnibin\") pod \"multus-xlwqp\" (UID: \"3072709c-b577-46c4-9616-f0ca460c4196\") " pod="openshift-multus/multus-xlwqp" Jul 18 11:48:53.401564 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:53.401414 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mlxwx\" (UniqueName: \"kubernetes.io/projected/c11da53c-730b-4a02-b2f6-fbf3ee1a6cbd-kube-api-access-mlxwx\") pod \"iptables-alerter-vrm8v\" (UID: \"c11da53c-730b-4a02-b2f6-fbf3ee1a6cbd\") " pod="openshift-network-operator/iptables-alerter-vrm8v" Jul 18 11:48:53.401564 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:53.401442 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/3072709c-b577-46c4-9616-f0ca460c4196-os-release\") pod \"multus-xlwqp\" (UID: \"3072709c-b577-46c4-9616-f0ca460c4196\") " pod="openshift-multus/multus-xlwqp" Jul 18 11:48:53.401564 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:53.401464 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/3072709c-b577-46c4-9616-f0ca460c4196-host-var-lib-cni-multus\") pod \"multus-xlwqp\" (UID: \"3072709c-b577-46c4-9616-f0ca460c4196\") " pod="openshift-multus/multus-xlwqp" Jul 18 11:48:53.401564 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:53.401503 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3072709c-b577-46c4-9616-f0ca460c4196-host-var-lib-kubelet\") pod \"multus-xlwqp\" (UID: \"3072709c-b577-46c4-9616-f0ca460c4196\") " pod="openshift-multus/multus-xlwqp" Jul 18 11:48:53.401564 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:53.401506 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/3072709c-b577-46c4-9616-f0ca460c4196-cnibin\") pod \"multus-xlwqp\" (UID: \"3072709c-b577-46c4-9616-f0ca460c4196\") " pod="openshift-multus/multus-xlwqp" Jul 18 11:48:53.401564 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:53.401525 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/3072709c-b577-46c4-9616-f0ca460c4196-multus-conf-dir\") pod \"multus-xlwqp\" (UID: \"3072709c-b577-46c4-9616-f0ca460c4196\") " pod="openshift-multus/multus-xlwqp" Jul 18 11:48:53.401564 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:53.401549 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/247c97c8-982b-4176-ae10-afc396745360-log-socket\") pod \"ovnkube-node-wq4lg\" (UID: \"247c97c8-982b-4176-ae10-afc396745360\") " pod="openshift-ovn-kubernetes/ovnkube-node-wq4lg" Jul 18 11:48:53.401913 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:53.401571 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/247c97c8-982b-4176-ae10-afc396745360-ovn-node-metrics-cert\") pod \"ovnkube-node-wq4lg\" (UID: \"247c97c8-982b-4176-ae10-afc396745360\") " pod="openshift-ovn-kubernetes/ovnkube-node-wq4lg" Jul 18 11:48:53.401913 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:53.401579 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/3072709c-b577-46c4-9616-f0ca460c4196-host-var-lib-cni-multus\") pod \"multus-xlwqp\" (UID: \"3072709c-b577-46c4-9616-f0ca460c4196\") " pod="openshift-multus/multus-xlwqp" Jul 18 11:48:53.401913 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:53.401553 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/3072709c-b577-46c4-9616-f0ca460c4196-os-release\") pod \"multus-xlwqp\" (UID: \"3072709c-b577-46c4-9616-f0ca460c4196\") " pod="openshift-multus/multus-xlwqp" Jul 18 11:48:53.401913 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:53.401597 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3072709c-b577-46c4-9616-f0ca460c4196-host-var-lib-kubelet\") pod \"multus-xlwqp\" (UID: \"3072709c-b577-46c4-9616-f0ca460c4196\") " pod="openshift-multus/multus-xlwqp" Jul 18 11:48:53.401913 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:53.401593 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/99561f5b-25bc-4089-b635-e8e896bfe43d-lib-modules\") pod \"tuned-c6jpl\" (UID: \"99561f5b-25bc-4089-b635-e8e896bfe43d\") " pod="openshift-cluster-node-tuning-operator/tuned-c6jpl" Jul 18 11:48:53.401913 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:53.401647 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/3072709c-b577-46c4-9616-f0ca460c4196-host-var-lib-cni-bin\") pod \"multus-xlwqp\" (UID: \"3072709c-b577-46c4-9616-f0ca460c4196\") " pod="openshift-multus/multus-xlwqp" Jul 18 11:48:53.401913 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:53.401665 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/3072709c-b577-46c4-9616-f0ca460c4196-multus-conf-dir\") pod \"multus-xlwqp\" (UID: \"3072709c-b577-46c4-9616-f0ca460c4196\") " pod="openshift-multus/multus-xlwqp" Jul 18 11:48:53.401913 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:53.401678 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/247c97c8-982b-4176-ae10-afc396745360-host-run-netns\") pod \"ovnkube-node-wq4lg\" (UID: \"247c97c8-982b-4176-ae10-afc396745360\") " pod="openshift-ovn-kubernetes/ovnkube-node-wq4lg" Jul 18 11:48:53.401913 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:53.401701 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/3072709c-b577-46c4-9616-f0ca460c4196-host-var-lib-cni-bin\") pod \"multus-xlwqp\" (UID: \"3072709c-b577-46c4-9616-f0ca460c4196\") " pod="openshift-multus/multus-xlwqp" Jul 18 11:48:53.401913 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:53.401707 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/99561f5b-25bc-4089-b635-e8e896bfe43d-lib-modules\") pod \"tuned-c6jpl\" (UID: \"99561f5b-25bc-4089-b635-e8e896bfe43d\") " pod="openshift-cluster-node-tuning-operator/tuned-c6jpl" Jul 18 11:48:53.401913 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:53.401702 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/247c97c8-982b-4176-ae10-afc396745360-host-run-ovn-kubernetes\") pod \"ovnkube-node-wq4lg\" (UID: \"247c97c8-982b-4176-ae10-afc396745360\") " pod="openshift-ovn-kubernetes/ovnkube-node-wq4lg" Jul 18 11:48:53.401913 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:53.401744 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/247c97c8-982b-4176-ae10-afc396745360-host-run-ovn-kubernetes\") pod \"ovnkube-node-wq4lg\" (UID: \"247c97c8-982b-4176-ae10-afc396745360\") " pod="openshift-ovn-kubernetes/ovnkube-node-wq4lg" Jul 18 11:48:53.401913 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:53.401754 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k5dxp\" (UniqueName: \"kubernetes.io/projected/99561f5b-25bc-4089-b635-e8e896bfe43d-kube-api-access-k5dxp\") pod \"tuned-c6jpl\" (UID: \"99561f5b-25bc-4089-b635-e8e896bfe43d\") " pod="openshift-cluster-node-tuning-operator/tuned-c6jpl" Jul 18 11:48:53.401913 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:53.401759 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/247c97c8-982b-4176-ae10-afc396745360-host-run-netns\") pod \"ovnkube-node-wq4lg\" (UID: \"247c97c8-982b-4176-ae10-afc396745360\") " pod="openshift-ovn-kubernetes/ovnkube-node-wq4lg" Jul 18 11:48:53.401913 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:53.401777 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q7lfh\" (UniqueName: \"kubernetes.io/projected/d12e3a2e-a05d-4659-b74e-c30e765448bf-kube-api-access-q7lfh\") pod \"node-ca-nzpfs\" (UID: \"d12e3a2e-a05d-4659-b74e-c30e765448bf\") " pod="openshift-image-registry/node-ca-nzpfs" Jul 18 11:48:53.401913 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:53.401799 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/99561f5b-25bc-4089-b635-e8e896bfe43d-etc-tuned\") pod \"tuned-c6jpl\" (UID: \"99561f5b-25bc-4089-b635-e8e896bfe43d\") " pod="openshift-cluster-node-tuning-operator/tuned-c6jpl" Jul 18 11:48:53.401913 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:53.401821 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/99561f5b-25bc-4089-b635-e8e896bfe43d-tmp\") pod \"tuned-c6jpl\" (UID: \"99561f5b-25bc-4089-b635-e8e896bfe43d\") " pod="openshift-cluster-node-tuning-operator/tuned-c6jpl" Jul 18 11:48:53.401913 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:53.401849 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/247c97c8-982b-4176-ae10-afc396745360-log-socket\") pod \"ovnkube-node-wq4lg\" (UID: \"247c97c8-982b-4176-ae10-afc396745360\") " pod="openshift-ovn-kubernetes/ovnkube-node-wq4lg" Jul 18 11:48:53.402804 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:53.401947 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3072709c-b577-46c4-9616-f0ca460c4196-etc-kubernetes\") pod \"multus-xlwqp\" (UID: \"3072709c-b577-46c4-9616-f0ca460c4196\") " pod="openshift-multus/multus-xlwqp" Jul 18 11:48:53.402804 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:53.401970 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/247c97c8-982b-4176-ae10-afc396745360-run-ovn\") pod \"ovnkube-node-wq4lg\" (UID: \"247c97c8-982b-4176-ae10-afc396745360\") " pod="openshift-ovn-kubernetes/ovnkube-node-wq4lg" Jul 18 11:48:53.402804 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:53.402015 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3072709c-b577-46c4-9616-f0ca460c4196-etc-kubernetes\") pod \"multus-xlwqp\" (UID: \"3072709c-b577-46c4-9616-f0ca460c4196\") " pod="openshift-multus/multus-xlwqp" Jul 18 11:48:53.402804 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:53.402019 2585 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Jul 18 11:48:53.402804 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:53.402038 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/247c97c8-982b-4176-ae10-afc396745360-ovnkube-script-lib\") pod \"ovnkube-node-wq4lg\" (UID: \"247c97c8-982b-4176-ae10-afc396745360\") " pod="openshift-ovn-kubernetes/ovnkube-node-wq4lg" Jul 18 11:48:53.402804 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:53.402064 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/97228923-3172-4691-927f-723ef88035cf-tmp-dir\") pod \"node-resolver-hv6g8\" (UID: \"97228923-3172-4691-927f-723ef88035cf\") " pod="openshift-dns/node-resolver-hv6g8" Jul 18 11:48:53.402804 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:53.402066 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/247c97c8-982b-4176-ae10-afc396745360-run-ovn\") pod \"ovnkube-node-wq4lg\" (UID: \"247c97c8-982b-4176-ae10-afc396745360\") " pod="openshift-ovn-kubernetes/ovnkube-node-wq4lg" Jul 18 11:48:53.402804 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:53.402085 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/a2e064ca-7d18-4e4a-952c-c00aebf5deb4-konnectivity-ca\") pod \"konnectivity-agent-844xq\" (UID: \"a2e064ca-7d18-4e4a-952c-c00aebf5deb4\") " pod="kube-system/konnectivity-agent-844xq" Jul 18 11:48:53.402804 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:53.402108 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/760609c0-c9c8-44d2-87f4-7fd81b6eb7da-tuning-conf-dir\") pod \"multus-additional-cni-plugins-9xp94\" (UID: \"760609c0-c9c8-44d2-87f4-7fd81b6eb7da\") " pod="openshift-multus/multus-additional-cni-plugins-9xp94" Jul 18 11:48:53.402804 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:53.402131 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w2cf9\" (UniqueName: \"kubernetes.io/projected/760609c0-c9c8-44d2-87f4-7fd81b6eb7da-kube-api-access-w2cf9\") pod \"multus-additional-cni-plugins-9xp94\" (UID: \"760609c0-c9c8-44d2-87f4-7fd81b6eb7da\") " pod="openshift-multus/multus-additional-cni-plugins-9xp94" Jul 18 11:48:53.402804 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:53.402158 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3072709c-b577-46c4-9616-f0ca460c4196-multus-cni-dir\") pod \"multus-xlwqp\" (UID: \"3072709c-b577-46c4-9616-f0ca460c4196\") " pod="openshift-multus/multus-xlwqp" Jul 18 11:48:53.402804 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:53.402189 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/3072709c-b577-46c4-9616-f0ca460c4196-cni-binary-copy\") pod \"multus-xlwqp\" (UID: \"3072709c-b577-46c4-9616-f0ca460c4196\") " pod="openshift-multus/multus-xlwqp" Jul 18 11:48:53.402804 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:53.402211 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/3072709c-b577-46c4-9616-f0ca460c4196-host-run-k8s-cni-cncf-io\") pod \"multus-xlwqp\" (UID: \"3072709c-b577-46c4-9616-f0ca460c4196\") " pod="openshift-multus/multus-xlwqp" Jul 18 11:48:53.402804 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:53.402235 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/3072709c-b577-46c4-9616-f0ca460c4196-host-run-netns\") pod \"multus-xlwqp\" (UID: \"3072709c-b577-46c4-9616-f0ca460c4196\") " pod="openshift-multus/multus-xlwqp" Jul 18 11:48:53.402804 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:53.402255 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tgvmr\" (UniqueName: \"kubernetes.io/projected/4731170e-9d96-4544-a641-5fc8e1659d8e-kube-api-access-tgvmr\") pod \"network-metrics-daemon-hppk2\" (UID: \"4731170e-9d96-4544-a641-5fc8e1659d8e\") " pod="openshift-multus/network-metrics-daemon-hppk2" Jul 18 11:48:53.402804 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:53.402279 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/9e7851c1-2896-4b77-9f84-cb29edf29b87-socket-dir\") pod \"aws-ebs-csi-driver-node-fjjlw\" (UID: \"9e7851c1-2896-4b77-9f84-cb29edf29b87\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fjjlw" Jul 18 11:48:53.402804 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:53.402304 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/9e7851c1-2896-4b77-9f84-cb29edf29b87-registration-dir\") pod \"aws-ebs-csi-driver-node-fjjlw\" (UID: \"9e7851c1-2896-4b77-9f84-cb29edf29b87\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fjjlw" Jul 18 11:48:53.402804 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:53.402329 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/9e7851c1-2896-4b77-9f84-cb29edf29b87-etc-selinux\") pod \"aws-ebs-csi-driver-node-fjjlw\" (UID: \"9e7851c1-2896-4b77-9f84-cb29edf29b87\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fjjlw" Jul 18 11:48:53.403744 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:53.402337 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/760609c0-c9c8-44d2-87f4-7fd81b6eb7da-tuning-conf-dir\") pod \"multus-additional-cni-plugins-9xp94\" (UID: \"760609c0-c9c8-44d2-87f4-7fd81b6eb7da\") " pod="openshift-multus/multus-additional-cni-plugins-9xp94" Jul 18 11:48:53.403744 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:53.402351 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/247c97c8-982b-4176-ae10-afc396745360-host-cni-bin\") pod \"ovnkube-node-wq4lg\" (UID: \"247c97c8-982b-4176-ae10-afc396745360\") " pod="openshift-ovn-kubernetes/ovnkube-node-wq4lg" Jul 18 11:48:53.403744 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:53.402389 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/9e7851c1-2896-4b77-9f84-cb29edf29b87-kubelet-dir\") pod \"aws-ebs-csi-driver-node-fjjlw\" (UID: \"9e7851c1-2896-4b77-9f84-cb29edf29b87\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fjjlw" Jul 18 11:48:53.403744 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:53.402401 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/97228923-3172-4691-927f-723ef88035cf-tmp-dir\") pod \"node-resolver-hv6g8\" (UID: \"97228923-3172-4691-927f-723ef88035cf\") " pod="openshift-dns/node-resolver-hv6g8" Jul 18 11:48:53.403744 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:53.402421 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v4gpm\" (UniqueName: \"kubernetes.io/projected/9e7851c1-2896-4b77-9f84-cb29edf29b87-kube-api-access-v4gpm\") pod \"aws-ebs-csi-driver-node-fjjlw\" (UID: \"9e7851c1-2896-4b77-9f84-cb29edf29b87\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fjjlw" Jul 18 11:48:53.403744 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:53.402450 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/3072709c-b577-46c4-9616-f0ca460c4196-multus-daemon-config\") pod \"multus-xlwqp\" (UID: \"3072709c-b577-46c4-9616-f0ca460c4196\") " pod="openshift-multus/multus-xlwqp" Jul 18 11:48:53.403744 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:53.402490 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/247c97c8-982b-4176-ae10-afc396745360-ovnkube-config\") pod \"ovnkube-node-wq4lg\" (UID: \"247c97c8-982b-4176-ae10-afc396745360\") " pod="openshift-ovn-kubernetes/ovnkube-node-wq4lg" Jul 18 11:48:53.403744 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:53.402514 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/62e02fd7-1e2e-4f92-9989-08bb7b819cfa-kubelet-config\") pod \"global-pull-secret-syncer-sp74l\" (UID: \"62e02fd7-1e2e-4f92-9989-08bb7b819cfa\") " pod="kube-system/global-pull-secret-syncer-sp74l" Jul 18 11:48:53.403744 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:53.402517 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3072709c-b577-46c4-9616-f0ca460c4196-multus-cni-dir\") pod \"multus-xlwqp\" (UID: \"3072709c-b577-46c4-9616-f0ca460c4196\") " pod="openshift-multus/multus-xlwqp" Jul 18 11:48:53.403744 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:53.402544 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/3072709c-b577-46c4-9616-f0ca460c4196-hostroot\") pod \"multus-xlwqp\" (UID: \"3072709c-b577-46c4-9616-f0ca460c4196\") " pod="openshift-multus/multus-xlwqp" Jul 18 11:48:53.403744 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:53.402568 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/3072709c-b577-46c4-9616-f0ca460c4196-host-run-multus-certs\") pod \"multus-xlwqp\" (UID: \"3072709c-b577-46c4-9616-f0ca460c4196\") " pod="openshift-multus/multus-xlwqp" Jul 18 11:48:53.403744 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:53.402591 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/a2e064ca-7d18-4e4a-952c-c00aebf5deb4-agent-certs\") pod \"konnectivity-agent-844xq\" (UID: \"a2e064ca-7d18-4e4a-952c-c00aebf5deb4\") " pod="kube-system/konnectivity-agent-844xq" Jul 18 11:48:53.403744 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:53.402615 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/247c97c8-982b-4176-ae10-afc396745360-systemd-units\") pod \"ovnkube-node-wq4lg\" (UID: \"247c97c8-982b-4176-ae10-afc396745360\") " pod="openshift-ovn-kubernetes/ovnkube-node-wq4lg" Jul 18 11:48:53.403744 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:53.402676 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/247c97c8-982b-4176-ae10-afc396745360-host-slash\") pod \"ovnkube-node-wq4lg\" (UID: \"247c97c8-982b-4176-ae10-afc396745360\") " pod="openshift-ovn-kubernetes/ovnkube-node-wq4lg" Jul 18 11:48:53.403744 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:53.402699 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/99561f5b-25bc-4089-b635-e8e896bfe43d-run\") pod \"tuned-c6jpl\" (UID: \"99561f5b-25bc-4089-b635-e8e896bfe43d\") " pod="openshift-cluster-node-tuning-operator/tuned-c6jpl" Jul 18 11:48:53.403744 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:53.402728 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/62e02fd7-1e2e-4f92-9989-08bb7b819cfa-original-pull-secret\") pod \"global-pull-secret-syncer-sp74l\" (UID: \"62e02fd7-1e2e-4f92-9989-08bb7b819cfa\") " pod="kube-system/global-pull-secret-syncer-sp74l" Jul 18 11:48:53.403744 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:53.402753 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/a2e064ca-7d18-4e4a-952c-c00aebf5deb4-konnectivity-ca\") pod \"konnectivity-agent-844xq\" (UID: \"a2e064ca-7d18-4e4a-952c-c00aebf5deb4\") " pod="kube-system/konnectivity-agent-844xq" Jul 18 11:48:53.404842 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:53.402758 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/760609c0-c9c8-44d2-87f4-7fd81b6eb7da-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-9xp94\" (UID: \"760609c0-c9c8-44d2-87f4-7fd81b6eb7da\") " pod="openshift-multus/multus-additional-cni-plugins-9xp94" Jul 18 11:48:53.404842 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:53.402782 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/247c97c8-982b-4176-ae10-afc396745360-node-log\") pod \"ovnkube-node-wq4lg\" (UID: \"247c97c8-982b-4176-ae10-afc396745360\") " pod="openshift-ovn-kubernetes/ovnkube-node-wq4lg" Jul 18 11:48:53.404842 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:53.402805 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/760609c0-c9c8-44d2-87f4-7fd81b6eb7da-system-cni-dir\") pod \"multus-additional-cni-plugins-9xp94\" (UID: \"760609c0-c9c8-44d2-87f4-7fd81b6eb7da\") " pod="openshift-multus/multus-additional-cni-plugins-9xp94" Jul 18 11:48:53.404842 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:53.402830 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/99561f5b-25bc-4089-b635-e8e896bfe43d-etc-modprobe-d\") pod \"tuned-c6jpl\" (UID: \"99561f5b-25bc-4089-b635-e8e896bfe43d\") " pod="openshift-cluster-node-tuning-operator/tuned-c6jpl" Jul 18 11:48:53.404842 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:53.402854 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/99561f5b-25bc-4089-b635-e8e896bfe43d-etc-systemd\") pod \"tuned-c6jpl\" (UID: \"99561f5b-25bc-4089-b635-e8e896bfe43d\") " pod="openshift-cluster-node-tuning-operator/tuned-c6jpl" Jul 18 11:48:53.404842 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:53.402854 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/247c97c8-982b-4176-ae10-afc396745360-systemd-units\") pod \"ovnkube-node-wq4lg\" (UID: \"247c97c8-982b-4176-ae10-afc396745360\") " pod="openshift-ovn-kubernetes/ovnkube-node-wq4lg" Jul 18 11:48:53.404842 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:53.402878 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/99561f5b-25bc-4089-b635-e8e896bfe43d-var-lib-kubelet\") pod \"tuned-c6jpl\" (UID: \"99561f5b-25bc-4089-b635-e8e896bfe43d\") " pod="openshift-cluster-node-tuning-operator/tuned-c6jpl" Jul 18 11:48:53.404842 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:53.402899 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/3072709c-b577-46c4-9616-f0ca460c4196-host-run-k8s-cni-cncf-io\") pod \"multus-xlwqp\" (UID: \"3072709c-b577-46c4-9616-f0ca460c4196\") " pod="openshift-multus/multus-xlwqp" Jul 18 11:48:53.404842 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:53.402905 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/760609c0-c9c8-44d2-87f4-7fd81b6eb7da-cnibin\") pod \"multus-additional-cni-plugins-9xp94\" (UID: \"760609c0-c9c8-44d2-87f4-7fd81b6eb7da\") " pod="openshift-multus/multus-additional-cni-plugins-9xp94" Jul 18 11:48:53.404842 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:53.402941 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/3072709c-b577-46c4-9616-f0ca460c4196-multus-socket-dir-parent\") pod \"multus-xlwqp\" (UID: \"3072709c-b577-46c4-9616-f0ca460c4196\") " pod="openshift-multus/multus-xlwqp" Jul 18 11:48:53.404842 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:53.402950 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/760609c0-c9c8-44d2-87f4-7fd81b6eb7da-cnibin\") pod \"multus-additional-cni-plugins-9xp94\" (UID: \"760609c0-c9c8-44d2-87f4-7fd81b6eb7da\") " pod="openshift-multus/multus-additional-cni-plugins-9xp94" Jul 18 11:48:53.404842 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:53.402972 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/247c97c8-982b-4176-ae10-afc396745360-var-lib-openvswitch\") pod \"ovnkube-node-wq4lg\" (UID: \"247c97c8-982b-4176-ae10-afc396745360\") " pod="openshift-ovn-kubernetes/ovnkube-node-wq4lg" Jul 18 11:48:53.404842 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:53.402997 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/247c97c8-982b-4176-ae10-afc396745360-host-cni-netd\") pod \"ovnkube-node-wq4lg\" (UID: \"247c97c8-982b-4176-ae10-afc396745360\") " pod="openshift-ovn-kubernetes/ovnkube-node-wq4lg" Jul 18 11:48:53.404842 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:53.403023 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/247c97c8-982b-4176-ae10-afc396745360-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-wq4lg\" (UID: \"247c97c8-982b-4176-ae10-afc396745360\") " pod="openshift-ovn-kubernetes/ovnkube-node-wq4lg" Jul 18 11:48:53.404842 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:53.403039 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/99561f5b-25bc-4089-b635-e8e896bfe43d-run\") pod \"tuned-c6jpl\" (UID: \"99561f5b-25bc-4089-b635-e8e896bfe43d\") " pod="openshift-cluster-node-tuning-operator/tuned-c6jpl" Jul 18 11:48:53.404842 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:53.403051 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/99561f5b-25bc-4089-b635-e8e896bfe43d-etc-kubernetes\") pod \"tuned-c6jpl\" (UID: \"99561f5b-25bc-4089-b635-e8e896bfe43d\") " pod="openshift-cluster-node-tuning-operator/tuned-c6jpl" Jul 18 11:48:53.404842 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:53.403093 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/3072709c-b577-46c4-9616-f0ca460c4196-cni-binary-copy\") pod \"multus-xlwqp\" (UID: \"3072709c-b577-46c4-9616-f0ca460c4196\") " pod="openshift-multus/multus-xlwqp" Jul 18 11:48:53.405796 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:53.403099 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/99561f5b-25bc-4089-b635-e8e896bfe43d-etc-kubernetes\") pod \"tuned-c6jpl\" (UID: \"99561f5b-25bc-4089-b635-e8e896bfe43d\") " pod="openshift-cluster-node-tuning-operator/tuned-c6jpl" Jul 18 11:48:53.405796 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:53.403098 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/99561f5b-25bc-4089-b635-e8e896bfe43d-etc-sysctl-conf\") pod \"tuned-c6jpl\" (UID: \"99561f5b-25bc-4089-b635-e8e896bfe43d\") " pod="openshift-cluster-node-tuning-operator/tuned-c6jpl" Jul 18 11:48:53.405796 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:53.403107 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/247c97c8-982b-4176-ae10-afc396745360-node-log\") pod \"ovnkube-node-wq4lg\" (UID: \"247c97c8-982b-4176-ae10-afc396745360\") " pod="openshift-ovn-kubernetes/ovnkube-node-wq4lg" Jul 18 11:48:53.405796 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:53.403098 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/3072709c-b577-46c4-9616-f0ca460c4196-multus-daemon-config\") pod \"multus-xlwqp\" (UID: \"3072709c-b577-46c4-9616-f0ca460c4196\") " pod="openshift-multus/multus-xlwqp" Jul 18 11:48:53.405796 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:53.402859 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/9e7851c1-2896-4b77-9f84-cb29edf29b87-kubelet-dir\") pod \"aws-ebs-csi-driver-node-fjjlw\" (UID: \"9e7851c1-2896-4b77-9f84-cb29edf29b87\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fjjlw" Jul 18 11:48:53.405796 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:53.403143 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/3072709c-b577-46c4-9616-f0ca460c4196-host-run-netns\") pod \"multus-xlwqp\" (UID: \"3072709c-b577-46c4-9616-f0ca460c4196\") " pod="openshift-multus/multus-xlwqp" Jul 18 11:48:53.405796 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:53.403148 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/3072709c-b577-46c4-9616-f0ca460c4196-multus-socket-dir-parent\") pod \"multus-xlwqp\" (UID: \"3072709c-b577-46c4-9616-f0ca460c4196\") " pod="openshift-multus/multus-xlwqp" Jul 18 11:48:53.405796 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:53.403178 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/9e7851c1-2896-4b77-9f84-cb29edf29b87-registration-dir\") pod \"aws-ebs-csi-driver-node-fjjlw\" (UID: \"9e7851c1-2896-4b77-9f84-cb29edf29b87\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fjjlw" Jul 18 11:48:53.405796 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:53.403191 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/62e02fd7-1e2e-4f92-9989-08bb7b819cfa-kubelet-config\") pod \"global-pull-secret-syncer-sp74l\" (UID: \"62e02fd7-1e2e-4f92-9989-08bb7b819cfa\") " pod="kube-system/global-pull-secret-syncer-sp74l" Jul 18 11:48:53.405796 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:53.403221 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/9e7851c1-2896-4b77-9f84-cb29edf29b87-etc-selinux\") pod \"aws-ebs-csi-driver-node-fjjlw\" (UID: \"9e7851c1-2896-4b77-9f84-cb29edf29b87\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fjjlw" Jul 18 11:48:53.405796 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:53.403260 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/9e7851c1-2896-4b77-9f84-cb29edf29b87-socket-dir\") pod \"aws-ebs-csi-driver-node-fjjlw\" (UID: \"9e7851c1-2896-4b77-9f84-cb29edf29b87\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fjjlw" Jul 18 11:48:53.405796 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:53.403269 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/760609c0-c9c8-44d2-87f4-7fd81b6eb7da-system-cni-dir\") pod \"multus-additional-cni-plugins-9xp94\" (UID: \"760609c0-c9c8-44d2-87f4-7fd81b6eb7da\") " pod="openshift-multus/multus-additional-cni-plugins-9xp94" Jul 18 11:48:53.405796 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:53.402392 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/247c97c8-982b-4176-ae10-afc396745360-host-cni-bin\") pod \"ovnkube-node-wq4lg\" (UID: \"247c97c8-982b-4176-ae10-afc396745360\") " pod="openshift-ovn-kubernetes/ovnkube-node-wq4lg" Jul 18 11:48:53.405796 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:53.403303 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/3072709c-b577-46c4-9616-f0ca460c4196-host-run-multus-certs\") pod \"multus-xlwqp\" (UID: \"3072709c-b577-46c4-9616-f0ca460c4196\") " pod="openshift-multus/multus-xlwqp" Jul 18 11:48:53.405796 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:53.403333 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/247c97c8-982b-4176-ae10-afc396745360-var-lib-openvswitch\") pod \"ovnkube-node-wq4lg\" (UID: \"247c97c8-982b-4176-ae10-afc396745360\") " pod="openshift-ovn-kubernetes/ovnkube-node-wq4lg" Jul 18 11:48:53.405796 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:53.403361 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/99561f5b-25bc-4089-b635-e8e896bfe43d-etc-modprobe-d\") pod \"tuned-c6jpl\" (UID: \"99561f5b-25bc-4089-b635-e8e896bfe43d\") " pod="openshift-cluster-node-tuning-operator/tuned-c6jpl" Jul 18 11:48:53.405796 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:53.403372 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/247c97c8-982b-4176-ae10-afc396745360-host-cni-netd\") pod \"ovnkube-node-wq4lg\" (UID: \"247c97c8-982b-4176-ae10-afc396745360\") " pod="openshift-ovn-kubernetes/ovnkube-node-wq4lg" Jul 18 11:48:53.406438 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:53.402998 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/247c97c8-982b-4176-ae10-afc396745360-host-slash\") pod \"ovnkube-node-wq4lg\" (UID: \"247c97c8-982b-4176-ae10-afc396745360\") " pod="openshift-ovn-kubernetes/ovnkube-node-wq4lg" Jul 18 11:48:53.406438 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:53.403409 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/99561f5b-25bc-4089-b635-e8e896bfe43d-etc-systemd\") pod \"tuned-c6jpl\" (UID: \"99561f5b-25bc-4089-b635-e8e896bfe43d\") " pod="openshift-cluster-node-tuning-operator/tuned-c6jpl" Jul 18 11:48:53.406438 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:53.403416 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/247c97c8-982b-4176-ae10-afc396745360-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-wq4lg\" (UID: \"247c97c8-982b-4176-ae10-afc396745360\") " pod="openshift-ovn-kubernetes/ovnkube-node-wq4lg" Jul 18 11:48:53.406438 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:53.403455 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/99561f5b-25bc-4089-b635-e8e896bfe43d-var-lib-kubelet\") pod \"tuned-c6jpl\" (UID: \"99561f5b-25bc-4089-b635-e8e896bfe43d\") " pod="openshift-cluster-node-tuning-operator/tuned-c6jpl" Jul 18 11:48:53.406438 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:53.403467 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d12e3a2e-a05d-4659-b74e-c30e765448bf-host\") pod \"node-ca-nzpfs\" (UID: \"d12e3a2e-a05d-4659-b74e-c30e765448bf\") " pod="openshift-image-registry/node-ca-nzpfs" Jul 18 11:48:53.406438 ip-10-0-136-240 kubenswrapper[2585]: E0718 11:48:53.403504 2585 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 18 11:48:53.406438 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:53.403522 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c11da53c-730b-4a02-b2f6-fbf3ee1a6cbd-host-slash\") pod \"iptables-alerter-vrm8v\" (UID: \"c11da53c-730b-4a02-b2f6-fbf3ee1a6cbd\") " pod="openshift-network-operator/iptables-alerter-vrm8v" Jul 18 11:48:53.406438 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:53.403555 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/760609c0-c9c8-44d2-87f4-7fd81b6eb7da-os-release\") pod \"multus-additional-cni-plugins-9xp94\" (UID: \"760609c0-c9c8-44d2-87f4-7fd81b6eb7da\") " pod="openshift-multus/multus-additional-cni-plugins-9xp94" Jul 18 11:48:53.406438 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:53.403581 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/760609c0-c9c8-44d2-87f4-7fd81b6eb7da-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-9xp94\" (UID: \"760609c0-c9c8-44d2-87f4-7fd81b6eb7da\") " pod="openshift-multus/multus-additional-cni-plugins-9xp94" Jul 18 11:48:53.406438 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:53.403588 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/99561f5b-25bc-4089-b635-e8e896bfe43d-etc-sysctl-conf\") pod \"tuned-c6jpl\" (UID: \"99561f5b-25bc-4089-b635-e8e896bfe43d\") " pod="openshift-cluster-node-tuning-operator/tuned-c6jpl" Jul 18 11:48:53.406438 ip-10-0-136-240 kubenswrapper[2585]: E0718 11:48:53.403618 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/62e02fd7-1e2e-4f92-9989-08bb7b819cfa-original-pull-secret podName:62e02fd7-1e2e-4f92-9989-08bb7b819cfa nodeName:}" failed. No retries permitted until 2026-07-18 11:48:53.903597435 +0000 UTC m=+2.116878294 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/62e02fd7-1e2e-4f92-9989-08bb7b819cfa-original-pull-secret") pod "global-pull-secret-syncer-sp74l" (UID: "62e02fd7-1e2e-4f92-9989-08bb7b819cfa") : object "kube-system"/"original-pull-secret" not registered Jul 18 11:48:53.406438 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:53.403639 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/247c97c8-982b-4176-ae10-afc396745360-env-overrides\") pod \"ovnkube-node-wq4lg\" (UID: \"247c97c8-982b-4176-ae10-afc396745360\") " pod="openshift-ovn-kubernetes/ovnkube-node-wq4lg" Jul 18 11:48:53.406438 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:53.403623 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/247c97c8-982b-4176-ae10-afc396745360-ovnkube-config\") pod \"ovnkube-node-wq4lg\" (UID: \"247c97c8-982b-4176-ae10-afc396745360\") " pod="openshift-ovn-kubernetes/ovnkube-node-wq4lg" Jul 18 11:48:53.406438 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:53.403668 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/c11da53c-730b-4a02-b2f6-fbf3ee1a6cbd-iptables-alerter-script\") pod \"iptables-alerter-vrm8v\" (UID: \"c11da53c-730b-4a02-b2f6-fbf3ee1a6cbd\") " pod="openshift-network-operator/iptables-alerter-vrm8v" Jul 18 11:48:53.406438 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:53.403695 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/247c97c8-982b-4176-ae10-afc396745360-etc-openvswitch\") pod \"ovnkube-node-wq4lg\" (UID: \"247c97c8-982b-4176-ae10-afc396745360\") " pod="openshift-ovn-kubernetes/ovnkube-node-wq4lg" Jul 18 11:48:53.406438 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:53.403723 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ldhwg\" (UniqueName: \"kubernetes.io/projected/247c97c8-982b-4176-ae10-afc396745360-kube-api-access-ldhwg\") pod \"ovnkube-node-wq4lg\" (UID: \"247c97c8-982b-4176-ae10-afc396745360\") " pod="openshift-ovn-kubernetes/ovnkube-node-wq4lg" Jul 18 11:48:53.406438 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:53.403749 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-frt9b\" (UniqueName: \"kubernetes.io/projected/c1cdc202-852a-4bc3-a68c-8f5bf5baab20-kube-api-access-frt9b\") pod \"network-check-target-vd8wb\" (UID: \"c1cdc202-852a-4bc3-a68c-8f5bf5baab20\") " pod="openshift-network-diagnostics/network-check-target-vd8wb" Jul 18 11:48:53.406995 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:53.403774 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/9e7851c1-2896-4b77-9f84-cb29edf29b87-device-dir\") pod \"aws-ebs-csi-driver-node-fjjlw\" (UID: \"9e7851c1-2896-4b77-9f84-cb29edf29b87\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fjjlw" Jul 18 11:48:53.406995 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:53.403798 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/760609c0-c9c8-44d2-87f4-7fd81b6eb7da-cni-binary-copy\") pod \"multus-additional-cni-plugins-9xp94\" (UID: \"760609c0-c9c8-44d2-87f4-7fd81b6eb7da\") " pod="openshift-multus/multus-additional-cni-plugins-9xp94" Jul 18 11:48:53.406995 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:53.403822 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q2pxv\" (UniqueName: \"kubernetes.io/projected/3072709c-b577-46c4-9616-f0ca460c4196-kube-api-access-q2pxv\") pod \"multus-xlwqp\" (UID: \"3072709c-b577-46c4-9616-f0ca460c4196\") " pod="openshift-multus/multus-xlwqp" Jul 18 11:48:53.406995 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:53.403848 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/247c97c8-982b-4176-ae10-afc396745360-host-kubelet\") pod \"ovnkube-node-wq4lg\" (UID: \"247c97c8-982b-4176-ae10-afc396745360\") " pod="openshift-ovn-kubernetes/ovnkube-node-wq4lg" Jul 18 11:48:53.406995 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:53.403868 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/760609c0-c9c8-44d2-87f4-7fd81b6eb7da-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-9xp94\" (UID: \"760609c0-c9c8-44d2-87f4-7fd81b6eb7da\") " pod="openshift-multus/multus-additional-cni-plugins-9xp94" Jul 18 11:48:53.406995 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:53.403872 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/247c97c8-982b-4176-ae10-afc396745360-run-openvswitch\") pod \"ovnkube-node-wq4lg\" (UID: \"247c97c8-982b-4176-ae10-afc396745360\") " pod="openshift-ovn-kubernetes/ovnkube-node-wq4lg" Jul 18 11:48:53.406995 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:53.403895 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/62e02fd7-1e2e-4f92-9989-08bb7b819cfa-dbus\") pod \"global-pull-secret-syncer-sp74l\" (UID: \"62e02fd7-1e2e-4f92-9989-08bb7b819cfa\") " pod="kube-system/global-pull-secret-syncer-sp74l" Jul 18 11:48:53.406995 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:53.403912 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/3072709c-b577-46c4-9616-f0ca460c4196-hostroot\") pod \"multus-xlwqp\" (UID: \"3072709c-b577-46c4-9616-f0ca460c4196\") " pod="openshift-multus/multus-xlwqp" Jul 18 11:48:53.406995 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:53.403920 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/99561f5b-25bc-4089-b635-e8e896bfe43d-etc-sysctl-d\") pod \"tuned-c6jpl\" (UID: \"99561f5b-25bc-4089-b635-e8e896bfe43d\") " pod="openshift-cluster-node-tuning-operator/tuned-c6jpl" Jul 18 11:48:53.406995 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:53.403943 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/99561f5b-25bc-4089-b635-e8e896bfe43d-host\") pod \"tuned-c6jpl\" (UID: \"99561f5b-25bc-4089-b635-e8e896bfe43d\") " pod="openshift-cluster-node-tuning-operator/tuned-c6jpl" Jul 18 11:48:53.406995 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:53.403961 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c11da53c-730b-4a02-b2f6-fbf3ee1a6cbd-host-slash\") pod \"iptables-alerter-vrm8v\" (UID: \"c11da53c-730b-4a02-b2f6-fbf3ee1a6cbd\") " pod="openshift-network-operator/iptables-alerter-vrm8v" Jul 18 11:48:53.406995 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:53.403967 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4731170e-9d96-4544-a641-5fc8e1659d8e-metrics-certs\") pod \"network-metrics-daemon-hppk2\" (UID: \"4731170e-9d96-4544-a641-5fc8e1659d8e\") " pod="openshift-multus/network-metrics-daemon-hppk2" Jul 18 11:48:53.406995 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:53.403991 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/247c97c8-982b-4176-ae10-afc396745360-run-systemd\") pod \"ovnkube-node-wq4lg\" (UID: \"247c97c8-982b-4176-ae10-afc396745360\") " pod="openshift-ovn-kubernetes/ovnkube-node-wq4lg" Jul 18 11:48:53.406995 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:53.403997 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d12e3a2e-a05d-4659-b74e-c30e765448bf-host\") pod \"node-ca-nzpfs\" (UID: \"d12e3a2e-a05d-4659-b74e-c30e765448bf\") " pod="openshift-image-registry/node-ca-nzpfs" Jul 18 11:48:53.406995 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:53.404013 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/99561f5b-25bc-4089-b635-e8e896bfe43d-etc-sysconfig\") pod \"tuned-c6jpl\" (UID: \"99561f5b-25bc-4089-b635-e8e896bfe43d\") " pod="openshift-cluster-node-tuning-operator/tuned-c6jpl" Jul 18 11:48:53.406995 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:53.404014 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/760609c0-c9c8-44d2-87f4-7fd81b6eb7da-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-9xp94\" (UID: \"760609c0-c9c8-44d2-87f4-7fd81b6eb7da\") " pod="openshift-multus/multus-additional-cni-plugins-9xp94" Jul 18 11:48:53.406995 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:53.404064 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/247c97c8-982b-4176-ae10-afc396745360-run-openvswitch\") pod \"ovnkube-node-wq4lg\" (UID: \"247c97c8-982b-4176-ae10-afc396745360\") " pod="openshift-ovn-kubernetes/ovnkube-node-wq4lg" Jul 18 11:48:53.407500 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:53.404078 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/62e02fd7-1e2e-4f92-9989-08bb7b819cfa-dbus\") pod \"global-pull-secret-syncer-sp74l\" (UID: \"62e02fd7-1e2e-4f92-9989-08bb7b819cfa\") " pod="kube-system/global-pull-secret-syncer-sp74l" Jul 18 11:48:53.407500 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:53.404108 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/99561f5b-25bc-4089-b635-e8e896bfe43d-sys\") pod \"tuned-c6jpl\" (UID: \"99561f5b-25bc-4089-b635-e8e896bfe43d\") " pod="openshift-cluster-node-tuning-operator/tuned-c6jpl" Jul 18 11:48:53.407500 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:53.404119 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/99561f5b-25bc-4089-b635-e8e896bfe43d-etc-sysctl-d\") pod \"tuned-c6jpl\" (UID: \"99561f5b-25bc-4089-b635-e8e896bfe43d\") " pod="openshift-cluster-node-tuning-operator/tuned-c6jpl" Jul 18 11:48:53.407500 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:53.404134 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/d12e3a2e-a05d-4659-b74e-c30e765448bf-serviceca\") pod \"node-ca-nzpfs\" (UID: \"d12e3a2e-a05d-4659-b74e-c30e765448bf\") " pod="openshift-image-registry/node-ca-nzpfs" Jul 18 11:48:53.407500 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:53.404159 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/9e7851c1-2896-4b77-9f84-cb29edf29b87-sys-fs\") pod \"aws-ebs-csi-driver-node-fjjlw\" (UID: \"9e7851c1-2896-4b77-9f84-cb29edf29b87\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fjjlw" Jul 18 11:48:53.407500 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:53.404167 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/247c97c8-982b-4176-ae10-afc396745360-run-systemd\") pod \"ovnkube-node-wq4lg\" (UID: \"247c97c8-982b-4176-ae10-afc396745360\") " pod="openshift-ovn-kubernetes/ovnkube-node-wq4lg" Jul 18 11:48:53.407500 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:53.404182 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/97228923-3172-4691-927f-723ef88035cf-hosts-file\") pod \"node-resolver-hv6g8\" (UID: \"97228923-3172-4691-927f-723ef88035cf\") " pod="openshift-dns/node-resolver-hv6g8" Jul 18 11:48:53.407500 ip-10-0-136-240 kubenswrapper[2585]: E0718 11:48:53.404188 2585 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 18 11:48:53.407500 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:53.404208 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2z5vr\" (UniqueName: \"kubernetes.io/projected/97228923-3172-4691-927f-723ef88035cf-kube-api-access-2z5vr\") pod \"node-resolver-hv6g8\" (UID: \"97228923-3172-4691-927f-723ef88035cf\") " pod="openshift-dns/node-resolver-hv6g8" Jul 18 11:48:53.407500 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:53.404215 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/99561f5b-25bc-4089-b635-e8e896bfe43d-etc-sysconfig\") pod \"tuned-c6jpl\" (UID: \"99561f5b-25bc-4089-b635-e8e896bfe43d\") " pod="openshift-cluster-node-tuning-operator/tuned-c6jpl" Jul 18 11:48:53.407500 ip-10-0-136-240 kubenswrapper[2585]: E0718 11:48:53.404244 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4731170e-9d96-4544-a641-5fc8e1659d8e-metrics-certs podName:4731170e-9d96-4544-a641-5fc8e1659d8e nodeName:}" failed. No retries permitted until 2026-07-18 11:48:53.904227049 +0000 UTC m=+2.117507905 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4731170e-9d96-4544-a641-5fc8e1659d8e-metrics-certs") pod "network-metrics-daemon-hppk2" (UID: "4731170e-9d96-4544-a641-5fc8e1659d8e") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 18 11:48:53.407500 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:53.404262 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3072709c-b577-46c4-9616-f0ca460c4196-system-cni-dir\") pod \"multus-xlwqp\" (UID: \"3072709c-b577-46c4-9616-f0ca460c4196\") " pod="openshift-multus/multus-xlwqp" Jul 18 11:48:53.407500 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:53.404267 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/99561f5b-25bc-4089-b635-e8e896bfe43d-host\") pod \"tuned-c6jpl\" (UID: \"99561f5b-25bc-4089-b635-e8e896bfe43d\") " pod="openshift-cluster-node-tuning-operator/tuned-c6jpl" Jul 18 11:48:53.407500 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:53.404123 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/760609c0-c9c8-44d2-87f4-7fd81b6eb7da-os-release\") pod \"multus-additional-cni-plugins-9xp94\" (UID: \"760609c0-c9c8-44d2-87f4-7fd81b6eb7da\") " pod="openshift-multus/multus-additional-cni-plugins-9xp94" Jul 18 11:48:53.407500 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:53.404342 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/99561f5b-25bc-4089-b635-e8e896bfe43d-sys\") pod \"tuned-c6jpl\" (UID: \"99561f5b-25bc-4089-b635-e8e896bfe43d\") " pod="openshift-cluster-node-tuning-operator/tuned-c6jpl" Jul 18 11:48:53.407500 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:53.404344 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3072709c-b577-46c4-9616-f0ca460c4196-system-cni-dir\") pod \"multus-xlwqp\" (UID: \"3072709c-b577-46c4-9616-f0ca460c4196\") " pod="openshift-multus/multus-xlwqp" Jul 18 11:48:53.407500 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:53.404429 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/9e7851c1-2896-4b77-9f84-cb29edf29b87-sys-fs\") pod \"aws-ebs-csi-driver-node-fjjlw\" (UID: \"9e7851c1-2896-4b77-9f84-cb29edf29b87\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fjjlw" Jul 18 11:48:53.407500 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:53.404504 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/97228923-3172-4691-927f-723ef88035cf-hosts-file\") pod \"node-resolver-hv6g8\" (UID: \"97228923-3172-4691-927f-723ef88035cf\") " pod="openshift-dns/node-resolver-hv6g8" Jul 18 11:48:53.408065 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:53.404577 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/247c97c8-982b-4176-ae10-afc396745360-env-overrides\") pod \"ovnkube-node-wq4lg\" (UID: \"247c97c8-982b-4176-ae10-afc396745360\") " pod="openshift-ovn-kubernetes/ovnkube-node-wq4lg" Jul 18 11:48:53.408065 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:53.404660 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/247c97c8-982b-4176-ae10-afc396745360-etc-openvswitch\") pod \"ovnkube-node-wq4lg\" (UID: \"247c97c8-982b-4176-ae10-afc396745360\") " pod="openshift-ovn-kubernetes/ovnkube-node-wq4lg" Jul 18 11:48:53.408065 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:53.404725 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/9e7851c1-2896-4b77-9f84-cb29edf29b87-device-dir\") pod \"aws-ebs-csi-driver-node-fjjlw\" (UID: \"9e7851c1-2896-4b77-9f84-cb29edf29b87\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fjjlw" Jul 18 11:48:53.408065 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:53.404756 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/d12e3a2e-a05d-4659-b74e-c30e765448bf-serviceca\") pod \"node-ca-nzpfs\" (UID: \"d12e3a2e-a05d-4659-b74e-c30e765448bf\") " pod="openshift-image-registry/node-ca-nzpfs" Jul 18 11:48:53.408065 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:53.404841 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/247c97c8-982b-4176-ae10-afc396745360-ovnkube-script-lib\") pod \"ovnkube-node-wq4lg\" (UID: \"247c97c8-982b-4176-ae10-afc396745360\") " pod="openshift-ovn-kubernetes/ovnkube-node-wq4lg" Jul 18 11:48:53.408065 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:53.404920 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/247c97c8-982b-4176-ae10-afc396745360-host-kubelet\") pod \"ovnkube-node-wq4lg\" (UID: \"247c97c8-982b-4176-ae10-afc396745360\") " pod="openshift-ovn-kubernetes/ovnkube-node-wq4lg" Jul 18 11:48:53.408065 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:53.405007 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/c11da53c-730b-4a02-b2f6-fbf3ee1a6cbd-iptables-alerter-script\") pod \"iptables-alerter-vrm8v\" (UID: \"c11da53c-730b-4a02-b2f6-fbf3ee1a6cbd\") " pod="openshift-network-operator/iptables-alerter-vrm8v" Jul 18 11:48:53.408065 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:53.405207 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/760609c0-c9c8-44d2-87f4-7fd81b6eb7da-cni-binary-copy\") pod \"multus-additional-cni-plugins-9xp94\" (UID: \"760609c0-c9c8-44d2-87f4-7fd81b6eb7da\") " pod="openshift-multus/multus-additional-cni-plugins-9xp94" Jul 18 11:48:53.408065 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:53.406164 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/247c97c8-982b-4176-ae10-afc396745360-ovn-node-metrics-cert\") pod \"ovnkube-node-wq4lg\" (UID: \"247c97c8-982b-4176-ae10-afc396745360\") " pod="openshift-ovn-kubernetes/ovnkube-node-wq4lg" Jul 18 11:48:53.408065 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:53.406308 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/99561f5b-25bc-4089-b635-e8e896bfe43d-etc-tuned\") pod \"tuned-c6jpl\" (UID: \"99561f5b-25bc-4089-b635-e8e896bfe43d\") " pod="openshift-cluster-node-tuning-operator/tuned-c6jpl" Jul 18 11:48:53.408065 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:53.406417 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/a2e064ca-7d18-4e4a-952c-c00aebf5deb4-agent-certs\") pod \"konnectivity-agent-844xq\" (UID: \"a2e064ca-7d18-4e4a-952c-c00aebf5deb4\") " pod="kube-system/konnectivity-agent-844xq" Jul 18 11:48:53.408065 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:53.406581 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/99561f5b-25bc-4089-b635-e8e896bfe43d-tmp\") pod \"tuned-c6jpl\" (UID: \"99561f5b-25bc-4089-b635-e8e896bfe43d\") " pod="openshift-cluster-node-tuning-operator/tuned-c6jpl" Jul 18 11:48:53.411270 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:53.411242 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mlxwx\" (UniqueName: \"kubernetes.io/projected/c11da53c-730b-4a02-b2f6-fbf3ee1a6cbd-kube-api-access-mlxwx\") pod \"iptables-alerter-vrm8v\" (UID: \"c11da53c-730b-4a02-b2f6-fbf3ee1a6cbd\") " pod="openshift-network-operator/iptables-alerter-vrm8v" Jul 18 11:48:53.411346 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:53.411297 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q7lfh\" (UniqueName: \"kubernetes.io/projected/d12e3a2e-a05d-4659-b74e-c30e765448bf-kube-api-access-q7lfh\") pod \"node-ca-nzpfs\" (UID: \"d12e3a2e-a05d-4659-b74e-c30e765448bf\") " pod="openshift-image-registry/node-ca-nzpfs" Jul 18 11:48:53.411397 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:53.411363 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k5dxp\" (UniqueName: \"kubernetes.io/projected/99561f5b-25bc-4089-b635-e8e896bfe43d-kube-api-access-k5dxp\") pod \"tuned-c6jpl\" (UID: \"99561f5b-25bc-4089-b635-e8e896bfe43d\") " pod="openshift-cluster-node-tuning-operator/tuned-c6jpl" Jul 18 11:48:53.412349 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:53.412330 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w2cf9\" (UniqueName: \"kubernetes.io/projected/760609c0-c9c8-44d2-87f4-7fd81b6eb7da-kube-api-access-w2cf9\") pod \"multus-additional-cni-plugins-9xp94\" (UID: \"760609c0-c9c8-44d2-87f4-7fd81b6eb7da\") " pod="openshift-multus/multus-additional-cni-plugins-9xp94" Jul 18 11:48:53.414299 ip-10-0-136-240 kubenswrapper[2585]: E0718 11:48:53.414266 2585 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 18 11:48:53.414299 ip-10-0-136-240 kubenswrapper[2585]: E0718 11:48:53.414296 2585 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 18 11:48:53.414450 ip-10-0-136-240 kubenswrapper[2585]: E0718 11:48:53.414311 2585 projected.go:196] Error preparing data for projected volume kube-api-access-frt9b for pod openshift-network-diagnostics/network-check-target-vd8wb: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 18 11:48:53.414450 ip-10-0-136-240 kubenswrapper[2585]: E0718 11:48:53.414374 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c1cdc202-852a-4bc3-a68c-8f5bf5baab20-kube-api-access-frt9b podName:c1cdc202-852a-4bc3-a68c-8f5bf5baab20 nodeName:}" failed. No retries permitted until 2026-07-18 11:48:53.914357192 +0000 UTC m=+2.127638033 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-frt9b" (UniqueName: "kubernetes.io/projected/c1cdc202-852a-4bc3-a68c-8f5bf5baab20-kube-api-access-frt9b") pod "network-check-target-vd8wb" (UID: "c1cdc202-852a-4bc3-a68c-8f5bf5baab20") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 18 11:48:53.415788 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:53.415759 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v4gpm\" (UniqueName: \"kubernetes.io/projected/9e7851c1-2896-4b77-9f84-cb29edf29b87-kube-api-access-v4gpm\") pod \"aws-ebs-csi-driver-node-fjjlw\" (UID: \"9e7851c1-2896-4b77-9f84-cb29edf29b87\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fjjlw" Jul 18 11:48:53.415873 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:53.415833 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tgvmr\" (UniqueName: \"kubernetes.io/projected/4731170e-9d96-4544-a641-5fc8e1659d8e-kube-api-access-tgvmr\") pod \"network-metrics-daemon-hppk2\" (UID: \"4731170e-9d96-4544-a641-5fc8e1659d8e\") " pod="openshift-multus/network-metrics-daemon-hppk2" Jul 18 11:48:53.416410 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:53.416392 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ldhwg\" (UniqueName: \"kubernetes.io/projected/247c97c8-982b-4176-ae10-afc396745360-kube-api-access-ldhwg\") pod \"ovnkube-node-wq4lg\" (UID: \"247c97c8-982b-4176-ae10-afc396745360\") " pod="openshift-ovn-kubernetes/ovnkube-node-wq4lg" Jul 18 11:48:53.416625 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:53.416610 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q2pxv\" (UniqueName: \"kubernetes.io/projected/3072709c-b577-46c4-9616-f0ca460c4196-kube-api-access-q2pxv\") pod \"multus-xlwqp\" (UID: \"3072709c-b577-46c4-9616-f0ca460c4196\") " pod="openshift-multus/multus-xlwqp" Jul 18 11:48:53.416674 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:53.416617 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2z5vr\" (UniqueName: \"kubernetes.io/projected/97228923-3172-4691-927f-723ef88035cf-kube-api-access-2z5vr\") pod \"node-resolver-hv6g8\" (UID: \"97228923-3172-4691-927f-723ef88035cf\") " pod="openshift-dns/node-resolver-hv6g8" Jul 18 11:48:53.521679 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:53.521608 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-xlwqp" Jul 18 11:48:53.531321 ip-10-0-136-240 kubenswrapper[2585]: W0718 11:48:53.531299 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3072709c_b577_46c4_9616_f0ca460c4196.slice/crio-eb9e7fc38dc793a4e88743c3ddaf48eecf13c0f4b3286385b17a426494801a15 WatchSource:0}: Error finding container eb9e7fc38dc793a4e88743c3ddaf48eecf13c0f4b3286385b17a426494801a15: Status 404 returned error can't find the container with id eb9e7fc38dc793a4e88743c3ddaf48eecf13c0f4b3286385b17a426494801a15 Jul 18 11:48:53.539322 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:53.539306 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-844xq" Jul 18 11:48:53.547153 ip-10-0-136-240 kubenswrapper[2585]: W0718 11:48:53.547132 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda2e064ca_7d18_4e4a_952c_c00aebf5deb4.slice/crio-9bfbd327236172066f427df08acea3b9a0adb4c9469af2490c6a46c273edcddf WatchSource:0}: Error finding container 9bfbd327236172066f427df08acea3b9a0adb4c9469af2490c6a46c273edcddf: Status 404 returned error can't find the container with id 9bfbd327236172066f427df08acea3b9a0adb4c9469af2490c6a46c273edcddf Jul 18 11:48:53.556870 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:53.556851 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-vrm8v" Jul 18 11:48:53.565574 ip-10-0-136-240 kubenswrapper[2585]: W0718 11:48:53.565547 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc11da53c_730b_4a02_b2f6_fbf3ee1a6cbd.slice/crio-b1707b18a8783aa2f45c2098fc39f147e4a833db88256b43e291171921dbaa3c WatchSource:0}: Error finding container b1707b18a8783aa2f45c2098fc39f147e4a833db88256b43e291171921dbaa3c: Status 404 returned error can't find the container with id b1707b18a8783aa2f45c2098fc39f147e4a833db88256b43e291171921dbaa3c Jul 18 11:48:53.573311 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:53.573288 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-wq4lg" Jul 18 11:48:53.578055 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:53.578036 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fjjlw" Jul 18 11:48:53.581276 ip-10-0-136-240 kubenswrapper[2585]: W0718 11:48:53.581103 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod247c97c8_982b_4176_ae10_afc396745360.slice/crio-abb7be9a5bc56298c429533f40bb0d86a1d3b45968302289d77abaabdfe24700 WatchSource:0}: Error finding container abb7be9a5bc56298c429533f40bb0d86a1d3b45968302289d77abaabdfe24700: Status 404 returned error can't find the container with id abb7be9a5bc56298c429533f40bb0d86a1d3b45968302289d77abaabdfe24700 Jul 18 11:48:53.586531 ip-10-0-136-240 kubenswrapper[2585]: W0718 11:48:53.586513 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9e7851c1_2896_4b77_9f84_cb29edf29b87.slice/crio-953cc778bb292370eedb3c920eaa6eedf00753eb1cbddd51e48958d5777f63f4 WatchSource:0}: Error finding container 953cc778bb292370eedb3c920eaa6eedf00753eb1cbddd51e48958d5777f63f4: Status 404 returned error can't find the container with id 953cc778bb292370eedb3c920eaa6eedf00753eb1cbddd51e48958d5777f63f4 Jul 18 11:48:54.747125 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:54.747108 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-c6jpl" Jul 18 11:48:54.753796 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:54.753782 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-hv6g8" Jul 18 11:48:54.755899 ip-10-0-136-240 kubenswrapper[2585]: W0718 11:48:54.755882 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod99561f5b_25bc_4089_b635_e8e896bfe43d.slice/crio-10dcb55aa8766d5d6b18b825f2b864bbc901aaecf279962f055be3f838217703 WatchSource:0}: Error finding container 10dcb55aa8766d5d6b18b825f2b864bbc901aaecf279962f055be3f838217703: Status 404 returned error can't find the container with id 10dcb55aa8766d5d6b18b825f2b864bbc901aaecf279962f055be3f838217703 Jul 18 11:48:54.759051 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:54.759011 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-nzpfs" Jul 18 11:48:54.762009 ip-10-0-136-240 kubenswrapper[2585]: W0718 11:48:54.761988 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod97228923_3172_4691_927f_723ef88035cf.slice/crio-903a367de832eb53cd1cfeef983a4757237b0f75d3d44284fa82f050da56b5b3 WatchSource:0}: Error finding container 903a367de832eb53cd1cfeef983a4757237b0f75d3d44284fa82f050da56b5b3: Status 404 returned error can't find the container with id 903a367de832eb53cd1cfeef983a4757237b0f75d3d44284fa82f050da56b5b3 Jul 18 11:48:54.763764 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:54.763744 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-9xp94" Jul 18 11:48:54.769684 ip-10-0-136-240 kubenswrapper[2585]: W0718 11:48:54.769660 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd12e3a2e_a05d_4659_b74e_c30e765448bf.slice/crio-69e2dc836b4675dd08e681377f7ed037992fa5b2abcd1f1857e070865c7dcfb3 WatchSource:0}: Error finding container 69e2dc836b4675dd08e681377f7ed037992fa5b2abcd1f1857e070865c7dcfb3: Status 404 returned error can't find the container with id 69e2dc836b4675dd08e681377f7ed037992fa5b2abcd1f1857e070865c7dcfb3 Jul 18 11:48:54.769785 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:54.769770 2585 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 18 11:48:54.772617 ip-10-0-136-240 kubenswrapper[2585]: W0718 11:48:54.772540 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod760609c0_c9c8_44d2_87f4_7fd81b6eb7da.slice/crio-7539aea978b410bf0b4063002ae5fb8f7877653d130cb0c1e4f6da846a981688 WatchSource:0}: Error finding container 7539aea978b410bf0b4063002ae5fb8f7877653d130cb0c1e4f6da846a981688: Status 404 returned error can't find the container with id 7539aea978b410bf0b4063002ae5fb8f7877653d130cb0c1e4f6da846a981688 Jul 18 11:48:54.847824 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:54.847799 2585 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Jul 18 11:48:55.042309 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:55.042228 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/62e02fd7-1e2e-4f92-9989-08bb7b819cfa-original-pull-secret\") pod \"global-pull-secret-syncer-sp74l\" (UID: \"62e02fd7-1e2e-4f92-9989-08bb7b819cfa\") " pod="kube-system/global-pull-secret-syncer-sp74l" Jul 18 11:48:55.042462 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:55.042307 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4731170e-9d96-4544-a641-5fc8e1659d8e-metrics-certs\") pod \"network-metrics-daemon-hppk2\" (UID: \"4731170e-9d96-4544-a641-5fc8e1659d8e\") " pod="openshift-multus/network-metrics-daemon-hppk2" Jul 18 11:48:55.042594 ip-10-0-136-240 kubenswrapper[2585]: E0718 11:48:55.042538 2585 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 18 11:48:55.042654 ip-10-0-136-240 kubenswrapper[2585]: E0718 11:48:55.042630 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/62e02fd7-1e2e-4f92-9989-08bb7b819cfa-original-pull-secret podName:62e02fd7-1e2e-4f92-9989-08bb7b819cfa nodeName:}" failed. No retries permitted until 2026-07-18 11:48:56.04260278 +0000 UTC m=+3.122809119 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/62e02fd7-1e2e-4f92-9989-08bb7b819cfa-original-pull-secret") pod "global-pull-secret-syncer-sp74l" (UID: "62e02fd7-1e2e-4f92-9989-08bb7b819cfa") : object "kube-system"/"original-pull-secret" not registered Jul 18 11:48:55.043047 ip-10-0-136-240 kubenswrapper[2585]: E0718 11:48:55.043025 2585 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 18 11:48:55.043150 ip-10-0-136-240 kubenswrapper[2585]: E0718 11:48:55.043082 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4731170e-9d96-4544-a641-5fc8e1659d8e-metrics-certs podName:4731170e-9d96-4544-a641-5fc8e1659d8e nodeName:}" failed. No retries permitted until 2026-07-18 11:48:56.043067016 +0000 UTC m=+3.123273367 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4731170e-9d96-4544-a641-5fc8e1659d8e-metrics-certs") pod "network-metrics-daemon-hppk2" (UID: "4731170e-9d96-4544-a641-5fc8e1659d8e") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 18 11:48:55.143316 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:55.143283 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-frt9b\" (UniqueName: \"kubernetes.io/projected/c1cdc202-852a-4bc3-a68c-8f5bf5baab20-kube-api-access-frt9b\") pod \"network-check-target-vd8wb\" (UID: \"c1cdc202-852a-4bc3-a68c-8f5bf5baab20\") " pod="openshift-network-diagnostics/network-check-target-vd8wb" Jul 18 11:48:55.143499 ip-10-0-136-240 kubenswrapper[2585]: E0718 11:48:55.143482 2585 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 18 11:48:55.143584 ip-10-0-136-240 kubenswrapper[2585]: E0718 11:48:55.143501 2585 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 18 11:48:55.143584 ip-10-0-136-240 kubenswrapper[2585]: E0718 11:48:55.143514 2585 projected.go:196] Error preparing data for projected volume kube-api-access-frt9b for pod openshift-network-diagnostics/network-check-target-vd8wb: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 18 11:48:55.143941 ip-10-0-136-240 kubenswrapper[2585]: E0718 11:48:55.143905 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c1cdc202-852a-4bc3-a68c-8f5bf5baab20-kube-api-access-frt9b podName:c1cdc202-852a-4bc3-a68c-8f5bf5baab20 nodeName:}" failed. No retries permitted until 2026-07-18 11:48:56.143568506 +0000 UTC m=+3.223774853 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-frt9b" (UniqueName: "kubernetes.io/projected/c1cdc202-852a-4bc3-a68c-8f5bf5baab20-kube-api-access-frt9b") pod "network-check-target-vd8wb" (UID: "c1cdc202-852a-4bc3-a68c-8f5bf5baab20") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 18 11:48:55.371664 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:55.371529 2585 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-17 11:43:54 +0000 UTC" deadline="2027-12-14 20:29:01.667399462 +0000 UTC" Jul 18 11:48:55.371664 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:55.371613 2585 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12344h40m6.295791162s" Jul 18 11:48:55.466518 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:55.465208 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hppk2" Jul 18 11:48:55.466518 ip-10-0-136-240 kubenswrapper[2585]: E0718 11:48:55.465344 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hppk2" podUID="4731170e-9d96-4544-a641-5fc8e1659d8e" Jul 18 11:48:55.496006 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:55.495970 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-hv6g8" event={"ID":"97228923-3172-4691-927f-723ef88035cf","Type":"ContainerStarted","Data":"903a367de832eb53cd1cfeef983a4757237b0f75d3d44284fa82f050da56b5b3"} Jul 18 11:48:55.505205 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:55.505163 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-c6jpl" event={"ID":"99561f5b-25bc-4089-b635-e8e896bfe43d","Type":"ContainerStarted","Data":"10dcb55aa8766d5d6b18b825f2b864bbc901aaecf279962f055be3f838217703"} Jul 18 11:48:55.512951 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:55.512924 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-wq4lg" event={"ID":"247c97c8-982b-4176-ae10-afc396745360","Type":"ContainerStarted","Data":"abb7be9a5bc56298c429533f40bb0d86a1d3b45968302289d77abaabdfe24700"} Jul 18 11:48:55.525792 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:55.525762 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-9xp94" event={"ID":"760609c0-c9c8-44d2-87f4-7fd81b6eb7da","Type":"ContainerStarted","Data":"7539aea978b410bf0b4063002ae5fb8f7877653d130cb0c1e4f6da846a981688"} Jul 18 11:48:55.528256 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:55.528236 2585 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 18 11:48:55.542966 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:55.542938 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-nzpfs" event={"ID":"d12e3a2e-a05d-4659-b74e-c30e765448bf","Type":"ContainerStarted","Data":"69e2dc836b4675dd08e681377f7ed037992fa5b2abcd1f1857e070865c7dcfb3"} Jul 18 11:48:55.566992 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:55.566940 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fjjlw" event={"ID":"9e7851c1-2896-4b77-9f84-cb29edf29b87","Type":"ContainerStarted","Data":"953cc778bb292370eedb3c920eaa6eedf00753eb1cbddd51e48958d5777f63f4"} Jul 18 11:48:55.578352 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:55.578313 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-vrm8v" event={"ID":"c11da53c-730b-4a02-b2f6-fbf3ee1a6cbd","Type":"ContainerStarted","Data":"b1707b18a8783aa2f45c2098fc39f147e4a833db88256b43e291171921dbaa3c"} Jul 18 11:48:55.585570 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:55.585498 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-844xq" event={"ID":"a2e064ca-7d18-4e4a-952c-c00aebf5deb4","Type":"ContainerStarted","Data":"9bfbd327236172066f427df08acea3b9a0adb4c9469af2490c6a46c273edcddf"} Jul 18 11:48:55.601059 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:55.601032 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-xlwqp" event={"ID":"3072709c-b577-46c4-9616-f0ca460c4196","Type":"ContainerStarted","Data":"eb9e7fc38dc793a4e88743c3ddaf48eecf13c0f4b3286385b17a426494801a15"} Jul 18 11:48:56.049785 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:56.049748 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/62e02fd7-1e2e-4f92-9989-08bb7b819cfa-original-pull-secret\") pod \"global-pull-secret-syncer-sp74l\" (UID: \"62e02fd7-1e2e-4f92-9989-08bb7b819cfa\") " pod="kube-system/global-pull-secret-syncer-sp74l" Jul 18 11:48:56.049987 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:56.049880 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4731170e-9d96-4544-a641-5fc8e1659d8e-metrics-certs\") pod \"network-metrics-daemon-hppk2\" (UID: \"4731170e-9d96-4544-a641-5fc8e1659d8e\") " pod="openshift-multus/network-metrics-daemon-hppk2" Jul 18 11:48:56.050062 ip-10-0-136-240 kubenswrapper[2585]: E0718 11:48:56.050033 2585 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 18 11:48:56.050120 ip-10-0-136-240 kubenswrapper[2585]: E0718 11:48:56.050096 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4731170e-9d96-4544-a641-5fc8e1659d8e-metrics-certs podName:4731170e-9d96-4544-a641-5fc8e1659d8e nodeName:}" failed. No retries permitted until 2026-07-18 11:48:58.050077001 +0000 UTC m=+5.130283343 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4731170e-9d96-4544-a641-5fc8e1659d8e-metrics-certs") pod "network-metrics-daemon-hppk2" (UID: "4731170e-9d96-4544-a641-5fc8e1659d8e") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 18 11:48:56.050505 ip-10-0-136-240 kubenswrapper[2585]: E0718 11:48:56.050484 2585 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 18 11:48:56.050700 ip-10-0-136-240 kubenswrapper[2585]: E0718 11:48:56.050543 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/62e02fd7-1e2e-4f92-9989-08bb7b819cfa-original-pull-secret podName:62e02fd7-1e2e-4f92-9989-08bb7b819cfa nodeName:}" failed. No retries permitted until 2026-07-18 11:48:58.050528413 +0000 UTC m=+5.130734754 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/62e02fd7-1e2e-4f92-9989-08bb7b819cfa-original-pull-secret") pod "global-pull-secret-syncer-sp74l" (UID: "62e02fd7-1e2e-4f92-9989-08bb7b819cfa") : object "kube-system"/"original-pull-secret" not registered Jul 18 11:48:56.150480 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:56.150441 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-frt9b\" (UniqueName: \"kubernetes.io/projected/c1cdc202-852a-4bc3-a68c-8f5bf5baab20-kube-api-access-frt9b\") pod \"network-check-target-vd8wb\" (UID: \"c1cdc202-852a-4bc3-a68c-8f5bf5baab20\") " pod="openshift-network-diagnostics/network-check-target-vd8wb" Jul 18 11:48:56.150688 ip-10-0-136-240 kubenswrapper[2585]: E0718 11:48:56.150672 2585 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 18 11:48:56.150755 ip-10-0-136-240 kubenswrapper[2585]: E0718 11:48:56.150693 2585 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 18 11:48:56.150755 ip-10-0-136-240 kubenswrapper[2585]: E0718 11:48:56.150707 2585 projected.go:196] Error preparing data for projected volume kube-api-access-frt9b for pod openshift-network-diagnostics/network-check-target-vd8wb: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 18 11:48:56.150865 ip-10-0-136-240 kubenswrapper[2585]: E0718 11:48:56.150760 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c1cdc202-852a-4bc3-a68c-8f5bf5baab20-kube-api-access-frt9b podName:c1cdc202-852a-4bc3-a68c-8f5bf5baab20 nodeName:}" failed. No retries permitted until 2026-07-18 11:48:58.150743267 +0000 UTC m=+5.230949605 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-frt9b" (UniqueName: "kubernetes.io/projected/c1cdc202-852a-4bc3-a68c-8f5bf5baab20-kube-api-access-frt9b") pod "network-check-target-vd8wb" (UID: "c1cdc202-852a-4bc3-a68c-8f5bf5baab20") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 18 11:48:56.372123 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:56.372030 2585 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-17 11:43:54 +0000 UTC" deadline="2028-03-09 08:55:22.576508844 +0000 UTC" Jul 18 11:48:56.372123 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:56.372069 2585 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14397h6m26.204446043s" Jul 18 11:48:56.462910 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:56.462877 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sp74l" Jul 18 11:48:56.463083 ip-10-0-136-240 kubenswrapper[2585]: E0718 11:48:56.463005 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sp74l" podUID="62e02fd7-1e2e-4f92-9989-08bb7b819cfa" Jul 18 11:48:56.463083 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:56.463040 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vd8wb" Jul 18 11:48:56.463202 ip-10-0-136-240 kubenswrapper[2585]: E0718 11:48:56.463105 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vd8wb" podUID="c1cdc202-852a-4bc3-a68c-8f5bf5baab20" Jul 18 11:48:57.083032 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:57.083000 2585 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Jul 18 11:48:57.468217 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:57.467749 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hppk2" Jul 18 11:48:57.468217 ip-10-0-136-240 kubenswrapper[2585]: E0718 11:48:57.467885 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hppk2" podUID="4731170e-9d96-4544-a641-5fc8e1659d8e" Jul 18 11:48:58.065661 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:58.065625 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4731170e-9d96-4544-a641-5fc8e1659d8e-metrics-certs\") pod \"network-metrics-daemon-hppk2\" (UID: \"4731170e-9d96-4544-a641-5fc8e1659d8e\") " pod="openshift-multus/network-metrics-daemon-hppk2" Jul 18 11:48:58.065822 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:58.065709 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/62e02fd7-1e2e-4f92-9989-08bb7b819cfa-original-pull-secret\") pod \"global-pull-secret-syncer-sp74l\" (UID: \"62e02fd7-1e2e-4f92-9989-08bb7b819cfa\") " pod="kube-system/global-pull-secret-syncer-sp74l" Jul 18 11:48:58.065893 ip-10-0-136-240 kubenswrapper[2585]: E0718 11:48:58.065832 2585 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 18 11:48:58.065893 ip-10-0-136-240 kubenswrapper[2585]: E0718 11:48:58.065891 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/62e02fd7-1e2e-4f92-9989-08bb7b819cfa-original-pull-secret podName:62e02fd7-1e2e-4f92-9989-08bb7b819cfa nodeName:}" failed. No retries permitted until 2026-07-18 11:49:02.065873079 +0000 UTC m=+9.146079419 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/62e02fd7-1e2e-4f92-9989-08bb7b819cfa-original-pull-secret") pod "global-pull-secret-syncer-sp74l" (UID: "62e02fd7-1e2e-4f92-9989-08bb7b819cfa") : object "kube-system"/"original-pull-secret" not registered Jul 18 11:48:58.066277 ip-10-0-136-240 kubenswrapper[2585]: E0718 11:48:58.066256 2585 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 18 11:48:58.066413 ip-10-0-136-240 kubenswrapper[2585]: E0718 11:48:58.066308 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4731170e-9d96-4544-a641-5fc8e1659d8e-metrics-certs podName:4731170e-9d96-4544-a641-5fc8e1659d8e nodeName:}" failed. No retries permitted until 2026-07-18 11:49:02.066292816 +0000 UTC m=+9.146499166 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4731170e-9d96-4544-a641-5fc8e1659d8e-metrics-certs") pod "network-metrics-daemon-hppk2" (UID: "4731170e-9d96-4544-a641-5fc8e1659d8e") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 18 11:48:58.166961 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:58.166914 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-frt9b\" (UniqueName: \"kubernetes.io/projected/c1cdc202-852a-4bc3-a68c-8f5bf5baab20-kube-api-access-frt9b\") pod \"network-check-target-vd8wb\" (UID: \"c1cdc202-852a-4bc3-a68c-8f5bf5baab20\") " pod="openshift-network-diagnostics/network-check-target-vd8wb" Jul 18 11:48:58.167254 ip-10-0-136-240 kubenswrapper[2585]: E0718 11:48:58.167235 2585 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 18 11:48:58.167306 ip-10-0-136-240 kubenswrapper[2585]: E0718 11:48:58.167262 2585 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 18 11:48:58.167306 ip-10-0-136-240 kubenswrapper[2585]: E0718 11:48:58.167277 2585 projected.go:196] Error preparing data for projected volume kube-api-access-frt9b for pod openshift-network-diagnostics/network-check-target-vd8wb: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 18 11:48:58.167372 ip-10-0-136-240 kubenswrapper[2585]: E0718 11:48:58.167345 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c1cdc202-852a-4bc3-a68c-8f5bf5baab20-kube-api-access-frt9b podName:c1cdc202-852a-4bc3-a68c-8f5bf5baab20 nodeName:}" failed. No retries permitted until 2026-07-18 11:49:02.167324138 +0000 UTC m=+9.247530477 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-frt9b" (UniqueName: "kubernetes.io/projected/c1cdc202-852a-4bc3-a68c-8f5bf5baab20-kube-api-access-frt9b") pod "network-check-target-vd8wb" (UID: "c1cdc202-852a-4bc3-a68c-8f5bf5baab20") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 18 11:48:58.463364 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:58.463283 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sp74l" Jul 18 11:48:58.463516 ip-10-0-136-240 kubenswrapper[2585]: E0718 11:48:58.463408 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sp74l" podUID="62e02fd7-1e2e-4f92-9989-08bb7b819cfa" Jul 18 11:48:58.464009 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:58.463798 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vd8wb" Jul 18 11:48:58.464009 ip-10-0-136-240 kubenswrapper[2585]: E0718 11:48:58.463901 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vd8wb" podUID="c1cdc202-852a-4bc3-a68c-8f5bf5baab20" Jul 18 11:48:59.463142 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:48:59.463111 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hppk2" Jul 18 11:48:59.463604 ip-10-0-136-240 kubenswrapper[2585]: E0718 11:48:59.463245 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hppk2" podUID="4731170e-9d96-4544-a641-5fc8e1659d8e" Jul 18 11:49:00.463905 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:49:00.463298 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vd8wb" Jul 18 11:49:00.463905 ip-10-0-136-240 kubenswrapper[2585]: E0718 11:49:00.463433 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vd8wb" podUID="c1cdc202-852a-4bc3-a68c-8f5bf5baab20" Jul 18 11:49:00.463905 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:49:00.463510 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sp74l" Jul 18 11:49:00.463905 ip-10-0-136-240 kubenswrapper[2585]: E0718 11:49:00.463596 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sp74l" podUID="62e02fd7-1e2e-4f92-9989-08bb7b819cfa" Jul 18 11:49:01.463566 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:49:01.463511 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hppk2" Jul 18 11:49:01.463724 ip-10-0-136-240 kubenswrapper[2585]: E0718 11:49:01.463680 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hppk2" podUID="4731170e-9d96-4544-a641-5fc8e1659d8e" Jul 18 11:49:02.105907 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:49:02.105826 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4731170e-9d96-4544-a641-5fc8e1659d8e-metrics-certs\") pod \"network-metrics-daemon-hppk2\" (UID: \"4731170e-9d96-4544-a641-5fc8e1659d8e\") " pod="openshift-multus/network-metrics-daemon-hppk2" Jul 18 11:49:02.106327 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:49:02.105922 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/62e02fd7-1e2e-4f92-9989-08bb7b819cfa-original-pull-secret\") pod \"global-pull-secret-syncer-sp74l\" (UID: \"62e02fd7-1e2e-4f92-9989-08bb7b819cfa\") " pod="kube-system/global-pull-secret-syncer-sp74l" Jul 18 11:49:02.106327 ip-10-0-136-240 kubenswrapper[2585]: E0718 11:49:02.106017 2585 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 18 11:49:02.106327 ip-10-0-136-240 kubenswrapper[2585]: E0718 11:49:02.106052 2585 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 18 11:49:02.106327 ip-10-0-136-240 kubenswrapper[2585]: E0718 11:49:02.106091 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4731170e-9d96-4544-a641-5fc8e1659d8e-metrics-certs podName:4731170e-9d96-4544-a641-5fc8e1659d8e nodeName:}" failed. No retries permitted until 2026-07-18 11:49:10.106071564 +0000 UTC m=+17.186277913 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4731170e-9d96-4544-a641-5fc8e1659d8e-metrics-certs") pod "network-metrics-daemon-hppk2" (UID: "4731170e-9d96-4544-a641-5fc8e1659d8e") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 18 11:49:02.106327 ip-10-0-136-240 kubenswrapper[2585]: E0718 11:49:02.106116 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/62e02fd7-1e2e-4f92-9989-08bb7b819cfa-original-pull-secret podName:62e02fd7-1e2e-4f92-9989-08bb7b819cfa nodeName:}" failed. No retries permitted until 2026-07-18 11:49:10.106098478 +0000 UTC m=+17.186304836 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/62e02fd7-1e2e-4f92-9989-08bb7b819cfa-original-pull-secret") pod "global-pull-secret-syncer-sp74l" (UID: "62e02fd7-1e2e-4f92-9989-08bb7b819cfa") : object "kube-system"/"original-pull-secret" not registered Jul 18 11:49:02.207423 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:49:02.207364 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-frt9b\" (UniqueName: \"kubernetes.io/projected/c1cdc202-852a-4bc3-a68c-8f5bf5baab20-kube-api-access-frt9b\") pod \"network-check-target-vd8wb\" (UID: \"c1cdc202-852a-4bc3-a68c-8f5bf5baab20\") " pod="openshift-network-diagnostics/network-check-target-vd8wb" Jul 18 11:49:02.207659 ip-10-0-136-240 kubenswrapper[2585]: E0718 11:49:02.207640 2585 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 18 11:49:02.207743 ip-10-0-136-240 kubenswrapper[2585]: E0718 11:49:02.207663 2585 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 18 11:49:02.207743 ip-10-0-136-240 kubenswrapper[2585]: E0718 11:49:02.207673 2585 projected.go:196] Error preparing data for projected volume kube-api-access-frt9b for pod openshift-network-diagnostics/network-check-target-vd8wb: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 18 11:49:02.207743 ip-10-0-136-240 kubenswrapper[2585]: E0718 11:49:02.207726 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c1cdc202-852a-4bc3-a68c-8f5bf5baab20-kube-api-access-frt9b podName:c1cdc202-852a-4bc3-a68c-8f5bf5baab20 nodeName:}" failed. No retries permitted until 2026-07-18 11:49:10.207710811 +0000 UTC m=+17.287917164 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-frt9b" (UniqueName: "kubernetes.io/projected/c1cdc202-852a-4bc3-a68c-8f5bf5baab20-kube-api-access-frt9b") pod "network-check-target-vd8wb" (UID: "c1cdc202-852a-4bc3-a68c-8f5bf5baab20") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 18 11:49:02.463470 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:49:02.463389 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sp74l" Jul 18 11:49:02.463648 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:49:02.463520 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vd8wb" Jul 18 11:49:02.463648 ip-10-0-136-240 kubenswrapper[2585]: E0718 11:49:02.463525 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sp74l" podUID="62e02fd7-1e2e-4f92-9989-08bb7b819cfa" Jul 18 11:49:02.463763 ip-10-0-136-240 kubenswrapper[2585]: E0718 11:49:02.463666 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vd8wb" podUID="c1cdc202-852a-4bc3-a68c-8f5bf5baab20" Jul 18 11:49:03.466039 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:49:03.466012 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hppk2" Jul 18 11:49:03.466476 ip-10-0-136-240 kubenswrapper[2585]: E0718 11:49:03.466135 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hppk2" podUID="4731170e-9d96-4544-a641-5fc8e1659d8e" Jul 18 11:49:04.463394 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:49:04.462856 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sp74l" Jul 18 11:49:04.463394 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:49:04.462889 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vd8wb" Jul 18 11:49:04.463394 ip-10-0-136-240 kubenswrapper[2585]: E0718 11:49:04.462998 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sp74l" podUID="62e02fd7-1e2e-4f92-9989-08bb7b819cfa" Jul 18 11:49:04.463394 ip-10-0-136-240 kubenswrapper[2585]: E0718 11:49:04.463120 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vd8wb" podUID="c1cdc202-852a-4bc3-a68c-8f5bf5baab20" Jul 18 11:49:05.463570 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:49:05.463380 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hppk2" Jul 18 11:49:05.464041 ip-10-0-136-240 kubenswrapper[2585]: E0718 11:49:05.463663 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hppk2" podUID="4731170e-9d96-4544-a641-5fc8e1659d8e" Jul 18 11:49:06.463449 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:49:06.463415 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vd8wb" Jul 18 11:49:06.463631 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:49:06.463462 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sp74l" Jul 18 11:49:06.463631 ip-10-0-136-240 kubenswrapper[2585]: E0718 11:49:06.463544 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vd8wb" podUID="c1cdc202-852a-4bc3-a68c-8f5bf5baab20" Jul 18 11:49:06.463975 ip-10-0-136-240 kubenswrapper[2585]: E0718 11:49:06.463675 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sp74l" podUID="62e02fd7-1e2e-4f92-9989-08bb7b819cfa" Jul 18 11:49:07.462964 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:49:07.462933 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hppk2" Jul 18 11:49:07.463111 ip-10-0-136-240 kubenswrapper[2585]: E0718 11:49:07.463083 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hppk2" podUID="4731170e-9d96-4544-a641-5fc8e1659d8e" Jul 18 11:49:08.463596 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:49:08.463543 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sp74l" Jul 18 11:49:08.464002 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:49:08.463565 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vd8wb" Jul 18 11:49:08.464002 ip-10-0-136-240 kubenswrapper[2585]: E0718 11:49:08.463669 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sp74l" podUID="62e02fd7-1e2e-4f92-9989-08bb7b819cfa" Jul 18 11:49:08.464002 ip-10-0-136-240 kubenswrapper[2585]: E0718 11:49:08.463755 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vd8wb" podUID="c1cdc202-852a-4bc3-a68c-8f5bf5baab20" Jul 18 11:49:09.462928 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:49:09.462895 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hppk2" Jul 18 11:49:09.463070 ip-10-0-136-240 kubenswrapper[2585]: E0718 11:49:09.463012 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hppk2" podUID="4731170e-9d96-4544-a641-5fc8e1659d8e" Jul 18 11:49:10.169238 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:49:10.169209 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4731170e-9d96-4544-a641-5fc8e1659d8e-metrics-certs\") pod \"network-metrics-daemon-hppk2\" (UID: \"4731170e-9d96-4544-a641-5fc8e1659d8e\") " pod="openshift-multus/network-metrics-daemon-hppk2" Jul 18 11:49:10.169620 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:49:10.169262 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/62e02fd7-1e2e-4f92-9989-08bb7b819cfa-original-pull-secret\") pod \"global-pull-secret-syncer-sp74l\" (UID: \"62e02fd7-1e2e-4f92-9989-08bb7b819cfa\") " pod="kube-system/global-pull-secret-syncer-sp74l" Jul 18 11:49:10.169620 ip-10-0-136-240 kubenswrapper[2585]: E0718 11:49:10.169350 2585 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 18 11:49:10.169620 ip-10-0-136-240 kubenswrapper[2585]: E0718 11:49:10.169351 2585 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 18 11:49:10.169620 ip-10-0-136-240 kubenswrapper[2585]: E0718 11:49:10.169405 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/62e02fd7-1e2e-4f92-9989-08bb7b819cfa-original-pull-secret podName:62e02fd7-1e2e-4f92-9989-08bb7b819cfa nodeName:}" failed. No retries permitted until 2026-07-18 11:49:26.169388572 +0000 UTC m=+33.249594925 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/62e02fd7-1e2e-4f92-9989-08bb7b819cfa-original-pull-secret") pod "global-pull-secret-syncer-sp74l" (UID: "62e02fd7-1e2e-4f92-9989-08bb7b819cfa") : object "kube-system"/"original-pull-secret" not registered Jul 18 11:49:10.169620 ip-10-0-136-240 kubenswrapper[2585]: E0718 11:49:10.169420 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4731170e-9d96-4544-a641-5fc8e1659d8e-metrics-certs podName:4731170e-9d96-4544-a641-5fc8e1659d8e nodeName:}" failed. No retries permitted until 2026-07-18 11:49:26.169413045 +0000 UTC m=+33.249619379 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4731170e-9d96-4544-a641-5fc8e1659d8e-metrics-certs") pod "network-metrics-daemon-hppk2" (UID: "4731170e-9d96-4544-a641-5fc8e1659d8e") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 18 11:49:10.269867 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:49:10.269832 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-frt9b\" (UniqueName: \"kubernetes.io/projected/c1cdc202-852a-4bc3-a68c-8f5bf5baab20-kube-api-access-frt9b\") pod \"network-check-target-vd8wb\" (UID: \"c1cdc202-852a-4bc3-a68c-8f5bf5baab20\") " pod="openshift-network-diagnostics/network-check-target-vd8wb" Jul 18 11:49:10.270022 ip-10-0-136-240 kubenswrapper[2585]: E0718 11:49:10.269992 2585 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 18 11:49:10.270022 ip-10-0-136-240 kubenswrapper[2585]: E0718 11:49:10.270013 2585 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 18 11:49:10.270022 ip-10-0-136-240 kubenswrapper[2585]: E0718 11:49:10.270022 2585 projected.go:196] Error preparing data for projected volume kube-api-access-frt9b for pod openshift-network-diagnostics/network-check-target-vd8wb: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 18 11:49:10.270127 ip-10-0-136-240 kubenswrapper[2585]: E0718 11:49:10.270075 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c1cdc202-852a-4bc3-a68c-8f5bf5baab20-kube-api-access-frt9b podName:c1cdc202-852a-4bc3-a68c-8f5bf5baab20 nodeName:}" failed. No retries permitted until 2026-07-18 11:49:26.270058623 +0000 UTC m=+33.350264958 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-frt9b" (UniqueName: "kubernetes.io/projected/c1cdc202-852a-4bc3-a68c-8f5bf5baab20-kube-api-access-frt9b") pod "network-check-target-vd8wb" (UID: "c1cdc202-852a-4bc3-a68c-8f5bf5baab20") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 18 11:49:10.462977 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:49:10.462887 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vd8wb" Jul 18 11:49:10.463151 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:49:10.462892 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sp74l" Jul 18 11:49:10.463151 ip-10-0-136-240 kubenswrapper[2585]: E0718 11:49:10.463005 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vd8wb" podUID="c1cdc202-852a-4bc3-a68c-8f5bf5baab20" Jul 18 11:49:10.463151 ip-10-0-136-240 kubenswrapper[2585]: E0718 11:49:10.463097 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sp74l" podUID="62e02fd7-1e2e-4f92-9989-08bb7b819cfa" Jul 18 11:49:11.463438 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:49:11.463415 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hppk2" Jul 18 11:49:11.463876 ip-10-0-136-240 kubenswrapper[2585]: E0718 11:49:11.463609 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hppk2" podUID="4731170e-9d96-4544-a641-5fc8e1659d8e" Jul 18 11:49:12.463231 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:49:12.463192 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sp74l" Jul 18 11:49:12.463399 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:49:12.463197 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vd8wb" Jul 18 11:49:12.463399 ip-10-0-136-240 kubenswrapper[2585]: E0718 11:49:12.463338 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sp74l" podUID="62e02fd7-1e2e-4f92-9989-08bb7b819cfa" Jul 18 11:49:12.463493 ip-10-0-136-240 kubenswrapper[2585]: E0718 11:49:12.463414 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vd8wb" podUID="c1cdc202-852a-4bc3-a68c-8f5bf5baab20" Jul 18 11:49:13.463568 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:49:13.463512 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hppk2" Jul 18 11:49:13.464033 ip-10-0-136-240 kubenswrapper[2585]: E0718 11:49:13.463651 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hppk2" podUID="4731170e-9d96-4544-a641-5fc8e1659d8e" Jul 18 11:49:14.462805 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:49:14.462766 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sp74l" Jul 18 11:49:14.462972 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:49:14.462766 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vd8wb" Jul 18 11:49:14.462972 ip-10-0-136-240 kubenswrapper[2585]: E0718 11:49:14.462887 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sp74l" podUID="62e02fd7-1e2e-4f92-9989-08bb7b819cfa" Jul 18 11:49:14.462972 ip-10-0-136-240 kubenswrapper[2585]: E0718 11:49:14.462951 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vd8wb" podUID="c1cdc202-852a-4bc3-a68c-8f5bf5baab20" Jul 18 11:49:15.462977 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:49:15.462951 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hppk2" Jul 18 11:49:15.463430 ip-10-0-136-240 kubenswrapper[2585]: E0718 11:49:15.463106 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hppk2" podUID="4731170e-9d96-4544-a641-5fc8e1659d8e" Jul 18 11:49:15.645777 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:49:15.645750 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-xlwqp" event={"ID":"3072709c-b577-46c4-9616-f0ca460c4196","Type":"ContainerStarted","Data":"0cc98f2dd01441d11fe63d53df8e4b816049261dfb656f3607f9a3845cc0891c"} Jul 18 11:49:15.647276 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:49:15.647251 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-240.ec2.internal" event={"ID":"a5cb0d1fcbe931b511b49dbd31969429","Type":"ContainerStarted","Data":"439e31691199517961967f7bc24c971b2f6448539a78a08943dc0f0abc863878"} Jul 18 11:49:15.648734 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:49:15.648712 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-c6jpl" event={"ID":"99561f5b-25bc-4089-b635-e8e896bfe43d","Type":"ContainerStarted","Data":"2a98a779d37fbac29a21b7bd64bbd6d891f153792c66f9cf566e499ac2a420f9"} Jul 18 11:49:15.653062 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:49:15.653042 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-wq4lg" event={"ID":"247c97c8-982b-4176-ae10-afc396745360","Type":"ContainerStarted","Data":"6265dbd1a963d1684dec47f4f30c071bad6758c6df62fe4d30fdde6021671f85"} Jul 18 11:49:15.678362 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:49:15.678325 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-c6jpl" podStartSLOduration=2.827543814 podStartE2EDuration="22.678311308s" totalImagesPullingTime="19.850767494s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-18 11:48:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-18 11:49:15.678308246 +0000 UTC m=+22.758514603" watchObservedRunningTime="2026-07-18 11:49:15.678311308 +0000 UTC m=+22.758517664" Jul 18 11:49:15.678606 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:49:15.678569 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-xlwqp" podStartSLOduration=2.463931641 podStartE2EDuration="22.678541105s" totalImagesPullingTime="20.214609464s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-18 11:48:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-18 11:49:15.662499105 +0000 UTC m=+22.742705461" watchObservedRunningTime="2026-07-18 11:49:15.678541105 +0000 UTC m=+22.758747514" Jul 18 11:49:15.690990 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:49:15.690955 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-240.ec2.internal" podStartSLOduration=21.690938687 podStartE2EDuration="21.690938687s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-18 11:48:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-18 11:49:15.690677387 +0000 UTC m=+22.770883745" watchObservedRunningTime="2026-07-18 11:49:15.690938687 +0000 UTC m=+22.771145114" Jul 18 11:49:16.463795 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:49:16.463305 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sp74l" Jul 18 11:49:16.463795 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:49:16.463349 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vd8wb" Jul 18 11:49:16.464510 ip-10-0-136-240 kubenswrapper[2585]: E0718 11:49:16.463844 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sp74l" podUID="62e02fd7-1e2e-4f92-9989-08bb7b819cfa" Jul 18 11:49:16.464510 ip-10-0-136-240 kubenswrapper[2585]: E0718 11:49:16.463938 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vd8wb" podUID="c1cdc202-852a-4bc3-a68c-8f5bf5baab20" Jul 18 11:49:16.656736 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:49:16.656704 2585 generic.go:356] "Generic (PLEG): container finished" podID="868b3db141f7bc43ba875f2543fbda86" containerID="908954c57a8a6efdae48c24f3c73551cdd09ed9d8a19332fbb8aa36814013152" exitCode=0 Jul 18 11:49:16.656961 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:49:16.656781 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-240.ec2.internal" event={"ID":"868b3db141f7bc43ba875f2543fbda86","Type":"ContainerDied","Data":"908954c57a8a6efdae48c24f3c73551cdd09ed9d8a19332fbb8aa36814013152"} Jul 18 11:49:16.660778 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:49:16.660749 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-hv6g8" event={"ID":"97228923-3172-4691-927f-723ef88035cf","Type":"ContainerStarted","Data":"2b9184a3ada4df3c2bac2dc77c8c5b471bd558bf3da472f2e3379c0dfe0cf65d"} Jul 18 11:49:16.663363 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:49:16.663344 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-wq4lg_247c97c8-982b-4176-ae10-afc396745360/ovn-acl-logging/0.log" Jul 18 11:49:16.663726 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:49:16.663700 2585 generic.go:356] "Generic (PLEG): container finished" podID="247c97c8-982b-4176-ae10-afc396745360" containerID="f6f80532b065885d7c13e093714509f86dc6ffab9dc1cc617404461c7022c6c0" exitCode=1 Jul 18 11:49:16.663832 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:49:16.663763 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-wq4lg" event={"ID":"247c97c8-982b-4176-ae10-afc396745360","Type":"ContainerStarted","Data":"84d7ed85e385b08e2771dced0d398eadb7c6b868a1239b7b767929aa5a85007a"} Jul 18 11:49:16.663894 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:49:16.663841 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-wq4lg" event={"ID":"247c97c8-982b-4176-ae10-afc396745360","Type":"ContainerStarted","Data":"939681ee5bf43763aa45de6b76235d64a5f0f9793b108c571fa175711d8ef875"} Jul 18 11:49:16.663894 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:49:16.663890 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-wq4lg" event={"ID":"247c97c8-982b-4176-ae10-afc396745360","Type":"ContainerStarted","Data":"1faacb005d53f9ee26b4274a5f8090229eb8a83445347f8665f48ce26d3f7522"} Jul 18 11:49:16.663984 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:49:16.663936 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-wq4lg" event={"ID":"247c97c8-982b-4176-ae10-afc396745360","Type":"ContainerStarted","Data":"7fc006a201ea890c14c57648f53fad2e632c211441903da43b92a53d709d48a6"} Jul 18 11:49:16.664032 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:49:16.663994 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-wq4lg" event={"ID":"247c97c8-982b-4176-ae10-afc396745360","Type":"ContainerDied","Data":"f6f80532b065885d7c13e093714509f86dc6ffab9dc1cc617404461c7022c6c0"} Jul 18 11:49:16.665587 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:49:16.665541 2585 generic.go:356] "Generic (PLEG): container finished" podID="760609c0-c9c8-44d2-87f4-7fd81b6eb7da" containerID="9997796a0e1b24f8b47bb85e6e078fbd3f1224469406a2956346a3102b3d05bb" exitCode=0 Jul 18 11:49:16.665683 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:49:16.665583 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-9xp94" event={"ID":"760609c0-c9c8-44d2-87f4-7fd81b6eb7da","Type":"ContainerDied","Data":"9997796a0e1b24f8b47bb85e6e078fbd3f1224469406a2956346a3102b3d05bb"} Jul 18 11:49:16.667280 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:49:16.667238 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-nzpfs" event={"ID":"d12e3a2e-a05d-4659-b74e-c30e765448bf","Type":"ContainerStarted","Data":"d7ef831ef20b50d78b61bc32fbcb59565d130d8a3b3de38777a3c3d22c73be6d"} Jul 18 11:49:16.668715 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:49:16.668689 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fjjlw" event={"ID":"9e7851c1-2896-4b77-9f84-cb29edf29b87","Type":"ContainerStarted","Data":"d0c94c1b8f1ae331a2bbf5d36d5f09c412a3c7f7c299a999bddb3611742070ab"} Jul 18 11:49:16.670071 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:49:16.670048 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-vrm8v" event={"ID":"c11da53c-730b-4a02-b2f6-fbf3ee1a6cbd","Type":"ContainerStarted","Data":"b2e1fc59330f833620e655054c8ffd043167349541e38ecf3bbb5ac8214418dc"} Jul 18 11:49:16.671522 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:49:16.671493 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-844xq" event={"ID":"a2e064ca-7d18-4e4a-952c-c00aebf5deb4","Type":"ContainerStarted","Data":"1d0fcc4416c5c208ffcfc088dd641cd3c408634566ce00e416aebe45e8d78c18"} Jul 18 11:49:16.696677 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:49:16.696629 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-844xq" podStartSLOduration=3.692406225 podStartE2EDuration="23.696614152s" totalImagesPullingTime="20.004207927s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-18 11:48:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-18 11:49:16.696357284 +0000 UTC m=+23.776563641" watchObservedRunningTime="2026-07-18 11:49:16.696614152 +0000 UTC m=+23.776820511" Jul 18 11:49:16.730417 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:49:16.730317 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-vrm8v" podStartSLOduration=3.821907774 podStartE2EDuration="23.730299596s" totalImagesPullingTime="19.908391822s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-18 11:48:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-18 11:49:16.709626537 +0000 UTC m=+23.789832913" watchObservedRunningTime="2026-07-18 11:49:16.730299596 +0000 UTC m=+23.810505954" Jul 18 11:49:16.757227 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:49:16.757177 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-hv6g8" podStartSLOduration=3.828422638 podStartE2EDuration="23.757160568s" totalImagesPullingTime="19.92873793s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-18 11:48:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-18 11:49:16.743321389 +0000 UTC m=+23.823527770" watchObservedRunningTime="2026-07-18 11:49:16.757160568 +0000 UTC m=+23.837366919" Jul 18 11:49:16.757365 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:49:16.757293 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-nzpfs" podStartSLOduration=3.873551661 podStartE2EDuration="23.757286412s" totalImagesPullingTime="19.883734751s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-18 11:48:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-18 11:49:16.757145612 +0000 UTC m=+23.837351969" watchObservedRunningTime="2026-07-18 11:49:16.757286412 +0000 UTC m=+23.837492768" Jul 18 11:49:17.184774 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:49:17.184742 2585 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Jul 18 11:49:17.403878 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:49:17.403679 2585 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-18T11:49:17.184759362Z","UUID":"334cc09e-d367-4877-a054-cc71a6e524ca","Handler":null,"Name":"","Endpoint":""} Jul 18 11:49:17.405245 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:49:17.405224 2585 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Jul 18 11:49:17.405365 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:49:17.405265 2585 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Jul 18 11:49:17.463449 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:49:17.463419 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hppk2" Jul 18 11:49:17.463621 ip-10-0-136-240 kubenswrapper[2585]: E0718 11:49:17.463543 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hppk2" podUID="4731170e-9d96-4544-a641-5fc8e1659d8e" Jul 18 11:49:17.666239 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:49:17.666170 2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-844xq" Jul 18 11:49:17.666884 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:49:17.666869 2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-844xq" Jul 18 11:49:17.676825 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:49:17.676796 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-240.ec2.internal" event={"ID":"868b3db141f7bc43ba875f2543fbda86","Type":"ContainerStarted","Data":"b146e3c60de549e5d8470335a2fa0177e5c30d8bb28e25eb4b3391656df8f081"} Jul 18 11:49:17.680011 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:49:17.679984 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fjjlw" event={"ID":"9e7851c1-2896-4b77-9f84-cb29edf29b87","Type":"ContainerStarted","Data":"764133527faa6139dd8502f710d2796767a4e5b71bb83b6e2dd6beee026c040f"} Jul 18 11:49:17.680883 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:49:17.680864 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-844xq" Jul 18 11:49:17.681232 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:49:17.681215 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-844xq" Jul 18 11:49:17.703083 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:49:17.702933 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-240.ec2.internal" podStartSLOduration=23.702913074 podStartE2EDuration="23.702913074s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-18 11:48:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-18 11:49:17.702174018 +0000 UTC m=+24.782380377" watchObservedRunningTime="2026-07-18 11:49:17.702913074 +0000 UTC m=+24.783119434" Jul 18 11:49:18.462979 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:49:18.462949 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sp74l" Jul 18 11:49:18.463107 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:49:18.462987 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vd8wb" Jul 18 11:49:18.463107 ip-10-0-136-240 kubenswrapper[2585]: E0718 11:49:18.463052 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sp74l" podUID="62e02fd7-1e2e-4f92-9989-08bb7b819cfa" Jul 18 11:49:18.463364 ip-10-0-136-240 kubenswrapper[2585]: E0718 11:49:18.463344 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vd8wb" podUID="c1cdc202-852a-4bc3-a68c-8f5bf5baab20" Jul 18 11:49:18.685222 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:49:18.685150 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-wq4lg_247c97c8-982b-4176-ae10-afc396745360/ovn-acl-logging/0.log" Jul 18 11:49:18.685693 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:49:18.685659 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-wq4lg" event={"ID":"247c97c8-982b-4176-ae10-afc396745360","Type":"ContainerStarted","Data":"da5eb15a241a69343095a6d90455572bcdc29b97c27791263e7a4a43b65cbe02"} Jul 18 11:49:18.687953 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:49:18.687923 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fjjlw" event={"ID":"9e7851c1-2896-4b77-9f84-cb29edf29b87","Type":"ContainerStarted","Data":"927f7650e586ac374061923986c07a7eeca3b034929b25f4fe61f1ea7c6e5f89"} Jul 18 11:49:18.705945 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:49:18.705895 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fjjlw" podStartSLOduration=3.531279222 podStartE2EDuration="25.705880141s" totalImagesPullingTime="22.174600919s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-18 11:48:53 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-18 11:49:18.705527529 +0000 UTC m=+25.785733913" watchObservedRunningTime="2026-07-18 11:49:18.705880141 +0000 UTC m=+25.786086499" Jul 18 11:49:19.463493 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:49:19.463463 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hppk2" Jul 18 11:49:19.463687 ip-10-0-136-240 kubenswrapper[2585]: E0718 11:49:19.463618 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hppk2" podUID="4731170e-9d96-4544-a641-5fc8e1659d8e" Jul 18 11:49:20.463656 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:49:20.463613 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sp74l" Jul 18 11:49:20.464076 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:49:20.463613 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vd8wb" Jul 18 11:49:20.464076 ip-10-0-136-240 kubenswrapper[2585]: E0718 11:49:20.463757 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sp74l" podUID="62e02fd7-1e2e-4f92-9989-08bb7b819cfa" Jul 18 11:49:20.464076 ip-10-0-136-240 kubenswrapper[2585]: E0718 11:49:20.463839 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vd8wb" podUID="c1cdc202-852a-4bc3-a68c-8f5bf5baab20" Jul 18 11:49:21.463657 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:49:21.463462 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hppk2" Jul 18 11:49:21.463809 ip-10-0-136-240 kubenswrapper[2585]: E0718 11:49:21.463736 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hppk2" podUID="4731170e-9d96-4544-a641-5fc8e1659d8e" Jul 18 11:49:21.697655 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:49:21.697607 2585 generic.go:356] "Generic (PLEG): container finished" podID="760609c0-c9c8-44d2-87f4-7fd81b6eb7da" containerID="c9f9a1e09f0bac011bf97cafe22fb997c37bd8cb17b32671b5217acbed2a5299" exitCode=0 Jul 18 11:49:21.697852 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:49:21.697679 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-9xp94" event={"ID":"760609c0-c9c8-44d2-87f4-7fd81b6eb7da","Type":"ContainerDied","Data":"c9f9a1e09f0bac011bf97cafe22fb997c37bd8cb17b32671b5217acbed2a5299"} Jul 18 11:49:21.701949 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:49:21.701918 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-wq4lg_247c97c8-982b-4176-ae10-afc396745360/ovn-acl-logging/0.log" Jul 18 11:49:21.702312 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:49:21.702292 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-wq4lg" event={"ID":"247c97c8-982b-4176-ae10-afc396745360","Type":"ContainerStarted","Data":"223a55339a1fbdf54c7c470a7ed50e0ea567a73aa44a48fd55a96225b571dc1f"} Jul 18 11:49:21.702568 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:49:21.702537 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-wq4lg" Jul 18 11:49:21.702737 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:49:21.702723 2585 scope.go:117] "RemoveContainer" containerID="f6f80532b065885d7c13e093714509f86dc6ffab9dc1cc617404461c7022c6c0" Jul 18 11:49:21.720479 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:49:21.720458 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-wq4lg" Jul 18 11:49:22.463676 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:49:22.463476 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vd8wb" Jul 18 11:49:22.463886 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:49:22.463525 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sp74l" Jul 18 11:49:22.463886 ip-10-0-136-240 kubenswrapper[2585]: E0718 11:49:22.463766 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vd8wb" podUID="c1cdc202-852a-4bc3-a68c-8f5bf5baab20" Jul 18 11:49:22.463886 ip-10-0-136-240 kubenswrapper[2585]: E0718 11:49:22.463814 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sp74l" podUID="62e02fd7-1e2e-4f92-9989-08bb7b819cfa" Jul 18 11:49:22.707971 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:49:22.707941 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-wq4lg_247c97c8-982b-4176-ae10-afc396745360/ovn-acl-logging/0.log" Jul 18 11:49:22.708289 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:49:22.708268 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-wq4lg" event={"ID":"247c97c8-982b-4176-ae10-afc396745360","Type":"ContainerStarted","Data":"5b107153924d43c655a2a00da35c293fd1898426d71b4bcd1b0f50833f8dc276"} Jul 18 11:49:22.708472 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:49:22.708454 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-wq4lg" Jul 18 11:49:22.708664 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:49:22.708639 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-wq4lg" Jul 18 11:49:22.724420 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:49:22.724355 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-wq4lg" Jul 18 11:49:22.737354 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:49:22.737313 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-wq4lg" podStartSLOduration=8.99291194 podStartE2EDuration="29.737301233s" totalImagesPullingTime="20.744389293s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-18 11:48:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-18 11:49:22.737087277 +0000 UTC m=+29.817293656" watchObservedRunningTime="2026-07-18 11:49:22.737301233 +0000 UTC m=+29.817507589" Jul 18 11:49:22.774193 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:49:22.774172 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-sp74l"] Jul 18 11:49:22.774306 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:49:22.774284 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sp74l" Jul 18 11:49:22.774423 ip-10-0-136-240 kubenswrapper[2585]: E0718 11:49:22.774403 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sp74l" podUID="62e02fd7-1e2e-4f92-9989-08bb7b819cfa" Jul 18 11:49:22.777430 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:49:22.777407 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-vd8wb"] Jul 18 11:49:22.777592 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:49:22.777576 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vd8wb" Jul 18 11:49:22.777702 ip-10-0-136-240 kubenswrapper[2585]: E0718 11:49:22.777682 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vd8wb" podUID="c1cdc202-852a-4bc3-a68c-8f5bf5baab20" Jul 18 11:49:22.777985 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:49:22.777966 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-hppk2"] Jul 18 11:49:22.778082 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:49:22.778072 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hppk2" Jul 18 11:49:22.778168 ip-10-0-136-240 kubenswrapper[2585]: E0718 11:49:22.778152 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hppk2" podUID="4731170e-9d96-4544-a641-5fc8e1659d8e" Jul 18 11:49:23.712416 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:49:23.712387 2585 generic.go:356] "Generic (PLEG): container finished" podID="760609c0-c9c8-44d2-87f4-7fd81b6eb7da" containerID="e8b6518a1e2cc4e598771eb35a2cb74f858c410ee32d364d2d209612dc6a86d6" exitCode=0 Jul 18 11:49:23.712845 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:49:23.712474 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-9xp94" event={"ID":"760609c0-c9c8-44d2-87f4-7fd81b6eb7da","Type":"ContainerDied","Data":"e8b6518a1e2cc4e598771eb35a2cb74f858c410ee32d364d2d209612dc6a86d6"} Jul 18 11:49:24.463622 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:49:24.463533 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sp74l" Jul 18 11:49:24.463622 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:49:24.463569 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hppk2" Jul 18 11:49:24.463846 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:49:24.463675 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vd8wb" Jul 18 11:49:24.463846 ip-10-0-136-240 kubenswrapper[2585]: E0718 11:49:24.463676 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sp74l" podUID="62e02fd7-1e2e-4f92-9989-08bb7b819cfa" Jul 18 11:49:24.463846 ip-10-0-136-240 kubenswrapper[2585]: E0718 11:49:24.463813 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hppk2" podUID="4731170e-9d96-4544-a641-5fc8e1659d8e" Jul 18 11:49:24.464027 ip-10-0-136-240 kubenswrapper[2585]: E0718 11:49:24.463913 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vd8wb" podUID="c1cdc202-852a-4bc3-a68c-8f5bf5baab20" Jul 18 11:49:25.718972 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:49:25.718798 2585 generic.go:356] "Generic (PLEG): container finished" podID="760609c0-c9c8-44d2-87f4-7fd81b6eb7da" containerID="ce598c03e31dd2eb6b9621723da00ad1b40b17b9b5ee8236ec9dc1bdfdf89f6d" exitCode=0 Jul 18 11:49:25.719343 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:49:25.718886 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-9xp94" event={"ID":"760609c0-c9c8-44d2-87f4-7fd81b6eb7da","Type":"ContainerDied","Data":"ce598c03e31dd2eb6b9621723da00ad1b40b17b9b5ee8236ec9dc1bdfdf89f6d"} Jul 18 11:49:26.187519 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:49:26.187433 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/62e02fd7-1e2e-4f92-9989-08bb7b819cfa-original-pull-secret\") pod \"global-pull-secret-syncer-sp74l\" (UID: \"62e02fd7-1e2e-4f92-9989-08bb7b819cfa\") " pod="kube-system/global-pull-secret-syncer-sp74l" Jul 18 11:49:26.187519 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:49:26.187502 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4731170e-9d96-4544-a641-5fc8e1659d8e-metrics-certs\") pod \"network-metrics-daemon-hppk2\" (UID: \"4731170e-9d96-4544-a641-5fc8e1659d8e\") " pod="openshift-multus/network-metrics-daemon-hppk2" Jul 18 11:49:26.187804 ip-10-0-136-240 kubenswrapper[2585]: E0718 11:49:26.187619 2585 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 18 11:49:26.187804 ip-10-0-136-240 kubenswrapper[2585]: E0718 11:49:26.187625 2585 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 18 11:49:26.187804 ip-10-0-136-240 kubenswrapper[2585]: E0718 11:49:26.187697 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/62e02fd7-1e2e-4f92-9989-08bb7b819cfa-original-pull-secret podName:62e02fd7-1e2e-4f92-9989-08bb7b819cfa nodeName:}" failed. No retries permitted until 2026-07-18 11:49:58.187677545 +0000 UTC m=+65.267883883 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/62e02fd7-1e2e-4f92-9989-08bb7b819cfa-original-pull-secret") pod "global-pull-secret-syncer-sp74l" (UID: "62e02fd7-1e2e-4f92-9989-08bb7b819cfa") : object "kube-system"/"original-pull-secret" not registered Jul 18 11:49:26.187804 ip-10-0-136-240 kubenswrapper[2585]: E0718 11:49:26.187713 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4731170e-9d96-4544-a641-5fc8e1659d8e-metrics-certs podName:4731170e-9d96-4544-a641-5fc8e1659d8e nodeName:}" failed. No retries permitted until 2026-07-18 11:49:58.187707106 +0000 UTC m=+65.267913440 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4731170e-9d96-4544-a641-5fc8e1659d8e-metrics-certs") pod "network-metrics-daemon-hppk2" (UID: "4731170e-9d96-4544-a641-5fc8e1659d8e") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 18 11:49:26.287889 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:49:26.287849 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-frt9b\" (UniqueName: \"kubernetes.io/projected/c1cdc202-852a-4bc3-a68c-8f5bf5baab20-kube-api-access-frt9b\") pod \"network-check-target-vd8wb\" (UID: \"c1cdc202-852a-4bc3-a68c-8f5bf5baab20\") " pod="openshift-network-diagnostics/network-check-target-vd8wb" Jul 18 11:49:26.288280 ip-10-0-136-240 kubenswrapper[2585]: E0718 11:49:26.288245 2585 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 18 11:49:26.288410 ip-10-0-136-240 kubenswrapper[2585]: E0718 11:49:26.288290 2585 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 18 11:49:26.288410 ip-10-0-136-240 kubenswrapper[2585]: E0718 11:49:26.288305 2585 projected.go:196] Error preparing data for projected volume kube-api-access-frt9b for pod openshift-network-diagnostics/network-check-target-vd8wb: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 18 11:49:26.288410 ip-10-0-136-240 kubenswrapper[2585]: E0718 11:49:26.288386 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c1cdc202-852a-4bc3-a68c-8f5bf5baab20-kube-api-access-frt9b podName:c1cdc202-852a-4bc3-a68c-8f5bf5baab20 nodeName:}" failed. No retries permitted until 2026-07-18 11:49:58.288359272 +0000 UTC m=+65.368565622 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-frt9b" (UniqueName: "kubernetes.io/projected/c1cdc202-852a-4bc3-a68c-8f5bf5baab20-kube-api-access-frt9b") pod "network-check-target-vd8wb" (UID: "c1cdc202-852a-4bc3-a68c-8f5bf5baab20") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 18 11:49:26.462821 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:49:26.462778 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sp74l" Jul 18 11:49:26.463004 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:49:26.462802 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vd8wb" Jul 18 11:49:26.463004 ip-10-0-136-240 kubenswrapper[2585]: E0718 11:49:26.462915 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sp74l" podUID="62e02fd7-1e2e-4f92-9989-08bb7b819cfa" Jul 18 11:49:26.463109 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:49:26.462802 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hppk2" Jul 18 11:49:26.463109 ip-10-0-136-240 kubenswrapper[2585]: E0718 11:49:26.463016 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vd8wb" podUID="c1cdc202-852a-4bc3-a68c-8f5bf5baab20" Jul 18 11:49:26.463202 ip-10-0-136-240 kubenswrapper[2585]: E0718 11:49:26.463157 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hppk2" podUID="4731170e-9d96-4544-a641-5fc8e1659d8e" Jul 18 11:49:28.463487 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:49:28.463451 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sp74l" Jul 18 11:49:28.463487 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:49:28.463473 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hppk2" Jul 18 11:49:28.464002 ip-10-0-136-240 kubenswrapper[2585]: E0718 11:49:28.463594 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sp74l" podUID="62e02fd7-1e2e-4f92-9989-08bb7b819cfa" Jul 18 11:49:28.464002 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:49:28.463636 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vd8wb" Jul 18 11:49:28.464002 ip-10-0-136-240 kubenswrapper[2585]: E0718 11:49:28.463728 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hppk2" podUID="4731170e-9d96-4544-a641-5fc8e1659d8e" Jul 18 11:49:28.464002 ip-10-0-136-240 kubenswrapper[2585]: E0718 11:49:28.463813 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vd8wb" podUID="c1cdc202-852a-4bc3-a68c-8f5bf5baab20" Jul 18 11:49:29.236504 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:49:29.236476 2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-240.ec2.internal" event="NodeReady" Jul 18 11:49:29.236685 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:49:29.236653 2585 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Jul 18 11:49:29.279248 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:49:29.279212 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-mrh2r"] Jul 18 11:49:29.282415 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:49:29.282388 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-88757"] Jul 18 11:49:29.282583 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:49:29.282532 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-mrh2r" Jul 18 11:49:29.283988 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:49:29.283967 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-88757" Jul 18 11:49:29.285159 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:49:29.285131 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Jul 18 11:49:29.285262 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:49:29.285176 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-qxztp\"" Jul 18 11:49:29.285862 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:49:29.285838 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Jul 18 11:49:29.286247 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:49:29.286227 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-fbzht\"" Jul 18 11:49:29.286374 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:49:29.286352 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Jul 18 11:49:29.287273 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:49:29.286353 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Jul 18 11:49:29.287273 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:49:29.286519 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Jul 18 11:49:29.296272 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:49:29.294467 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-mrh2r"] Jul 18 11:49:29.296378 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:49:29.296307 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-88757"] Jul 18 11:49:29.413102 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:49:29.413070 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sjtgg\" (UniqueName: \"kubernetes.io/projected/056ad49c-5e1c-4331-8a8f-65c7202be995-kube-api-access-sjtgg\") pod \"ingress-canary-88757\" (UID: \"056ad49c-5e1c-4331-8a8f-65c7202be995\") " pod="openshift-ingress-canary/ingress-canary-88757" Jul 18 11:49:29.413270 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:49:29.413110 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/42cb6807-1262-4919-856b-80c7b6d08ae2-config-volume\") pod \"dns-default-mrh2r\" (UID: \"42cb6807-1262-4919-856b-80c7b6d08ae2\") " pod="openshift-dns/dns-default-mrh2r" Jul 18 11:49:29.413270 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:49:29.413220 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/42cb6807-1262-4919-856b-80c7b6d08ae2-metrics-tls\") pod \"dns-default-mrh2r\" (UID: \"42cb6807-1262-4919-856b-80c7b6d08ae2\") " pod="openshift-dns/dns-default-mrh2r" Jul 18 11:49:29.413270 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:49:29.413263 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/42cb6807-1262-4919-856b-80c7b6d08ae2-tmp-dir\") pod \"dns-default-mrh2r\" (UID: \"42cb6807-1262-4919-856b-80c7b6d08ae2\") " pod="openshift-dns/dns-default-mrh2r" Jul 18 11:49:29.413389 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:49:29.413284 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-22wcr\" (UniqueName: \"kubernetes.io/projected/42cb6807-1262-4919-856b-80c7b6d08ae2-kube-api-access-22wcr\") pod \"dns-default-mrh2r\" (UID: \"42cb6807-1262-4919-856b-80c7b6d08ae2\") " pod="openshift-dns/dns-default-mrh2r" Jul 18 11:49:29.413389 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:49:29.413311 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/056ad49c-5e1c-4331-8a8f-65c7202be995-cert\") pod \"ingress-canary-88757\" (UID: \"056ad49c-5e1c-4331-8a8f-65c7202be995\") " pod="openshift-ingress-canary/ingress-canary-88757" Jul 18 11:49:29.513666 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:49:29.513579 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/42cb6807-1262-4919-856b-80c7b6d08ae2-tmp-dir\") pod \"dns-default-mrh2r\" (UID: \"42cb6807-1262-4919-856b-80c7b6d08ae2\") " pod="openshift-dns/dns-default-mrh2r" Jul 18 11:49:29.513666 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:49:29.513621 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-22wcr\" (UniqueName: \"kubernetes.io/projected/42cb6807-1262-4919-856b-80c7b6d08ae2-kube-api-access-22wcr\") pod \"dns-default-mrh2r\" (UID: \"42cb6807-1262-4919-856b-80c7b6d08ae2\") " pod="openshift-dns/dns-default-mrh2r" Jul 18 11:49:29.514262 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:49:29.513660 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/056ad49c-5e1c-4331-8a8f-65c7202be995-cert\") pod \"ingress-canary-88757\" (UID: \"056ad49c-5e1c-4331-8a8f-65c7202be995\") " pod="openshift-ingress-canary/ingress-canary-88757" Jul 18 11:49:29.514262 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:49:29.513731 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sjtgg\" (UniqueName: \"kubernetes.io/projected/056ad49c-5e1c-4331-8a8f-65c7202be995-kube-api-access-sjtgg\") pod \"ingress-canary-88757\" (UID: \"056ad49c-5e1c-4331-8a8f-65c7202be995\") " pod="openshift-ingress-canary/ingress-canary-88757" Jul 18 11:49:29.514262 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:49:29.513769 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/42cb6807-1262-4919-856b-80c7b6d08ae2-config-volume\") pod \"dns-default-mrh2r\" (UID: \"42cb6807-1262-4919-856b-80c7b6d08ae2\") " pod="openshift-dns/dns-default-mrh2r" Jul 18 11:49:29.514262 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:49:29.513818 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/42cb6807-1262-4919-856b-80c7b6d08ae2-metrics-tls\") pod \"dns-default-mrh2r\" (UID: \"42cb6807-1262-4919-856b-80c7b6d08ae2\") " pod="openshift-dns/dns-default-mrh2r" Jul 18 11:49:29.514262 ip-10-0-136-240 kubenswrapper[2585]: E0718 11:49:29.513913 2585 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 18 11:49:29.514262 ip-10-0-136-240 kubenswrapper[2585]: E0718 11:49:29.513971 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/42cb6807-1262-4919-856b-80c7b6d08ae2-metrics-tls podName:42cb6807-1262-4919-856b-80c7b6d08ae2 nodeName:}" failed. No retries permitted until 2026-07-18 11:49:30.013952173 +0000 UTC m=+37.094158511 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/42cb6807-1262-4919-856b-80c7b6d08ae2-metrics-tls") pod "dns-default-mrh2r" (UID: "42cb6807-1262-4919-856b-80c7b6d08ae2") : secret "dns-default-metrics-tls" not found Jul 18 11:49:29.514262 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:49:29.513973 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/42cb6807-1262-4919-856b-80c7b6d08ae2-tmp-dir\") pod \"dns-default-mrh2r\" (UID: \"42cb6807-1262-4919-856b-80c7b6d08ae2\") " pod="openshift-dns/dns-default-mrh2r" Jul 18 11:49:29.514571 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:49:29.514461 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/42cb6807-1262-4919-856b-80c7b6d08ae2-config-volume\") pod \"dns-default-mrh2r\" (UID: \"42cb6807-1262-4919-856b-80c7b6d08ae2\") " pod="openshift-dns/dns-default-mrh2r" Jul 18 11:49:29.518531 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:49:29.518504 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/056ad49c-5e1c-4331-8a8f-65c7202be995-cert\") pod \"ingress-canary-88757\" (UID: \"056ad49c-5e1c-4331-8a8f-65c7202be995\") " pod="openshift-ingress-canary/ingress-canary-88757" Jul 18 11:49:29.523646 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:49:29.523597 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-22wcr\" (UniqueName: \"kubernetes.io/projected/42cb6807-1262-4919-856b-80c7b6d08ae2-kube-api-access-22wcr\") pod \"dns-default-mrh2r\" (UID: \"42cb6807-1262-4919-856b-80c7b6d08ae2\") " pod="openshift-dns/dns-default-mrh2r" Jul 18 11:49:29.523768 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:49:29.523660 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sjtgg\" (UniqueName: \"kubernetes.io/projected/056ad49c-5e1c-4331-8a8f-65c7202be995-kube-api-access-sjtgg\") pod \"ingress-canary-88757\" (UID: \"056ad49c-5e1c-4331-8a8f-65c7202be995\") " pod="openshift-ingress-canary/ingress-canary-88757" Jul 18 11:49:29.603219 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:49:29.603198 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-88757" Jul 18 11:49:29.732296 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:49:29.732049 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-9xp94" event={"ID":"760609c0-c9c8-44d2-87f4-7fd81b6eb7da","Type":"ContainerStarted","Data":"4e4955dac677464163037777e2b4b1cd2165884fc9a9050eb0fe5206919c3a62"} Jul 18 11:49:29.753149 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:49:29.753094 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-88757"] Jul 18 11:49:29.756934 ip-10-0-136-240 kubenswrapper[2585]: W0718 11:49:29.756907 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod056ad49c_5e1c_4331_8a8f_65c7202be995.slice/crio-a426957d7a1bd03fba06d0dfcd29c0427bdc0d312fdd0c083806784389d9c73f WatchSource:0}: Error finding container a426957d7a1bd03fba06d0dfcd29c0427bdc0d312fdd0c083806784389d9c73f: Status 404 returned error can't find the container with id a426957d7a1bd03fba06d0dfcd29c0427bdc0d312fdd0c083806784389d9c73f Jul 18 11:49:30.017517 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:49:30.017438 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/42cb6807-1262-4919-856b-80c7b6d08ae2-metrics-tls\") pod \"dns-default-mrh2r\" (UID: \"42cb6807-1262-4919-856b-80c7b6d08ae2\") " pod="openshift-dns/dns-default-mrh2r" Jul 18 11:49:30.017676 ip-10-0-136-240 kubenswrapper[2585]: E0718 11:49:30.017586 2585 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 18 11:49:30.017676 ip-10-0-136-240 kubenswrapper[2585]: E0718 11:49:30.017641 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/42cb6807-1262-4919-856b-80c7b6d08ae2-metrics-tls podName:42cb6807-1262-4919-856b-80c7b6d08ae2 nodeName:}" failed. No retries permitted until 2026-07-18 11:49:31.017625905 +0000 UTC m=+38.097832240 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/42cb6807-1262-4919-856b-80c7b6d08ae2-metrics-tls") pod "dns-default-mrh2r" (UID: "42cb6807-1262-4919-856b-80c7b6d08ae2") : secret "dns-default-metrics-tls" not found Jul 18 11:49:30.463380 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:49:30.463347 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hppk2" Jul 18 11:49:30.463541 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:49:30.463346 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vd8wb" Jul 18 11:49:30.463541 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:49:30.463346 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sp74l" Jul 18 11:49:30.467173 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:49:30.467143 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 18 11:49:30.467299 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:49:30.467198 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 18 11:49:30.467299 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:49:30.467219 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 18 11:49:30.467299 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:49:30.467227 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-7qw87\"" Jul 18 11:49:30.467299 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:49:30.467274 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 18 11:49:30.467495 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:49:30.467216 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-6q4p9\"" Jul 18 11:49:30.735796 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:49:30.735659 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-88757" event={"ID":"056ad49c-5e1c-4331-8a8f-65c7202be995","Type":"ContainerStarted","Data":"a426957d7a1bd03fba06d0dfcd29c0427bdc0d312fdd0c083806784389d9c73f"} Jul 18 11:49:30.739251 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:49:30.738949 2585 generic.go:356] "Generic (PLEG): container finished" podID="760609c0-c9c8-44d2-87f4-7fd81b6eb7da" containerID="4e4955dac677464163037777e2b4b1cd2165884fc9a9050eb0fe5206919c3a62" exitCode=0 Jul 18 11:49:30.739251 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:49:30.738995 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-9xp94" event={"ID":"760609c0-c9c8-44d2-87f4-7fd81b6eb7da","Type":"ContainerDied","Data":"4e4955dac677464163037777e2b4b1cd2165884fc9a9050eb0fe5206919c3a62"} Jul 18 11:49:31.026306 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:49:31.026148 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/42cb6807-1262-4919-856b-80c7b6d08ae2-metrics-tls\") pod \"dns-default-mrh2r\" (UID: \"42cb6807-1262-4919-856b-80c7b6d08ae2\") " pod="openshift-dns/dns-default-mrh2r" Jul 18 11:49:31.026306 ip-10-0-136-240 kubenswrapper[2585]: E0718 11:49:31.026270 2585 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 18 11:49:31.026606 ip-10-0-136-240 kubenswrapper[2585]: E0718 11:49:31.026343 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/42cb6807-1262-4919-856b-80c7b6d08ae2-metrics-tls podName:42cb6807-1262-4919-856b-80c7b6d08ae2 nodeName:}" failed. No retries permitted until 2026-07-18 11:49:33.026323644 +0000 UTC m=+40.106529979 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/42cb6807-1262-4919-856b-80c7b6d08ae2-metrics-tls") pod "dns-default-mrh2r" (UID: "42cb6807-1262-4919-856b-80c7b6d08ae2") : secret "dns-default-metrics-tls" not found Jul 18 11:49:31.745706 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:49:31.745663 2585 generic.go:356] "Generic (PLEG): container finished" podID="760609c0-c9c8-44d2-87f4-7fd81b6eb7da" containerID="5a9307415478e47ce28d4976576ec68df183d740c66c7d4d528a6d4a5eb5a345" exitCode=0 Jul 18 11:49:31.746182 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:49:31.745749 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-9xp94" event={"ID":"760609c0-c9c8-44d2-87f4-7fd81b6eb7da","Type":"ContainerDied","Data":"5a9307415478e47ce28d4976576ec68df183d740c66c7d4d528a6d4a5eb5a345"} Jul 18 11:49:32.749858 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:49:32.749641 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-88757" event={"ID":"056ad49c-5e1c-4331-8a8f-65c7202be995","Type":"ContainerStarted","Data":"a94d1096a7ff6198afa735c6cf9644971ded5ecbbb38718bfac6fdf86d0abb93"} Jul 18 11:49:32.753000 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:49:32.752971 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-9xp94" event={"ID":"760609c0-c9c8-44d2-87f4-7fd81b6eb7da","Type":"ContainerStarted","Data":"fe72dd934d3186e2dfb2c05e6425d87e54d50b2edb79a4f101524db868ea3c68"} Jul 18 11:49:32.764399 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:49:32.764316 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-88757" podStartSLOduration=1.441833589 podStartE2EDuration="3.76430095s" totalImagesPullingTime="2.322467361s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-18 11:49:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-18 11:49:32.763817273 +0000 UTC m=+39.844023630" watchObservedRunningTime="2026-07-18 11:49:32.76430095 +0000 UTC m=+39.844507308" Jul 18 11:49:32.785226 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:49:32.785191 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-9xp94" podStartSLOduration=9.482759013 podStartE2EDuration="39.785177045s" totalImagesPullingTime="30.302418032s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-18 11:48:53 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-18 11:49:32.784131227 +0000 UTC m=+39.864337584" watchObservedRunningTime="2026-07-18 11:49:32.785177045 +0000 UTC m=+39.865383402" Jul 18 11:49:33.041667 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:49:33.041589 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/42cb6807-1262-4919-856b-80c7b6d08ae2-metrics-tls\") pod \"dns-default-mrh2r\" (UID: \"42cb6807-1262-4919-856b-80c7b6d08ae2\") " pod="openshift-dns/dns-default-mrh2r" Jul 18 11:49:33.041807 ip-10-0-136-240 kubenswrapper[2585]: E0718 11:49:33.041736 2585 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 18 11:49:33.041807 ip-10-0-136-240 kubenswrapper[2585]: E0718 11:49:33.041800 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/42cb6807-1262-4919-856b-80c7b6d08ae2-metrics-tls podName:42cb6807-1262-4919-856b-80c7b6d08ae2 nodeName:}" failed. No retries permitted until 2026-07-18 11:49:37.041783481 +0000 UTC m=+44.121989816 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/42cb6807-1262-4919-856b-80c7b6d08ae2-metrics-tls") pod "dns-default-mrh2r" (UID: "42cb6807-1262-4919-856b-80c7b6d08ae2") : secret "dns-default-metrics-tls" not found Jul 18 11:49:37.068328 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:49:37.068285 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/42cb6807-1262-4919-856b-80c7b6d08ae2-metrics-tls\") pod \"dns-default-mrh2r\" (UID: \"42cb6807-1262-4919-856b-80c7b6d08ae2\") " pod="openshift-dns/dns-default-mrh2r" Jul 18 11:49:37.068738 ip-10-0-136-240 kubenswrapper[2585]: E0718 11:49:37.068446 2585 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 18 11:49:37.068738 ip-10-0-136-240 kubenswrapper[2585]: E0718 11:49:37.068509 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/42cb6807-1262-4919-856b-80c7b6d08ae2-metrics-tls podName:42cb6807-1262-4919-856b-80c7b6d08ae2 nodeName:}" failed. No retries permitted until 2026-07-18 11:49:45.068493476 +0000 UTC m=+52.148699811 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/42cb6807-1262-4919-856b-80c7b6d08ae2-metrics-tls") pod "dns-default-mrh2r" (UID: "42cb6807-1262-4919-856b-80c7b6d08ae2") : secret "dns-default-metrics-tls" not found Jul 18 11:49:41.667981 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:49:41.667947 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-78cf4c6f6-gftqx"] Jul 18 11:49:41.684143 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:49:41.684107 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-555d9d7885-44h2s"] Jul 18 11:49:41.684342 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:49:41.684321 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-78cf4c6f6-gftqx" Jul 18 11:49:41.686810 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:49:41.686787 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\"" Jul 18 11:49:41.687800 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:49:41.687776 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\"" Jul 18 11:49:41.687800 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:49:41.687797 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-7g9fd\"" Jul 18 11:49:41.696980 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:49:41.696958 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-78cf4c6f6-gftqx"] Jul 18 11:49:41.697069 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:49:41.696985 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-555d9d7885-44h2s"] Jul 18 11:49:41.697069 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:49:41.697049 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-555d9d7885-44h2s" Jul 18 11:49:41.699207 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:49:41.699186 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-jnxrj\"" Jul 18 11:49:41.699503 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:49:41.699479 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\"" Jul 18 11:49:41.699503 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:49:41.699495 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\"" Jul 18 11:49:41.699630 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:49:41.699578 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Jul 18 11:49:41.700186 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:49:41.700167 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Jul 18 11:49:41.704811 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:49:41.704788 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\"" Jul 18 11:49:41.783838 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:49:41.783802 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-29ngm"] Jul 18 11:49:41.792786 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:49:41.792737 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-565b7b569d-j9l7p"] Jul 18 11:49:41.792894 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:49:41.792881 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-29ngm" Jul 18 11:49:41.796039 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:49:41.796022 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\"" Jul 18 11:49:41.796528 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:49:41.796508 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\"" Jul 18 11:49:41.796684 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:49:41.796669 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\"" Jul 18 11:49:41.796790 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:49:41.796772 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-j8j62\"" Jul 18 11:49:41.797096 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:49:41.797079 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\"" Jul 18 11:49:41.800997 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:49:41.800965 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/43f38029-3002-493c-a718-2fe047c3cd99-service-ca-bundle\") pod \"insights-operator-555d9d7885-44h2s\" (UID: \"43f38029-3002-493c-a718-2fe047c3cd99\") " pod="openshift-insights/insights-operator-555d9d7885-44h2s" Jul 18 11:49:41.801106 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:49:41.801003 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z6sbj\" (UniqueName: \"kubernetes.io/projected/43f38029-3002-493c-a718-2fe047c3cd99-kube-api-access-z6sbj\") pod \"insights-operator-555d9d7885-44h2s\" (UID: \"43f38029-3002-493c-a718-2fe047c3cd99\") " pod="openshift-insights/insights-operator-555d9d7885-44h2s" Jul 18 11:49:41.801106 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:49:41.801048 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fzw9x\" (UniqueName: \"kubernetes.io/projected/79fb452b-b19a-48d8-930d-679ea447ee0a-kube-api-access-fzw9x\") pod \"volume-data-source-validator-78cf4c6f6-gftqx\" (UID: \"79fb452b-b19a-48d8-930d-679ea447ee0a\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-78cf4c6f6-gftqx" Jul 18 11:49:41.801211 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:49:41.801123 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/43f38029-3002-493c-a718-2fe047c3cd99-serving-cert\") pod \"insights-operator-555d9d7885-44h2s\" (UID: \"43f38029-3002-493c-a718-2fe047c3cd99\") " pod="openshift-insights/insights-operator-555d9d7885-44h2s" Jul 18 11:49:41.801211 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:49:41.801168 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/43f38029-3002-493c-a718-2fe047c3cd99-tmp\") pod \"insights-operator-555d9d7885-44h2s\" (UID: \"43f38029-3002-493c-a718-2fe047c3cd99\") " pod="openshift-insights/insights-operator-555d9d7885-44h2s" Jul 18 11:49:41.801211 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:49:41.801192 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/43f38029-3002-493c-a718-2fe047c3cd99-snapshots\") pod \"insights-operator-555d9d7885-44h2s\" (UID: \"43f38029-3002-493c-a718-2fe047c3cd99\") " pod="openshift-insights/insights-operator-555d9d7885-44h2s" Jul 18 11:49:41.801517 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:49:41.801235 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/43f38029-3002-493c-a718-2fe047c3cd99-trusted-ca-bundle\") pod \"insights-operator-555d9d7885-44h2s\" (UID: \"43f38029-3002-493c-a718-2fe047c3cd99\") " pod="openshift-insights/insights-operator-555d9d7885-44h2s" Jul 18 11:49:41.807067 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:49:41.807048 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-555986d5b4-8r9nw"] Jul 18 11:49:41.807202 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:49:41.807187 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-565b7b569d-j9l7p" Jul 18 11:49:41.809469 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:49:41.809453 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\"" Jul 18 11:49:41.809560 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:49:41.809473 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-5gsql\"" Jul 18 11:49:41.809738 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:49:41.809719 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\"" Jul 18 11:49:41.810061 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:49:41.810047 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\"" Jul 18 11:49:41.810124 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:49:41.810112 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\"" Jul 18 11:49:41.818660 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:49:41.818636 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-29ngm"] Jul 18 11:49:41.818759 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:49:41.818705 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-555986d5b4-8r9nw"] Jul 18 11:49:41.818759 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:49:41.818738 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-565b7b569d-j9l7p"] Jul 18 11:49:41.818854 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:49:41.818825 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-555986d5b4-8r9nw" Jul 18 11:49:41.824797 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:49:41.824777 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\"" Jul 18 11:49:41.826262 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:49:41.826242 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Jul 18 11:49:41.826421 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:49:41.826243 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Jul 18 11:49:41.826493 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:49:41.826436 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Jul 18 11:49:41.826493 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:49:41.826444 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-g4x2w\"" Jul 18 11:49:41.831216 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:49:41.831200 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Jul 18 11:49:41.887345 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:49:41.887321 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-696b95fb6d-hx9nd"] Jul 18 11:49:41.895768 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:49:41.895751 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-696b95fb6d-hx9nd" Jul 18 11:49:41.897974 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:49:41.897954 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\"" Jul 18 11:49:41.898054 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:49:41.898000 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\"" Jul 18 11:49:41.898054 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:49:41.897957 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\"" Jul 18 11:49:41.898228 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:49:41.898214 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\"" Jul 18 11:49:41.898298 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:49:41.898279 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Jul 18 11:49:41.898417 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:49:41.898405 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Jul 18 11:49:41.898534 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:49:41.898519 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-nq2ft\"" Jul 18 11:49:41.899960 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:49:41.899942 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-696b95fb6d-hx9nd"] Jul 18 11:49:41.901643 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:49:41.901624 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/43f38029-3002-493c-a718-2fe047c3cd99-serving-cert\") pod \"insights-operator-555d9d7885-44h2s\" (UID: \"43f38029-3002-493c-a718-2fe047c3cd99\") " pod="openshift-insights/insights-operator-555d9d7885-44h2s" Jul 18 11:49:41.901746 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:49:41.901654 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/43f38029-3002-493c-a718-2fe047c3cd99-tmp\") pod \"insights-operator-555d9d7885-44h2s\" (UID: \"43f38029-3002-493c-a718-2fe047c3cd99\") " pod="openshift-insights/insights-operator-555d9d7885-44h2s" Jul 18 11:49:41.901746 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:49:41.901671 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/43f38029-3002-493c-a718-2fe047c3cd99-snapshots\") pod \"insights-operator-555d9d7885-44h2s\" (UID: \"43f38029-3002-493c-a718-2fe047c3cd99\") " pod="openshift-insights/insights-operator-555d9d7885-44h2s" Jul 18 11:49:41.901746 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:49:41.901693 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/8ec47eff-5fbc-4130-baef-d8544aa4e786-serving-cert\") pod \"console-operator-565b7b569d-j9l7p\" (UID: \"8ec47eff-5fbc-4130-baef-d8544aa4e786\") " pod="openshift-console-operator/console-operator-565b7b569d-j9l7p" Jul 18 11:49:41.901746 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:49:41.901722 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/45cc3bd5-b19f-4ac1-9257-6c39802b8e2f-config\") pod \"kube-storage-version-migrator-operator-7658d9b749-29ngm\" (UID: \"45cc3bd5-b19f-4ac1-9257-6c39802b8e2f\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-29ngm" Jul 18 11:49:41.901944 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:49:41.901766 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/43f38029-3002-493c-a718-2fe047c3cd99-trusted-ca-bundle\") pod \"insights-operator-555d9d7885-44h2s\" (UID: \"43f38029-3002-493c-a718-2fe047c3cd99\") " pod="openshift-insights/insights-operator-555d9d7885-44h2s" Jul 18 11:49:41.901944 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:49:41.901824 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/43f38029-3002-493c-a718-2fe047c3cd99-service-ca-bundle\") pod \"insights-operator-555d9d7885-44h2s\" (UID: \"43f38029-3002-493c-a718-2fe047c3cd99\") " pod="openshift-insights/insights-operator-555d9d7885-44h2s" Jul 18 11:49:41.901944 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:49:41.901847 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z6sbj\" (UniqueName: \"kubernetes.io/projected/43f38029-3002-493c-a718-2fe047c3cd99-kube-api-access-z6sbj\") pod \"insights-operator-555d9d7885-44h2s\" (UID: \"43f38029-3002-493c-a718-2fe047c3cd99\") " pod="openshift-insights/insights-operator-555d9d7885-44h2s" Jul 18 11:49:41.901944 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:49:41.901870 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fzw9x\" (UniqueName: \"kubernetes.io/projected/79fb452b-b19a-48d8-930d-679ea447ee0a-kube-api-access-fzw9x\") pod \"volume-data-source-validator-78cf4c6f6-gftqx\" (UID: \"79fb452b-b19a-48d8-930d-679ea447ee0a\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-78cf4c6f6-gftqx" Jul 18 11:49:41.901944 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:49:41.901895 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/8ec47eff-5fbc-4130-baef-d8544aa4e786-trusted-ca\") pod \"console-operator-565b7b569d-j9l7p\" (UID: \"8ec47eff-5fbc-4130-baef-d8544aa4e786\") " pod="openshift-console-operator/console-operator-565b7b569d-j9l7p" Jul 18 11:49:41.901944 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:49:41.901937 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/45cc3bd5-b19f-4ac1-9257-6c39802b8e2f-serving-cert\") pod \"kube-storage-version-migrator-operator-7658d9b749-29ngm\" (UID: \"45cc3bd5-b19f-4ac1-9257-6c39802b8e2f\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-29ngm" Jul 18 11:49:41.902323 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:49:41.901977 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zv7n4\" (UniqueName: \"kubernetes.io/projected/8ec47eff-5fbc-4130-baef-d8544aa4e786-kube-api-access-zv7n4\") pod \"console-operator-565b7b569d-j9l7p\" (UID: \"8ec47eff-5fbc-4130-baef-d8544aa4e786\") " pod="openshift-console-operator/console-operator-565b7b569d-j9l7p" Jul 18 11:49:41.902323 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:49:41.902020 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9sm79\" (UniqueName: \"kubernetes.io/projected/45cc3bd5-b19f-4ac1-9257-6c39802b8e2f-kube-api-access-9sm79\") pod \"kube-storage-version-migrator-operator-7658d9b749-29ngm\" (UID: \"45cc3bd5-b19f-4ac1-9257-6c39802b8e2f\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-29ngm" Jul 18 11:49:41.902323 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:49:41.902046 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/8ec47eff-5fbc-4130-baef-d8544aa4e786-config\") pod \"console-operator-565b7b569d-j9l7p\" (UID: \"8ec47eff-5fbc-4130-baef-d8544aa4e786\") " pod="openshift-console-operator/console-operator-565b7b569d-j9l7p" Jul 18 11:49:41.902323 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:49:41.902070 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/43f38029-3002-493c-a718-2fe047c3cd99-tmp\") pod \"insights-operator-555d9d7885-44h2s\" (UID: \"43f38029-3002-493c-a718-2fe047c3cd99\") " pod="openshift-insights/insights-operator-555d9d7885-44h2s" Jul 18 11:49:41.902637 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:49:41.902607 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/43f38029-3002-493c-a718-2fe047c3cd99-snapshots\") pod \"insights-operator-555d9d7885-44h2s\" (UID: \"43f38029-3002-493c-a718-2fe047c3cd99\") " pod="openshift-insights/insights-operator-555d9d7885-44h2s" Jul 18 11:49:41.903574 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:49:41.903100 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/43f38029-3002-493c-a718-2fe047c3cd99-service-ca-bundle\") pod \"insights-operator-555d9d7885-44h2s\" (UID: \"43f38029-3002-493c-a718-2fe047c3cd99\") " pod="openshift-insights/insights-operator-555d9d7885-44h2s" Jul 18 11:49:41.903671 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:49:41.903645 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/43f38029-3002-493c-a718-2fe047c3cd99-trusted-ca-bundle\") pod \"insights-operator-555d9d7885-44h2s\" (UID: \"43f38029-3002-493c-a718-2fe047c3cd99\") " pod="openshift-insights/insights-operator-555d9d7885-44h2s" Jul 18 11:49:41.906476 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:49:41.906454 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/43f38029-3002-493c-a718-2fe047c3cd99-serving-cert\") pod \"insights-operator-555d9d7885-44h2s\" (UID: \"43f38029-3002-493c-a718-2fe047c3cd99\") " pod="openshift-insights/insights-operator-555d9d7885-44h2s" Jul 18 11:49:41.910722 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:49:41.910699 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fzw9x\" (UniqueName: \"kubernetes.io/projected/79fb452b-b19a-48d8-930d-679ea447ee0a-kube-api-access-fzw9x\") pod \"volume-data-source-validator-78cf4c6f6-gftqx\" (UID: \"79fb452b-b19a-48d8-930d-679ea447ee0a\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-78cf4c6f6-gftqx" Jul 18 11:49:41.910806 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:49:41.910795 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z6sbj\" (UniqueName: \"kubernetes.io/projected/43f38029-3002-493c-a718-2fe047c3cd99-kube-api-access-z6sbj\") pod \"insights-operator-555d9d7885-44h2s\" (UID: \"43f38029-3002-493c-a718-2fe047c3cd99\") " pod="openshift-insights/insights-operator-555d9d7885-44h2s" Jul 18 11:49:41.994543 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:49:41.994517 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-78cf4c6f6-gftqx" Jul 18 11:49:42.002808 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:49:42.002784 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9sm79\" (UniqueName: \"kubernetes.io/projected/45cc3bd5-b19f-4ac1-9257-6c39802b8e2f-kube-api-access-9sm79\") pod \"kube-storage-version-migrator-operator-7658d9b749-29ngm\" (UID: \"45cc3bd5-b19f-4ac1-9257-6c39802b8e2f\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-29ngm" Jul 18 11:49:42.002880 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:49:42.002823 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/9477954a-e6a6-4bba-ad8f-3a8788b2b2ec-default-certificate\") pod \"router-default-696b95fb6d-hx9nd\" (UID: \"9477954a-e6a6-4bba-ad8f-3a8788b2b2ec\") " pod="openshift-ingress/router-default-696b95fb6d-hx9nd" Jul 18 11:49:42.002880 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:49:42.002851 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/2a5a3d5e-ed10-4c8b-83b6-dc23ad399e00-registry-tls\") pod \"image-registry-555986d5b4-8r9nw\" (UID: \"2a5a3d5e-ed10-4c8b-83b6-dc23ad399e00\") " pod="openshift-image-registry/image-registry-555986d5b4-8r9nw" Jul 18 11:49:42.002995 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:49:42.002880 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/2a5a3d5e-ed10-4c8b-83b6-dc23ad399e00-trusted-ca\") pod \"image-registry-555986d5b4-8r9nw\" (UID: \"2a5a3d5e-ed10-4c8b-83b6-dc23ad399e00\") " pod="openshift-image-registry/image-registry-555986d5b4-8r9nw" Jul 18 11:49:42.002995 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:49:42.002923 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/9477954a-e6a6-4bba-ad8f-3a8788b2b2ec-stats-auth\") pod \"router-default-696b95fb6d-hx9nd\" (UID: \"9477954a-e6a6-4bba-ad8f-3a8788b2b2ec\") " pod="openshift-ingress/router-default-696b95fb6d-hx9nd" Jul 18 11:49:42.003058 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:49:42.002995 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/2a5a3d5e-ed10-4c8b-83b6-dc23ad399e00-ca-trust-extracted\") pod \"image-registry-555986d5b4-8r9nw\" (UID: \"2a5a3d5e-ed10-4c8b-83b6-dc23ad399e00\") " pod="openshift-image-registry/image-registry-555986d5b4-8r9nw" Jul 18 11:49:42.003058 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:49:42.003025 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/2a5a3d5e-ed10-4c8b-83b6-dc23ad399e00-registry-certificates\") pod \"image-registry-555986d5b4-8r9nw\" (UID: \"2a5a3d5e-ed10-4c8b-83b6-dc23ad399e00\") " pod="openshift-image-registry/image-registry-555986d5b4-8r9nw" Jul 18 11:49:42.003058 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:49:42.003042 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/2a5a3d5e-ed10-4c8b-83b6-dc23ad399e00-installation-pull-secrets\") pod \"image-registry-555986d5b4-8r9nw\" (UID: \"2a5a3d5e-ed10-4c8b-83b6-dc23ad399e00\") " pod="openshift-image-registry/image-registry-555986d5b4-8r9nw" Jul 18 11:49:42.003171 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:49:42.003070 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zv7n4\" (UniqueName: \"kubernetes.io/projected/8ec47eff-5fbc-4130-baef-d8544aa4e786-kube-api-access-zv7n4\") pod \"console-operator-565b7b569d-j9l7p\" (UID: \"8ec47eff-5fbc-4130-baef-d8544aa4e786\") " pod="openshift-console-operator/console-operator-565b7b569d-j9l7p" Jul 18 11:49:42.003171 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:49:42.003107 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/8ec47eff-5fbc-4130-baef-d8544aa4e786-config\") pod \"console-operator-565b7b569d-j9l7p\" (UID: \"8ec47eff-5fbc-4130-baef-d8544aa4e786\") " pod="openshift-console-operator/console-operator-565b7b569d-j9l7p" Jul 18 11:49:42.003171 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:49:42.003140 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/8ec47eff-5fbc-4130-baef-d8544aa4e786-serving-cert\") pod \"console-operator-565b7b569d-j9l7p\" (UID: \"8ec47eff-5fbc-4130-baef-d8544aa4e786\") " pod="openshift-console-operator/console-operator-565b7b569d-j9l7p" Jul 18 11:49:42.003171 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:49:42.003162 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/2a5a3d5e-ed10-4c8b-83b6-dc23ad399e00-image-registry-private-configuration\") pod \"image-registry-555986d5b4-8r9nw\" (UID: \"2a5a3d5e-ed10-4c8b-83b6-dc23ad399e00\") " pod="openshift-image-registry/image-registry-555986d5b4-8r9nw" Jul 18 11:49:42.003367 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:49:42.003182 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4g27h\" (UniqueName: \"kubernetes.io/projected/2a5a3d5e-ed10-4c8b-83b6-dc23ad399e00-kube-api-access-4g27h\") pod \"image-registry-555986d5b4-8r9nw\" (UID: \"2a5a3d5e-ed10-4c8b-83b6-dc23ad399e00\") " pod="openshift-image-registry/image-registry-555986d5b4-8r9nw" Jul 18 11:49:42.003367 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:49:42.003209 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/2a5a3d5e-ed10-4c8b-83b6-dc23ad399e00-bound-sa-token\") pod \"image-registry-555986d5b4-8r9nw\" (UID: \"2a5a3d5e-ed10-4c8b-83b6-dc23ad399e00\") " pod="openshift-image-registry/image-registry-555986d5b4-8r9nw" Jul 18 11:49:42.003367 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:49:42.003242 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/45cc3bd5-b19f-4ac1-9257-6c39802b8e2f-config\") pod \"kube-storage-version-migrator-operator-7658d9b749-29ngm\" (UID: \"45cc3bd5-b19f-4ac1-9257-6c39802b8e2f\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-29ngm" Jul 18 11:49:42.003367 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:49:42.003262 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tczrk\" (UniqueName: \"kubernetes.io/projected/9477954a-e6a6-4bba-ad8f-3a8788b2b2ec-kube-api-access-tczrk\") pod \"router-default-696b95fb6d-hx9nd\" (UID: \"9477954a-e6a6-4bba-ad8f-3a8788b2b2ec\") " pod="openshift-ingress/router-default-696b95fb6d-hx9nd" Jul 18 11:49:42.003367 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:49:42.003290 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9477954a-e6a6-4bba-ad8f-3a8788b2b2ec-service-ca-bundle\") pod \"router-default-696b95fb6d-hx9nd\" (UID: \"9477954a-e6a6-4bba-ad8f-3a8788b2b2ec\") " pod="openshift-ingress/router-default-696b95fb6d-hx9nd" Jul 18 11:49:42.003367 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:49:42.003313 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9477954a-e6a6-4bba-ad8f-3a8788b2b2ec-metrics-certs\") pod \"router-default-696b95fb6d-hx9nd\" (UID: \"9477954a-e6a6-4bba-ad8f-3a8788b2b2ec\") " pod="openshift-ingress/router-default-696b95fb6d-hx9nd" Jul 18 11:49:42.003367 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:49:42.003337 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/8ec47eff-5fbc-4130-baef-d8544aa4e786-trusted-ca\") pod \"console-operator-565b7b569d-j9l7p\" (UID: \"8ec47eff-5fbc-4130-baef-d8544aa4e786\") " pod="openshift-console-operator/console-operator-565b7b569d-j9l7p" Jul 18 11:49:42.003699 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:49:42.003370 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/45cc3bd5-b19f-4ac1-9257-6c39802b8e2f-serving-cert\") pod \"kube-storage-version-migrator-operator-7658d9b749-29ngm\" (UID: \"45cc3bd5-b19f-4ac1-9257-6c39802b8e2f\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-29ngm" Jul 18 11:49:42.003699 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:49:42.003622 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/8ec47eff-5fbc-4130-baef-d8544aa4e786-config\") pod \"console-operator-565b7b569d-j9l7p\" (UID: \"8ec47eff-5fbc-4130-baef-d8544aa4e786\") " pod="openshift-console-operator/console-operator-565b7b569d-j9l7p" Jul 18 11:49:42.003790 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:49:42.003771 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/45cc3bd5-b19f-4ac1-9257-6c39802b8e2f-config\") pod \"kube-storage-version-migrator-operator-7658d9b749-29ngm\" (UID: \"45cc3bd5-b19f-4ac1-9257-6c39802b8e2f\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-29ngm" Jul 18 11:49:42.004215 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:49:42.004147 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/8ec47eff-5fbc-4130-baef-d8544aa4e786-trusted-ca\") pod \"console-operator-565b7b569d-j9l7p\" (UID: \"8ec47eff-5fbc-4130-baef-d8544aa4e786\") " pod="openshift-console-operator/console-operator-565b7b569d-j9l7p" Jul 18 11:49:42.005666 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:49:42.005646 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/8ec47eff-5fbc-4130-baef-d8544aa4e786-serving-cert\") pod \"console-operator-565b7b569d-j9l7p\" (UID: \"8ec47eff-5fbc-4130-baef-d8544aa4e786\") " pod="openshift-console-operator/console-operator-565b7b569d-j9l7p" Jul 18 11:49:42.005746 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:49:42.005707 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/45cc3bd5-b19f-4ac1-9257-6c39802b8e2f-serving-cert\") pod \"kube-storage-version-migrator-operator-7658d9b749-29ngm\" (UID: \"45cc3bd5-b19f-4ac1-9257-6c39802b8e2f\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-29ngm" Jul 18 11:49:42.006613 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:49:42.006598 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-555d9d7885-44h2s" Jul 18 11:49:42.010907 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:49:42.010883 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zv7n4\" (UniqueName: \"kubernetes.io/projected/8ec47eff-5fbc-4130-baef-d8544aa4e786-kube-api-access-zv7n4\") pod \"console-operator-565b7b569d-j9l7p\" (UID: \"8ec47eff-5fbc-4130-baef-d8544aa4e786\") " pod="openshift-console-operator/console-operator-565b7b569d-j9l7p" Jul 18 11:49:42.011095 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:49:42.011079 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9sm79\" (UniqueName: \"kubernetes.io/projected/45cc3bd5-b19f-4ac1-9257-6c39802b8e2f-kube-api-access-9sm79\") pod \"kube-storage-version-migrator-operator-7658d9b749-29ngm\" (UID: \"45cc3bd5-b19f-4ac1-9257-6c39802b8e2f\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-29ngm" Jul 18 11:49:42.110444 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:49:42.105685 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-29ngm" Jul 18 11:49:42.110444 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:49:42.106161 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/2a5a3d5e-ed10-4c8b-83b6-dc23ad399e00-registry-tls\") pod \"image-registry-555986d5b4-8r9nw\" (UID: \"2a5a3d5e-ed10-4c8b-83b6-dc23ad399e00\") " pod="openshift-image-registry/image-registry-555986d5b4-8r9nw" Jul 18 11:49:42.110444 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:49:42.106200 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/2a5a3d5e-ed10-4c8b-83b6-dc23ad399e00-trusted-ca\") pod \"image-registry-555986d5b4-8r9nw\" (UID: \"2a5a3d5e-ed10-4c8b-83b6-dc23ad399e00\") " pod="openshift-image-registry/image-registry-555986d5b4-8r9nw" Jul 18 11:49:42.110444 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:49:42.106231 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/9477954a-e6a6-4bba-ad8f-3a8788b2b2ec-stats-auth\") pod \"router-default-696b95fb6d-hx9nd\" (UID: \"9477954a-e6a6-4bba-ad8f-3a8788b2b2ec\") " pod="openshift-ingress/router-default-696b95fb6d-hx9nd" Jul 18 11:49:42.110444 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:49:42.106275 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/2a5a3d5e-ed10-4c8b-83b6-dc23ad399e00-ca-trust-extracted\") pod \"image-registry-555986d5b4-8r9nw\" (UID: \"2a5a3d5e-ed10-4c8b-83b6-dc23ad399e00\") " pod="openshift-image-registry/image-registry-555986d5b4-8r9nw" Jul 18 11:49:42.110444 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:49:42.106302 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/2a5a3d5e-ed10-4c8b-83b6-dc23ad399e00-registry-certificates\") pod \"image-registry-555986d5b4-8r9nw\" (UID: \"2a5a3d5e-ed10-4c8b-83b6-dc23ad399e00\") " pod="openshift-image-registry/image-registry-555986d5b4-8r9nw" Jul 18 11:49:42.110444 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:49:42.106325 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/2a5a3d5e-ed10-4c8b-83b6-dc23ad399e00-installation-pull-secrets\") pod \"image-registry-555986d5b4-8r9nw\" (UID: \"2a5a3d5e-ed10-4c8b-83b6-dc23ad399e00\") " pod="openshift-image-registry/image-registry-555986d5b4-8r9nw" Jul 18 11:49:42.110444 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:49:42.106376 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/2a5a3d5e-ed10-4c8b-83b6-dc23ad399e00-image-registry-private-configuration\") pod \"image-registry-555986d5b4-8r9nw\" (UID: \"2a5a3d5e-ed10-4c8b-83b6-dc23ad399e00\") " pod="openshift-image-registry/image-registry-555986d5b4-8r9nw" Jul 18 11:49:42.110444 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:49:42.106404 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4g27h\" (UniqueName: \"kubernetes.io/projected/2a5a3d5e-ed10-4c8b-83b6-dc23ad399e00-kube-api-access-4g27h\") pod \"image-registry-555986d5b4-8r9nw\" (UID: \"2a5a3d5e-ed10-4c8b-83b6-dc23ad399e00\") " pod="openshift-image-registry/image-registry-555986d5b4-8r9nw" Jul 18 11:49:42.110444 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:49:42.106431 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/2a5a3d5e-ed10-4c8b-83b6-dc23ad399e00-bound-sa-token\") pod \"image-registry-555986d5b4-8r9nw\" (UID: \"2a5a3d5e-ed10-4c8b-83b6-dc23ad399e00\") " pod="openshift-image-registry/image-registry-555986d5b4-8r9nw" Jul 18 11:49:42.110444 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:49:42.106458 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tczrk\" (UniqueName: \"kubernetes.io/projected/9477954a-e6a6-4bba-ad8f-3a8788b2b2ec-kube-api-access-tczrk\") pod \"router-default-696b95fb6d-hx9nd\" (UID: \"9477954a-e6a6-4bba-ad8f-3a8788b2b2ec\") " pod="openshift-ingress/router-default-696b95fb6d-hx9nd" Jul 18 11:49:42.110444 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:49:42.106492 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9477954a-e6a6-4bba-ad8f-3a8788b2b2ec-service-ca-bundle\") pod \"router-default-696b95fb6d-hx9nd\" (UID: \"9477954a-e6a6-4bba-ad8f-3a8788b2b2ec\") " pod="openshift-ingress/router-default-696b95fb6d-hx9nd" Jul 18 11:49:42.110444 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:49:42.106514 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9477954a-e6a6-4bba-ad8f-3a8788b2b2ec-metrics-certs\") pod \"router-default-696b95fb6d-hx9nd\" (UID: \"9477954a-e6a6-4bba-ad8f-3a8788b2b2ec\") " pod="openshift-ingress/router-default-696b95fb6d-hx9nd" Jul 18 11:49:42.110444 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:49:42.106593 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/9477954a-e6a6-4bba-ad8f-3a8788b2b2ec-default-certificate\") pod \"router-default-696b95fb6d-hx9nd\" (UID: \"9477954a-e6a6-4bba-ad8f-3a8788b2b2ec\") " pod="openshift-ingress/router-default-696b95fb6d-hx9nd" Jul 18 11:49:42.110444 ip-10-0-136-240 kubenswrapper[2585]: E0718 11:49:42.108316 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/9477954a-e6a6-4bba-ad8f-3a8788b2b2ec-service-ca-bundle podName:9477954a-e6a6-4bba-ad8f-3a8788b2b2ec nodeName:}" failed. No retries permitted until 2026-07-18 11:49:42.60829536 +0000 UTC m=+49.688501697 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/9477954a-e6a6-4bba-ad8f-3a8788b2b2ec-service-ca-bundle") pod "router-default-696b95fb6d-hx9nd" (UID: "9477954a-e6a6-4bba-ad8f-3a8788b2b2ec") : configmap references non-existent config key: service-ca.crt Jul 18 11:49:42.110444 ip-10-0-136-240 kubenswrapper[2585]: E0718 11:49:42.108776 2585 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 18 11:49:42.110444 ip-10-0-136-240 kubenswrapper[2585]: E0718 11:49:42.108792 2585 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-555986d5b4-8r9nw: secret "image-registry-tls" not found Jul 18 11:49:42.111232 ip-10-0-136-240 kubenswrapper[2585]: E0718 11:49:42.108829 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2a5a3d5e-ed10-4c8b-83b6-dc23ad399e00-registry-tls podName:2a5a3d5e-ed10-4c8b-83b6-dc23ad399e00 nodeName:}" failed. No retries permitted until 2026-07-18 11:49:42.60881602 +0000 UTC m=+49.689022360 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/2a5a3d5e-ed10-4c8b-83b6-dc23ad399e00-registry-tls") pod "image-registry-555986d5b4-8r9nw" (UID: "2a5a3d5e-ed10-4c8b-83b6-dc23ad399e00") : secret "image-registry-tls" not found Jul 18 11:49:42.111232 ip-10-0-136-240 kubenswrapper[2585]: E0718 11:49:42.108887 2585 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 18 11:49:42.111232 ip-10-0-136-240 kubenswrapper[2585]: E0718 11:49:42.108917 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9477954a-e6a6-4bba-ad8f-3a8788b2b2ec-metrics-certs podName:9477954a-e6a6-4bba-ad8f-3a8788b2b2ec nodeName:}" failed. No retries permitted until 2026-07-18 11:49:42.608907143 +0000 UTC m=+49.689113491 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9477954a-e6a6-4bba-ad8f-3a8788b2b2ec-metrics-certs") pod "router-default-696b95fb6d-hx9nd" (UID: "9477954a-e6a6-4bba-ad8f-3a8788b2b2ec") : secret "router-metrics-certs-default" not found Jul 18 11:49:42.111232 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:49:42.109191 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/2a5a3d5e-ed10-4c8b-83b6-dc23ad399e00-ca-trust-extracted\") pod \"image-registry-555986d5b4-8r9nw\" (UID: \"2a5a3d5e-ed10-4c8b-83b6-dc23ad399e00\") " pod="openshift-image-registry/image-registry-555986d5b4-8r9nw" Jul 18 11:49:42.112280 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:49:42.112253 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/9477954a-e6a6-4bba-ad8f-3a8788b2b2ec-default-certificate\") pod \"router-default-696b95fb6d-hx9nd\" (UID: \"9477954a-e6a6-4bba-ad8f-3a8788b2b2ec\") " pod="openshift-ingress/router-default-696b95fb6d-hx9nd" Jul 18 11:49:42.113297 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:49:42.112839 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/9477954a-e6a6-4bba-ad8f-3a8788b2b2ec-stats-auth\") pod \"router-default-696b95fb6d-hx9nd\" (UID: \"9477954a-e6a6-4bba-ad8f-3a8788b2b2ec\") " pod="openshift-ingress/router-default-696b95fb6d-hx9nd" Jul 18 11:49:42.113297 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:49:42.113216 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/2a5a3d5e-ed10-4c8b-83b6-dc23ad399e00-trusted-ca\") pod \"image-registry-555986d5b4-8r9nw\" (UID: \"2a5a3d5e-ed10-4c8b-83b6-dc23ad399e00\") " pod="openshift-image-registry/image-registry-555986d5b4-8r9nw" Jul 18 11:49:42.113689 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:49:42.113657 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/2a5a3d5e-ed10-4c8b-83b6-dc23ad399e00-registry-certificates\") pod \"image-registry-555986d5b4-8r9nw\" (UID: \"2a5a3d5e-ed10-4c8b-83b6-dc23ad399e00\") " pod="openshift-image-registry/image-registry-555986d5b4-8r9nw" Jul 18 11:49:42.113782 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:49:42.113753 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/2a5a3d5e-ed10-4c8b-83b6-dc23ad399e00-installation-pull-secrets\") pod \"image-registry-555986d5b4-8r9nw\" (UID: \"2a5a3d5e-ed10-4c8b-83b6-dc23ad399e00\") " pod="openshift-image-registry/image-registry-555986d5b4-8r9nw" Jul 18 11:49:42.114208 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:49:42.114118 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/2a5a3d5e-ed10-4c8b-83b6-dc23ad399e00-image-registry-private-configuration\") pod \"image-registry-555986d5b4-8r9nw\" (UID: \"2a5a3d5e-ed10-4c8b-83b6-dc23ad399e00\") " pod="openshift-image-registry/image-registry-555986d5b4-8r9nw" Jul 18 11:49:42.119316 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:49:42.119128 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-565b7b569d-j9l7p" Jul 18 11:49:42.129225 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:49:42.129193 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tczrk\" (UniqueName: \"kubernetes.io/projected/9477954a-e6a6-4bba-ad8f-3a8788b2b2ec-kube-api-access-tczrk\") pod \"router-default-696b95fb6d-hx9nd\" (UID: \"9477954a-e6a6-4bba-ad8f-3a8788b2b2ec\") " pod="openshift-ingress/router-default-696b95fb6d-hx9nd" Jul 18 11:49:42.129367 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:49:42.129341 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4g27h\" (UniqueName: \"kubernetes.io/projected/2a5a3d5e-ed10-4c8b-83b6-dc23ad399e00-kube-api-access-4g27h\") pod \"image-registry-555986d5b4-8r9nw\" (UID: \"2a5a3d5e-ed10-4c8b-83b6-dc23ad399e00\") " pod="openshift-image-registry/image-registry-555986d5b4-8r9nw" Jul 18 11:49:42.130356 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:49:42.130331 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/2a5a3d5e-ed10-4c8b-83b6-dc23ad399e00-bound-sa-token\") pod \"image-registry-555986d5b4-8r9nw\" (UID: \"2a5a3d5e-ed10-4c8b-83b6-dc23ad399e00\") " pod="openshift-image-registry/image-registry-555986d5b4-8r9nw" Jul 18 11:49:42.160793 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:49:42.160760 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-78cf4c6f6-gftqx"] Jul 18 11:49:42.164200 ip-10-0-136-240 kubenswrapper[2585]: W0718 11:49:42.164169 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod79fb452b_b19a_48d8_930d_679ea447ee0a.slice/crio-2014897e3b1ee481661ef6172dfb0973d7ba440722de2da2e070e6cdbc83e2d6 WatchSource:0}: Error finding container 2014897e3b1ee481661ef6172dfb0973d7ba440722de2da2e070e6cdbc83e2d6: Status 404 returned error can't find the container with id 2014897e3b1ee481661ef6172dfb0973d7ba440722de2da2e070e6cdbc83e2d6 Jul 18 11:49:42.180043 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:49:42.179315 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-555d9d7885-44h2s"] Jul 18 11:49:42.182288 ip-10-0-136-240 kubenswrapper[2585]: W0718 11:49:42.182073 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod43f38029_3002_493c_a718_2fe047c3cd99.slice/crio-66ee74dff85cb68195cd72ee9a6fcf38488b4fb4b63ad5cefef583f2801e685d WatchSource:0}: Error finding container 66ee74dff85cb68195cd72ee9a6fcf38488b4fb4b63ad5cefef583f2801e685d: Status 404 returned error can't find the container with id 66ee74dff85cb68195cd72ee9a6fcf38488b4fb4b63ad5cefef583f2801e685d Jul 18 11:49:42.264259 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:49:42.264181 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-29ngm"] Jul 18 11:49:42.266920 ip-10-0-136-240 kubenswrapper[2585]: W0718 11:49:42.266893 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod45cc3bd5_b19f_4ac1_9257_6c39802b8e2f.slice/crio-d6b69ed95d3d40d0c4262416997ac8f1a8bfb03c1ab07749bfe33d8ee74353d8 WatchSource:0}: Error finding container d6b69ed95d3d40d0c4262416997ac8f1a8bfb03c1ab07749bfe33d8ee74353d8: Status 404 returned error can't find the container with id d6b69ed95d3d40d0c4262416997ac8f1a8bfb03c1ab07749bfe33d8ee74353d8 Jul 18 11:49:42.280455 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:49:42.280434 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-565b7b569d-j9l7p"] Jul 18 11:49:42.282953 ip-10-0-136-240 kubenswrapper[2585]: W0718 11:49:42.282932 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8ec47eff_5fbc_4130_baef_d8544aa4e786.slice/crio-151d1c720c339e69ec7ddfc29fa79c4eb4982573db715ec6b051c36c883d0fc0 WatchSource:0}: Error finding container 151d1c720c339e69ec7ddfc29fa79c4eb4982573db715ec6b051c36c883d0fc0: Status 404 returned error can't find the container with id 151d1c720c339e69ec7ddfc29fa79c4eb4982573db715ec6b051c36c883d0fc0 Jul 18 11:49:42.609784 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:49:42.609695 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9477954a-e6a6-4bba-ad8f-3a8788b2b2ec-service-ca-bundle\") pod \"router-default-696b95fb6d-hx9nd\" (UID: \"9477954a-e6a6-4bba-ad8f-3a8788b2b2ec\") " pod="openshift-ingress/router-default-696b95fb6d-hx9nd" Jul 18 11:49:42.609784 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:49:42.609730 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9477954a-e6a6-4bba-ad8f-3a8788b2b2ec-metrics-certs\") pod \"router-default-696b95fb6d-hx9nd\" (UID: \"9477954a-e6a6-4bba-ad8f-3a8788b2b2ec\") " pod="openshift-ingress/router-default-696b95fb6d-hx9nd" Jul 18 11:49:42.609784 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:49:42.609773 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/2a5a3d5e-ed10-4c8b-83b6-dc23ad399e00-registry-tls\") pod \"image-registry-555986d5b4-8r9nw\" (UID: \"2a5a3d5e-ed10-4c8b-83b6-dc23ad399e00\") " pod="openshift-image-registry/image-registry-555986d5b4-8r9nw" Jul 18 11:49:42.610035 ip-10-0-136-240 kubenswrapper[2585]: E0718 11:49:42.609858 2585 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 18 11:49:42.610035 ip-10-0-136-240 kubenswrapper[2585]: E0718 11:49:42.609891 2585 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 18 11:49:42.610035 ip-10-0-136-240 kubenswrapper[2585]: E0718 11:49:42.609896 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/9477954a-e6a6-4bba-ad8f-3a8788b2b2ec-service-ca-bundle podName:9477954a-e6a6-4bba-ad8f-3a8788b2b2ec nodeName:}" failed. No retries permitted until 2026-07-18 11:49:43.609875989 +0000 UTC m=+50.690082328 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/9477954a-e6a6-4bba-ad8f-3a8788b2b2ec-service-ca-bundle") pod "router-default-696b95fb6d-hx9nd" (UID: "9477954a-e6a6-4bba-ad8f-3a8788b2b2ec") : configmap references non-existent config key: service-ca.crt Jul 18 11:49:42.610035 ip-10-0-136-240 kubenswrapper[2585]: E0718 11:49:42.609902 2585 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-555986d5b4-8r9nw: secret "image-registry-tls" not found Jul 18 11:49:42.610035 ip-10-0-136-240 kubenswrapper[2585]: E0718 11:49:42.609922 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9477954a-e6a6-4bba-ad8f-3a8788b2b2ec-metrics-certs podName:9477954a-e6a6-4bba-ad8f-3a8788b2b2ec nodeName:}" failed. No retries permitted until 2026-07-18 11:49:43.609913043 +0000 UTC m=+50.690119378 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9477954a-e6a6-4bba-ad8f-3a8788b2b2ec-metrics-certs") pod "router-default-696b95fb6d-hx9nd" (UID: "9477954a-e6a6-4bba-ad8f-3a8788b2b2ec") : secret "router-metrics-certs-default" not found Jul 18 11:49:42.610035 ip-10-0-136-240 kubenswrapper[2585]: E0718 11:49:42.609942 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2a5a3d5e-ed10-4c8b-83b6-dc23ad399e00-registry-tls podName:2a5a3d5e-ed10-4c8b-83b6-dc23ad399e00 nodeName:}" failed. No retries permitted until 2026-07-18 11:49:43.609932165 +0000 UTC m=+50.690138503 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/2a5a3d5e-ed10-4c8b-83b6-dc23ad399e00-registry-tls") pod "image-registry-555986d5b4-8r9nw" (UID: "2a5a3d5e-ed10-4c8b-83b6-dc23ad399e00") : secret "image-registry-tls" not found Jul 18 11:49:42.777508 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:49:42.777469 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-78cf4c6f6-gftqx" event={"ID":"79fb452b-b19a-48d8-930d-679ea447ee0a","Type":"ContainerStarted","Data":"2014897e3b1ee481661ef6172dfb0973d7ba440722de2da2e070e6cdbc83e2d6"} Jul 18 11:49:42.778690 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:49:42.778660 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-555d9d7885-44h2s" event={"ID":"43f38029-3002-493c-a718-2fe047c3cd99","Type":"ContainerStarted","Data":"66ee74dff85cb68195cd72ee9a6fcf38488b4fb4b63ad5cefef583f2801e685d"} Jul 18 11:49:42.783024 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:49:42.782994 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-565b7b569d-j9l7p" event={"ID":"8ec47eff-5fbc-4130-baef-d8544aa4e786","Type":"ContainerStarted","Data":"151d1c720c339e69ec7ddfc29fa79c4eb4982573db715ec6b051c36c883d0fc0"} Jul 18 11:49:42.784058 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:49:42.784039 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-29ngm" event={"ID":"45cc3bd5-b19f-4ac1-9257-6c39802b8e2f","Type":"ContainerStarted","Data":"d6b69ed95d3d40d0c4262416997ac8f1a8bfb03c1ab07749bfe33d8ee74353d8"} Jul 18 11:49:43.619681 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:49:43.619639 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9477954a-e6a6-4bba-ad8f-3a8788b2b2ec-service-ca-bundle\") pod \"router-default-696b95fb6d-hx9nd\" (UID: \"9477954a-e6a6-4bba-ad8f-3a8788b2b2ec\") " pod="openshift-ingress/router-default-696b95fb6d-hx9nd" Jul 18 11:49:43.619681 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:49:43.619694 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9477954a-e6a6-4bba-ad8f-3a8788b2b2ec-metrics-certs\") pod \"router-default-696b95fb6d-hx9nd\" (UID: \"9477954a-e6a6-4bba-ad8f-3a8788b2b2ec\") " pod="openshift-ingress/router-default-696b95fb6d-hx9nd" Jul 18 11:49:43.619908 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:49:43.619760 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/2a5a3d5e-ed10-4c8b-83b6-dc23ad399e00-registry-tls\") pod \"image-registry-555986d5b4-8r9nw\" (UID: \"2a5a3d5e-ed10-4c8b-83b6-dc23ad399e00\") " pod="openshift-image-registry/image-registry-555986d5b4-8r9nw" Jul 18 11:49:43.619908 ip-10-0-136-240 kubenswrapper[2585]: E0718 11:49:43.619776 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/9477954a-e6a6-4bba-ad8f-3a8788b2b2ec-service-ca-bundle podName:9477954a-e6a6-4bba-ad8f-3a8788b2b2ec nodeName:}" failed. No retries permitted until 2026-07-18 11:49:45.619757193 +0000 UTC m=+52.699963551 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/9477954a-e6a6-4bba-ad8f-3a8788b2b2ec-service-ca-bundle") pod "router-default-696b95fb6d-hx9nd" (UID: "9477954a-e6a6-4bba-ad8f-3a8788b2b2ec") : configmap references non-existent config key: service-ca.crt Jul 18 11:49:43.619908 ip-10-0-136-240 kubenswrapper[2585]: E0718 11:49:43.619842 2585 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 18 11:49:43.619908 ip-10-0-136-240 kubenswrapper[2585]: E0718 11:49:43.619855 2585 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-555986d5b4-8r9nw: secret "image-registry-tls" not found Jul 18 11:49:43.619908 ip-10-0-136-240 kubenswrapper[2585]: E0718 11:49:43.619891 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2a5a3d5e-ed10-4c8b-83b6-dc23ad399e00-registry-tls podName:2a5a3d5e-ed10-4c8b-83b6-dc23ad399e00 nodeName:}" failed. No retries permitted until 2026-07-18 11:49:45.619879395 +0000 UTC m=+52.700085744 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/2a5a3d5e-ed10-4c8b-83b6-dc23ad399e00-registry-tls") pod "image-registry-555986d5b4-8r9nw" (UID: "2a5a3d5e-ed10-4c8b-83b6-dc23ad399e00") : secret "image-registry-tls" not found Jul 18 11:49:43.620168 ip-10-0-136-240 kubenswrapper[2585]: E0718 11:49:43.619983 2585 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 18 11:49:43.620168 ip-10-0-136-240 kubenswrapper[2585]: E0718 11:49:43.620043 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9477954a-e6a6-4bba-ad8f-3a8788b2b2ec-metrics-certs podName:9477954a-e6a6-4bba-ad8f-3a8788b2b2ec nodeName:}" failed. No retries permitted until 2026-07-18 11:49:45.620029679 +0000 UTC m=+52.700236017 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9477954a-e6a6-4bba-ad8f-3a8788b2b2ec-metrics-certs") pod "router-default-696b95fb6d-hx9nd" (UID: "9477954a-e6a6-4bba-ad8f-3a8788b2b2ec") : secret "router-metrics-certs-default" not found Jul 18 11:49:45.133994 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:49:45.133956 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/42cb6807-1262-4919-856b-80c7b6d08ae2-metrics-tls\") pod \"dns-default-mrh2r\" (UID: \"42cb6807-1262-4919-856b-80c7b6d08ae2\") " pod="openshift-dns/dns-default-mrh2r" Jul 18 11:49:45.134432 ip-10-0-136-240 kubenswrapper[2585]: E0718 11:49:45.134145 2585 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 18 11:49:45.134432 ip-10-0-136-240 kubenswrapper[2585]: E0718 11:49:45.134230 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/42cb6807-1262-4919-856b-80c7b6d08ae2-metrics-tls podName:42cb6807-1262-4919-856b-80c7b6d08ae2 nodeName:}" failed. No retries permitted until 2026-07-18 11:50:01.134207333 +0000 UTC m=+68.214413684 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/42cb6807-1262-4919-856b-80c7b6d08ae2-metrics-tls") pod "dns-default-mrh2r" (UID: "42cb6807-1262-4919-856b-80c7b6d08ae2") : secret "dns-default-metrics-tls" not found Jul 18 11:49:45.638038 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:49:45.637994 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9477954a-e6a6-4bba-ad8f-3a8788b2b2ec-service-ca-bundle\") pod \"router-default-696b95fb6d-hx9nd\" (UID: \"9477954a-e6a6-4bba-ad8f-3a8788b2b2ec\") " pod="openshift-ingress/router-default-696b95fb6d-hx9nd" Jul 18 11:49:45.638295 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:49:45.638050 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9477954a-e6a6-4bba-ad8f-3a8788b2b2ec-metrics-certs\") pod \"router-default-696b95fb6d-hx9nd\" (UID: \"9477954a-e6a6-4bba-ad8f-3a8788b2b2ec\") " pod="openshift-ingress/router-default-696b95fb6d-hx9nd" Jul 18 11:49:45.638295 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:49:45.638123 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/2a5a3d5e-ed10-4c8b-83b6-dc23ad399e00-registry-tls\") pod \"image-registry-555986d5b4-8r9nw\" (UID: \"2a5a3d5e-ed10-4c8b-83b6-dc23ad399e00\") " pod="openshift-image-registry/image-registry-555986d5b4-8r9nw" Jul 18 11:49:45.638295 ip-10-0-136-240 kubenswrapper[2585]: E0718 11:49:45.638146 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/9477954a-e6a6-4bba-ad8f-3a8788b2b2ec-service-ca-bundle podName:9477954a-e6a6-4bba-ad8f-3a8788b2b2ec nodeName:}" failed. No retries permitted until 2026-07-18 11:49:49.638121526 +0000 UTC m=+56.718327884 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/9477954a-e6a6-4bba-ad8f-3a8788b2b2ec-service-ca-bundle") pod "router-default-696b95fb6d-hx9nd" (UID: "9477954a-e6a6-4bba-ad8f-3a8788b2b2ec") : configmap references non-existent config key: service-ca.crt Jul 18 11:49:45.638295 ip-10-0-136-240 kubenswrapper[2585]: E0718 11:49:45.638155 2585 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 18 11:49:45.638295 ip-10-0-136-240 kubenswrapper[2585]: E0718 11:49:45.638191 2585 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 18 11:49:45.638295 ip-10-0-136-240 kubenswrapper[2585]: E0718 11:49:45.638203 2585 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-555986d5b4-8r9nw: secret "image-registry-tls" not found Jul 18 11:49:45.638295 ip-10-0-136-240 kubenswrapper[2585]: E0718 11:49:45.638231 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9477954a-e6a6-4bba-ad8f-3a8788b2b2ec-metrics-certs podName:9477954a-e6a6-4bba-ad8f-3a8788b2b2ec nodeName:}" failed. No retries permitted until 2026-07-18 11:49:49.638214703 +0000 UTC m=+56.718421065 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9477954a-e6a6-4bba-ad8f-3a8788b2b2ec-metrics-certs") pod "router-default-696b95fb6d-hx9nd" (UID: "9477954a-e6a6-4bba-ad8f-3a8788b2b2ec") : secret "router-metrics-certs-default" not found Jul 18 11:49:45.638295 ip-10-0-136-240 kubenswrapper[2585]: E0718 11:49:45.638253 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2a5a3d5e-ed10-4c8b-83b6-dc23ad399e00-registry-tls podName:2a5a3d5e-ed10-4c8b-83b6-dc23ad399e00 nodeName:}" failed. No retries permitted until 2026-07-18 11:49:49.638241815 +0000 UTC m=+56.718448158 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/2a5a3d5e-ed10-4c8b-83b6-dc23ad399e00-registry-tls") pod "image-registry-555986d5b4-8r9nw" (UID: "2a5a3d5e-ed10-4c8b-83b6-dc23ad399e00") : secret "image-registry-tls" not found Jul 18 11:49:46.796383 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:49:46.796356 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-j9l7p_8ec47eff-5fbc-4130-baef-d8544aa4e786/console-operator/0.log" Jul 18 11:49:46.796836 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:49:46.796399 2585 generic.go:356] "Generic (PLEG): container finished" podID="8ec47eff-5fbc-4130-baef-d8544aa4e786" containerID="42544565cfafac737f67184300df1c02d64108a73c2fc14ee6348c5d41f25956" exitCode=255 Jul 18 11:49:46.796836 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:49:46.796483 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-565b7b569d-j9l7p" event={"ID":"8ec47eff-5fbc-4130-baef-d8544aa4e786","Type":"ContainerDied","Data":"42544565cfafac737f67184300df1c02d64108a73c2fc14ee6348c5d41f25956"} Jul 18 11:49:46.796836 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:49:46.796794 2585 scope.go:117] "RemoveContainer" containerID="42544565cfafac737f67184300df1c02d64108a73c2fc14ee6348c5d41f25956" Jul 18 11:49:46.798035 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:49:46.798013 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-29ngm" event={"ID":"45cc3bd5-b19f-4ac1-9257-6c39802b8e2f","Type":"ContainerStarted","Data":"0aad6b21f4690d70307e6bfe3ffe63e0614b5a15c4361c1c8f55261f44ef95ff"} Jul 18 11:49:46.800447 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:49:46.799729 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-78cf4c6f6-gftqx" event={"ID":"79fb452b-b19a-48d8-930d-679ea447ee0a","Type":"ContainerStarted","Data":"34af35abba084c83007dd901bfe35b5a0859625bbe9ba0e19b49cba70dde54a3"} Jul 18 11:49:46.805914 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:49:46.805892 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-555d9d7885-44h2s" event={"ID":"43f38029-3002-493c-a718-2fe047c3cd99","Type":"ContainerStarted","Data":"2d2217de63ea65770ab2586b286489496b4872c915f4a2155fbd913035bd376f"} Jul 18 11:49:46.807192 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:49:46.807172 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-hhwkn"] Jul 18 11:49:46.837426 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:49:46.837406 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-hhwkn"] Jul 18 11:49:46.837536 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:49:46.837520 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-hhwkn" Jul 18 11:49:46.841222 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:49:46.841205 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-29g54\"" Jul 18 11:49:46.841773 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:49:46.841751 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Jul 18 11:49:46.845527 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:49:46.845510 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Jul 18 11:49:46.848849 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:49:46.848828 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f44aedd4-16f8-4adb-a7b7-51719a187834-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-hhwkn\" (UID: \"f44aedd4-16f8-4adb-a7b7-51719a187834\") " pod="openshift-insights/insights-runtime-extractor-hhwkn" Jul 18 11:49:46.848933 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:49:46.848873 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/f44aedd4-16f8-4adb-a7b7-51719a187834-data-volume\") pod \"insights-runtime-extractor-hhwkn\" (UID: \"f44aedd4-16f8-4adb-a7b7-51719a187834\") " pod="openshift-insights/insights-runtime-extractor-hhwkn" Jul 18 11:49:46.849404 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:49:46.849178 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/f44aedd4-16f8-4adb-a7b7-51719a187834-crio-socket\") pod \"insights-runtime-extractor-hhwkn\" (UID: \"f44aedd4-16f8-4adb-a7b7-51719a187834\") " pod="openshift-insights/insights-runtime-extractor-hhwkn" Jul 18 11:49:46.849404 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:49:46.849277 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rwqnd\" (UniqueName: \"kubernetes.io/projected/f44aedd4-16f8-4adb-a7b7-51719a187834-kube-api-access-rwqnd\") pod \"insights-runtime-extractor-hhwkn\" (UID: \"f44aedd4-16f8-4adb-a7b7-51719a187834\") " pod="openshift-insights/insights-runtime-extractor-hhwkn" Jul 18 11:49:46.849404 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:49:46.849400 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/f44aedd4-16f8-4adb-a7b7-51719a187834-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-hhwkn\" (UID: \"f44aedd4-16f8-4adb-a7b7-51719a187834\") " pod="openshift-insights/insights-runtime-extractor-hhwkn" Jul 18 11:49:46.859397 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:49:46.859359 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-78cf4c6f6-gftqx" podStartSLOduration=1.7739023550000002 podStartE2EDuration="5.859345837s" totalImagesPullingTime="4.085443482s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-18 11:49:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-18 11:49:46.85790981 +0000 UTC m=+53.938116167" watchObservedRunningTime="2026-07-18 11:49:46.859345837 +0000 UTC m=+53.939552194" Jul 18 11:49:46.881432 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:49:46.881388 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-29ngm" podStartSLOduration=1.897231831 podStartE2EDuration="5.881370531s" totalImagesPullingTime="3.9841387s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-18 11:49:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-18 11:49:46.880714989 +0000 UTC m=+53.960921346" watchObservedRunningTime="2026-07-18 11:49:46.881370531 +0000 UTC m=+53.961576886" Jul 18 11:49:46.899773 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:49:46.899725 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-operator-555d9d7885-44h2s" podStartSLOduration=1.8254194689999999 podStartE2EDuration="5.899710278s" totalImagesPullingTime="4.074290809s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-18 11:49:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-18 11:49:46.898712092 +0000 UTC m=+53.978918450" watchObservedRunningTime="2026-07-18 11:49:46.899710278 +0000 UTC m=+53.979916682" Jul 18 11:49:46.949851 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:49:46.949826 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f44aedd4-16f8-4adb-a7b7-51719a187834-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-hhwkn\" (UID: \"f44aedd4-16f8-4adb-a7b7-51719a187834\") " pod="openshift-insights/insights-runtime-extractor-hhwkn" Jul 18 11:49:46.949974 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:49:46.949868 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/f44aedd4-16f8-4adb-a7b7-51719a187834-data-volume\") pod \"insights-runtime-extractor-hhwkn\" (UID: \"f44aedd4-16f8-4adb-a7b7-51719a187834\") " pod="openshift-insights/insights-runtime-extractor-hhwkn" Jul 18 11:49:46.949974 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:49:46.949901 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/f44aedd4-16f8-4adb-a7b7-51719a187834-crio-socket\") pod \"insights-runtime-extractor-hhwkn\" (UID: \"f44aedd4-16f8-4adb-a7b7-51719a187834\") " pod="openshift-insights/insights-runtime-extractor-hhwkn" Jul 18 11:49:46.949974 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:49:46.949956 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rwqnd\" (UniqueName: \"kubernetes.io/projected/f44aedd4-16f8-4adb-a7b7-51719a187834-kube-api-access-rwqnd\") pod \"insights-runtime-extractor-hhwkn\" (UID: \"f44aedd4-16f8-4adb-a7b7-51719a187834\") " pod="openshift-insights/insights-runtime-extractor-hhwkn" Jul 18 11:49:46.949974 ip-10-0-136-240 kubenswrapper[2585]: E0718 11:49:46.949960 2585 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 18 11:49:46.950189 ip-10-0-136-240 kubenswrapper[2585]: E0718 11:49:46.950017 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f44aedd4-16f8-4adb-a7b7-51719a187834-insights-runtime-extractor-tls podName:f44aedd4-16f8-4adb-a7b7-51719a187834 nodeName:}" failed. No retries permitted until 2026-07-18 11:49:47.450000557 +0000 UTC m=+54.530206892 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/f44aedd4-16f8-4adb-a7b7-51719a187834-insights-runtime-extractor-tls") pod "insights-runtime-extractor-hhwkn" (UID: "f44aedd4-16f8-4adb-a7b7-51719a187834") : secret "insights-runtime-extractor-tls" not found Jul 18 11:49:46.950189 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:49:46.950054 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/f44aedd4-16f8-4adb-a7b7-51719a187834-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-hhwkn\" (UID: \"f44aedd4-16f8-4adb-a7b7-51719a187834\") " pod="openshift-insights/insights-runtime-extractor-hhwkn" Jul 18 11:49:46.950189 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:49:46.950138 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/f44aedd4-16f8-4adb-a7b7-51719a187834-crio-socket\") pod \"insights-runtime-extractor-hhwkn\" (UID: \"f44aedd4-16f8-4adb-a7b7-51719a187834\") " pod="openshift-insights/insights-runtime-extractor-hhwkn" Jul 18 11:49:46.950189 ip-10-0-136-240 kubenswrapper[2585]: E0718 11:49:46.950145 2585 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 18 11:49:46.950334 ip-10-0-136-240 kubenswrapper[2585]: E0718 11:49:46.950205 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/f44aedd4-16f8-4adb-a7b7-51719a187834-kube-rbac-proxy-cm podName:f44aedd4-16f8-4adb-a7b7-51719a187834 nodeName:}" failed. No retries permitted until 2026-07-18 11:49:47.450193537 +0000 UTC m=+54.530399873 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/f44aedd4-16f8-4adb-a7b7-51719a187834-kube-rbac-proxy-cm") pod "insights-runtime-extractor-hhwkn" (UID: "f44aedd4-16f8-4adb-a7b7-51719a187834") : configmap "kube-rbac-proxy" not found Jul 18 11:49:46.950334 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:49:46.950208 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/f44aedd4-16f8-4adb-a7b7-51719a187834-data-volume\") pod \"insights-runtime-extractor-hhwkn\" (UID: \"f44aedd4-16f8-4adb-a7b7-51719a187834\") " pod="openshift-insights/insights-runtime-extractor-hhwkn" Jul 18 11:49:46.958194 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:49:46.958173 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rwqnd\" (UniqueName: \"kubernetes.io/projected/f44aedd4-16f8-4adb-a7b7-51719a187834-kube-api-access-rwqnd\") pod \"insights-runtime-extractor-hhwkn\" (UID: \"f44aedd4-16f8-4adb-a7b7-51719a187834\") " pod="openshift-insights/insights-runtime-extractor-hhwkn" Jul 18 11:49:47.455174 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:49:47.455142 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/f44aedd4-16f8-4adb-a7b7-51719a187834-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-hhwkn\" (UID: \"f44aedd4-16f8-4adb-a7b7-51719a187834\") " pod="openshift-insights/insights-runtime-extractor-hhwkn" Jul 18 11:49:47.455361 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:49:47.455190 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f44aedd4-16f8-4adb-a7b7-51719a187834-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-hhwkn\" (UID: \"f44aedd4-16f8-4adb-a7b7-51719a187834\") " pod="openshift-insights/insights-runtime-extractor-hhwkn" Jul 18 11:49:47.455361 ip-10-0-136-240 kubenswrapper[2585]: E0718 11:49:47.455283 2585 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 18 11:49:47.455361 ip-10-0-136-240 kubenswrapper[2585]: E0718 11:49:47.455358 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/f44aedd4-16f8-4adb-a7b7-51719a187834-kube-rbac-proxy-cm podName:f44aedd4-16f8-4adb-a7b7-51719a187834 nodeName:}" failed. No retries permitted until 2026-07-18 11:49:48.455338557 +0000 UTC m=+55.535544906 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/f44aedd4-16f8-4adb-a7b7-51719a187834-kube-rbac-proxy-cm") pod "insights-runtime-extractor-hhwkn" (UID: "f44aedd4-16f8-4adb-a7b7-51719a187834") : configmap "kube-rbac-proxy" not found Jul 18 11:49:47.455527 ip-10-0-136-240 kubenswrapper[2585]: E0718 11:49:47.455288 2585 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 18 11:49:47.455527 ip-10-0-136-240 kubenswrapper[2585]: E0718 11:49:47.455422 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f44aedd4-16f8-4adb-a7b7-51719a187834-insights-runtime-extractor-tls podName:f44aedd4-16f8-4adb-a7b7-51719a187834 nodeName:}" failed. No retries permitted until 2026-07-18 11:49:48.455409545 +0000 UTC m=+55.535615881 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/f44aedd4-16f8-4adb-a7b7-51719a187834-insights-runtime-extractor-tls") pod "insights-runtime-extractor-hhwkn" (UID: "f44aedd4-16f8-4adb-a7b7-51719a187834") : secret "insights-runtime-extractor-tls" not found Jul 18 11:49:47.810504 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:49:47.810479 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-j9l7p_8ec47eff-5fbc-4130-baef-d8544aa4e786/console-operator/1.log" Jul 18 11:49:47.811015 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:49:47.810933 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-j9l7p_8ec47eff-5fbc-4130-baef-d8544aa4e786/console-operator/0.log" Jul 18 11:49:47.811015 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:49:47.810966 2585 generic.go:356] "Generic (PLEG): container finished" podID="8ec47eff-5fbc-4130-baef-d8544aa4e786" containerID="1f0859a9aac58c8bfd25851002fba1bcdb7504d48038449b980d1fb3bdf0442a" exitCode=255 Jul 18 11:49:47.811089 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:49:47.811038 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-565b7b569d-j9l7p" event={"ID":"8ec47eff-5fbc-4130-baef-d8544aa4e786","Type":"ContainerDied","Data":"1f0859a9aac58c8bfd25851002fba1bcdb7504d48038449b980d1fb3bdf0442a"} Jul 18 11:49:47.811122 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:49:47.811101 2585 scope.go:117] "RemoveContainer" containerID="42544565cfafac737f67184300df1c02d64108a73c2fc14ee6348c5d41f25956" Jul 18 11:49:47.811537 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:49:47.811519 2585 scope.go:117] "RemoveContainer" containerID="1f0859a9aac58c8bfd25851002fba1bcdb7504d48038449b980d1fb3bdf0442a" Jul 18 11:49:47.811760 ip-10-0-136-240 kubenswrapper[2585]: E0718 11:49:47.811741 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-565b7b569d-j9l7p_openshift-console-operator(8ec47eff-5fbc-4130-baef-d8544aa4e786)\"" pod="openshift-console-operator/console-operator-565b7b569d-j9l7p" podUID="8ec47eff-5fbc-4130-baef-d8544aa4e786" Jul 18 11:49:48.464151 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:49:48.464117 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/f44aedd4-16f8-4adb-a7b7-51719a187834-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-hhwkn\" (UID: \"f44aedd4-16f8-4adb-a7b7-51719a187834\") " pod="openshift-insights/insights-runtime-extractor-hhwkn" Jul 18 11:49:48.464356 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:49:48.464179 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f44aedd4-16f8-4adb-a7b7-51719a187834-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-hhwkn\" (UID: \"f44aedd4-16f8-4adb-a7b7-51719a187834\") " pod="openshift-insights/insights-runtime-extractor-hhwkn" Jul 18 11:49:48.464356 ip-10-0-136-240 kubenswrapper[2585]: E0718 11:49:48.464242 2585 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 18 11:49:48.464356 ip-10-0-136-240 kubenswrapper[2585]: E0718 11:49:48.464307 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/f44aedd4-16f8-4adb-a7b7-51719a187834-kube-rbac-proxy-cm podName:f44aedd4-16f8-4adb-a7b7-51719a187834 nodeName:}" failed. No retries permitted until 2026-07-18 11:49:50.464289708 +0000 UTC m=+57.544496042 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/f44aedd4-16f8-4adb-a7b7-51719a187834-kube-rbac-proxy-cm") pod "insights-runtime-extractor-hhwkn" (UID: "f44aedd4-16f8-4adb-a7b7-51719a187834") : configmap "kube-rbac-proxy" not found Jul 18 11:49:48.464356 ip-10-0-136-240 kubenswrapper[2585]: E0718 11:49:48.464323 2585 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 18 11:49:48.464521 ip-10-0-136-240 kubenswrapper[2585]: E0718 11:49:48.464375 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f44aedd4-16f8-4adb-a7b7-51719a187834-insights-runtime-extractor-tls podName:f44aedd4-16f8-4adb-a7b7-51719a187834 nodeName:}" failed. No retries permitted until 2026-07-18 11:49:50.464358912 +0000 UTC m=+57.544565292 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/f44aedd4-16f8-4adb-a7b7-51719a187834-insights-runtime-extractor-tls") pod "insights-runtime-extractor-hhwkn" (UID: "f44aedd4-16f8-4adb-a7b7-51719a187834") : secret "insights-runtime-extractor-tls" not found Jul 18 11:49:48.670905 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:49:48.670879 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-hv6g8_97228923-3172-4691-927f-723ef88035cf/dns-node-resolver/0.log" Jul 18 11:49:48.816044 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:49:48.815977 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-j9l7p_8ec47eff-5fbc-4130-baef-d8544aa4e786/console-operator/1.log" Jul 18 11:49:48.816374 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:49:48.816264 2585 scope.go:117] "RemoveContainer" containerID="1f0859a9aac58c8bfd25851002fba1bcdb7504d48038449b980d1fb3bdf0442a" Jul 18 11:49:48.816419 ip-10-0-136-240 kubenswrapper[2585]: E0718 11:49:48.816402 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-565b7b569d-j9l7p_openshift-console-operator(8ec47eff-5fbc-4130-baef-d8544aa4e786)\"" pod="openshift-console-operator/console-operator-565b7b569d-j9l7p" podUID="8ec47eff-5fbc-4130-baef-d8544aa4e786" Jul 18 11:49:49.673785 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:49:49.673756 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9477954a-e6a6-4bba-ad8f-3a8788b2b2ec-service-ca-bundle\") pod \"router-default-696b95fb6d-hx9nd\" (UID: \"9477954a-e6a6-4bba-ad8f-3a8788b2b2ec\") " pod="openshift-ingress/router-default-696b95fb6d-hx9nd" Jul 18 11:49:49.673785 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:49:49.673789 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9477954a-e6a6-4bba-ad8f-3a8788b2b2ec-metrics-certs\") pod \"router-default-696b95fb6d-hx9nd\" (UID: \"9477954a-e6a6-4bba-ad8f-3a8788b2b2ec\") " pod="openshift-ingress/router-default-696b95fb6d-hx9nd" Jul 18 11:49:49.673992 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:49:49.673831 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/2a5a3d5e-ed10-4c8b-83b6-dc23ad399e00-registry-tls\") pod \"image-registry-555986d5b4-8r9nw\" (UID: \"2a5a3d5e-ed10-4c8b-83b6-dc23ad399e00\") " pod="openshift-image-registry/image-registry-555986d5b4-8r9nw" Jul 18 11:49:49.673992 ip-10-0-136-240 kubenswrapper[2585]: E0718 11:49:49.673922 2585 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 18 11:49:49.673992 ip-10-0-136-240 kubenswrapper[2585]: E0718 11:49:49.673920 2585 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 18 11:49:49.673992 ip-10-0-136-240 kubenswrapper[2585]: E0718 11:49:49.673932 2585 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-555986d5b4-8r9nw: secret "image-registry-tls" not found Jul 18 11:49:49.673992 ip-10-0-136-240 kubenswrapper[2585]: E0718 11:49:49.673928 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/9477954a-e6a6-4bba-ad8f-3a8788b2b2ec-service-ca-bundle podName:9477954a-e6a6-4bba-ad8f-3a8788b2b2ec nodeName:}" failed. No retries permitted until 2026-07-18 11:49:57.673910414 +0000 UTC m=+64.754116749 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/9477954a-e6a6-4bba-ad8f-3a8788b2b2ec-service-ca-bundle") pod "router-default-696b95fb6d-hx9nd" (UID: "9477954a-e6a6-4bba-ad8f-3a8788b2b2ec") : configmap references non-existent config key: service-ca.crt Jul 18 11:49:49.673992 ip-10-0-136-240 kubenswrapper[2585]: E0718 11:49:49.673979 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9477954a-e6a6-4bba-ad8f-3a8788b2b2ec-metrics-certs podName:9477954a-e6a6-4bba-ad8f-3a8788b2b2ec nodeName:}" failed. No retries permitted until 2026-07-18 11:49:57.673961791 +0000 UTC m=+64.754168129 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9477954a-e6a6-4bba-ad8f-3a8788b2b2ec-metrics-certs") pod "router-default-696b95fb6d-hx9nd" (UID: "9477954a-e6a6-4bba-ad8f-3a8788b2b2ec") : secret "router-metrics-certs-default" not found Jul 18 11:49:49.673992 ip-10-0-136-240 kubenswrapper[2585]: E0718 11:49:49.673996 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2a5a3d5e-ed10-4c8b-83b6-dc23ad399e00-registry-tls podName:2a5a3d5e-ed10-4c8b-83b6-dc23ad399e00 nodeName:}" failed. No retries permitted until 2026-07-18 11:49:57.673990249 +0000 UTC m=+64.754196586 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/2a5a3d5e-ed10-4c8b-83b6-dc23ad399e00-registry-tls") pod "image-registry-555986d5b4-8r9nw" (UID: "2a5a3d5e-ed10-4c8b-83b6-dc23ad399e00") : secret "image-registry-tls" not found Jul 18 11:49:50.070853 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:49:50.070820 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-nzpfs_d12e3a2e-a05d-4659-b74e-c30e765448bf/node-ca/0.log" Jul 18 11:49:50.481013 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:49:50.480979 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/f44aedd4-16f8-4adb-a7b7-51719a187834-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-hhwkn\" (UID: \"f44aedd4-16f8-4adb-a7b7-51719a187834\") " pod="openshift-insights/insights-runtime-extractor-hhwkn" Jul 18 11:49:50.481171 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:49:50.481025 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f44aedd4-16f8-4adb-a7b7-51719a187834-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-hhwkn\" (UID: \"f44aedd4-16f8-4adb-a7b7-51719a187834\") " pod="openshift-insights/insights-runtime-extractor-hhwkn" Jul 18 11:49:50.481171 ip-10-0-136-240 kubenswrapper[2585]: E0718 11:49:50.481106 2585 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 18 11:49:50.481171 ip-10-0-136-240 kubenswrapper[2585]: E0718 11:49:50.481154 2585 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 18 11:49:50.481171 ip-10-0-136-240 kubenswrapper[2585]: E0718 11:49:50.481170 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/f44aedd4-16f8-4adb-a7b7-51719a187834-kube-rbac-proxy-cm podName:f44aedd4-16f8-4adb-a7b7-51719a187834 nodeName:}" failed. No retries permitted until 2026-07-18 11:49:54.481155502 +0000 UTC m=+61.561361839 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/f44aedd4-16f8-4adb-a7b7-51719a187834-kube-rbac-proxy-cm") pod "insights-runtime-extractor-hhwkn" (UID: "f44aedd4-16f8-4adb-a7b7-51719a187834") : configmap "kube-rbac-proxy" not found Jul 18 11:49:50.481301 ip-10-0-136-240 kubenswrapper[2585]: E0718 11:49:50.481227 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f44aedd4-16f8-4adb-a7b7-51719a187834-insights-runtime-extractor-tls podName:f44aedd4-16f8-4adb-a7b7-51719a187834 nodeName:}" failed. No retries permitted until 2026-07-18 11:49:54.481210603 +0000 UTC m=+61.561416937 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/f44aedd4-16f8-4adb-a7b7-51719a187834-insights-runtime-extractor-tls") pod "insights-runtime-extractor-hhwkn" (UID: "f44aedd4-16f8-4adb-a7b7-51719a187834") : secret "insights-runtime-extractor-tls" not found Jul 18 11:49:50.870638 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:49:50.870512 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-88757_056ad49c-5e1c-4331-8a8f-65c7202be995/serve-healthcheck-canary/0.log" Jul 18 11:49:52.119975 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:49:52.119944 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-565b7b569d-j9l7p" Jul 18 11:49:52.120341 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:49:52.120026 2585 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-565b7b569d-j9l7p" Jul 18 11:49:52.120341 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:49:52.120283 2585 scope.go:117] "RemoveContainer" containerID="1f0859a9aac58c8bfd25851002fba1bcdb7504d48038449b980d1fb3bdf0442a" Jul 18 11:49:52.120434 ip-10-0-136-240 kubenswrapper[2585]: E0718 11:49:52.120416 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-565b7b569d-j9l7p_openshift-console-operator(8ec47eff-5fbc-4130-baef-d8544aa4e786)\"" pod="openshift-console-operator/console-operator-565b7b569d-j9l7p" podUID="8ec47eff-5fbc-4130-baef-d8544aa4e786" Jul 18 11:49:52.829588 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:49:52.829543 2585 scope.go:117] "RemoveContainer" containerID="1f0859a9aac58c8bfd25851002fba1bcdb7504d48038449b980d1fb3bdf0442a" Jul 18 11:49:52.829747 ip-10-0-136-240 kubenswrapper[2585]: E0718 11:49:52.829724 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-565b7b569d-j9l7p_openshift-console-operator(8ec47eff-5fbc-4130-baef-d8544aa4e786)\"" pod="openshift-console-operator/console-operator-565b7b569d-j9l7p" podUID="8ec47eff-5fbc-4130-baef-d8544aa4e786" Jul 18 11:49:54.512111 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:49:54.512076 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/f44aedd4-16f8-4adb-a7b7-51719a187834-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-hhwkn\" (UID: \"f44aedd4-16f8-4adb-a7b7-51719a187834\") " pod="openshift-insights/insights-runtime-extractor-hhwkn" Jul 18 11:49:54.512519 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:49:54.512134 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f44aedd4-16f8-4adb-a7b7-51719a187834-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-hhwkn\" (UID: \"f44aedd4-16f8-4adb-a7b7-51719a187834\") " pod="openshift-insights/insights-runtime-extractor-hhwkn" Jul 18 11:49:54.512519 ip-10-0-136-240 kubenswrapper[2585]: E0718 11:49:54.512203 2585 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 18 11:49:54.512519 ip-10-0-136-240 kubenswrapper[2585]: E0718 11:49:54.512264 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/f44aedd4-16f8-4adb-a7b7-51719a187834-kube-rbac-proxy-cm podName:f44aedd4-16f8-4adb-a7b7-51719a187834 nodeName:}" failed. No retries permitted until 2026-07-18 11:50:02.51224783 +0000 UTC m=+69.592454164 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/f44aedd4-16f8-4adb-a7b7-51719a187834-kube-rbac-proxy-cm") pod "insights-runtime-extractor-hhwkn" (UID: "f44aedd4-16f8-4adb-a7b7-51719a187834") : configmap "kube-rbac-proxy" not found Jul 18 11:49:54.512519 ip-10-0-136-240 kubenswrapper[2585]: E0718 11:49:54.512285 2585 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 18 11:49:54.512519 ip-10-0-136-240 kubenswrapper[2585]: E0718 11:49:54.512329 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f44aedd4-16f8-4adb-a7b7-51719a187834-insights-runtime-extractor-tls podName:f44aedd4-16f8-4adb-a7b7-51719a187834 nodeName:}" failed. No retries permitted until 2026-07-18 11:50:02.512315123 +0000 UTC m=+69.592521458 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/f44aedd4-16f8-4adb-a7b7-51719a187834-insights-runtime-extractor-tls") pod "insights-runtime-extractor-hhwkn" (UID: "f44aedd4-16f8-4adb-a7b7-51719a187834") : secret "insights-runtime-extractor-tls" not found Jul 18 11:49:54.731438 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:49:54.731409 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-wq4lg" Jul 18 11:49:57.741573 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:49:57.741521 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9477954a-e6a6-4bba-ad8f-3a8788b2b2ec-service-ca-bundle\") pod \"router-default-696b95fb6d-hx9nd\" (UID: \"9477954a-e6a6-4bba-ad8f-3a8788b2b2ec\") " pod="openshift-ingress/router-default-696b95fb6d-hx9nd" Jul 18 11:49:57.741573 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:49:57.741575 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9477954a-e6a6-4bba-ad8f-3a8788b2b2ec-metrics-certs\") pod \"router-default-696b95fb6d-hx9nd\" (UID: \"9477954a-e6a6-4bba-ad8f-3a8788b2b2ec\") " pod="openshift-ingress/router-default-696b95fb6d-hx9nd" Jul 18 11:49:57.741985 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:49:57.741623 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/2a5a3d5e-ed10-4c8b-83b6-dc23ad399e00-registry-tls\") pod \"image-registry-555986d5b4-8r9nw\" (UID: \"2a5a3d5e-ed10-4c8b-83b6-dc23ad399e00\") " pod="openshift-image-registry/image-registry-555986d5b4-8r9nw" Jul 18 11:49:57.741985 ip-10-0-136-240 kubenswrapper[2585]: E0718 11:49:57.741685 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/9477954a-e6a6-4bba-ad8f-3a8788b2b2ec-service-ca-bundle podName:9477954a-e6a6-4bba-ad8f-3a8788b2b2ec nodeName:}" failed. No retries permitted until 2026-07-18 11:50:13.741662439 +0000 UTC m=+80.821868782 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/9477954a-e6a6-4bba-ad8f-3a8788b2b2ec-service-ca-bundle") pod "router-default-696b95fb6d-hx9nd" (UID: "9477954a-e6a6-4bba-ad8f-3a8788b2b2ec") : configmap references non-existent config key: service-ca.crt Jul 18 11:49:57.744185 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:49:57.744155 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9477954a-e6a6-4bba-ad8f-3a8788b2b2ec-metrics-certs\") pod \"router-default-696b95fb6d-hx9nd\" (UID: \"9477954a-e6a6-4bba-ad8f-3a8788b2b2ec\") " pod="openshift-ingress/router-default-696b95fb6d-hx9nd" Jul 18 11:49:57.744299 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:49:57.744222 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/2a5a3d5e-ed10-4c8b-83b6-dc23ad399e00-registry-tls\") pod \"image-registry-555986d5b4-8r9nw\" (UID: \"2a5a3d5e-ed10-4c8b-83b6-dc23ad399e00\") " pod="openshift-image-registry/image-registry-555986d5b4-8r9nw" Jul 18 11:49:58.034935 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:49:58.034862 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-g4x2w\"" Jul 18 11:49:58.042270 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:49:58.042253 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-555986d5b4-8r9nw" Jul 18 11:49:58.168606 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:49:58.168577 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-555986d5b4-8r9nw"] Jul 18 11:49:58.171217 ip-10-0-136-240 kubenswrapper[2585]: W0718 11:49:58.171180 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2a5a3d5e_ed10_4c8b_83b6_dc23ad399e00.slice/crio-6cfc3072a5dce4640f24fd6c74dc665dbd6c2a067057d4f9e9b0d5d6c9a81641 WatchSource:0}: Error finding container 6cfc3072a5dce4640f24fd6c74dc665dbd6c2a067057d4f9e9b0d5d6c9a81641: Status 404 returned error can't find the container with id 6cfc3072a5dce4640f24fd6c74dc665dbd6c2a067057d4f9e9b0d5d6c9a81641 Jul 18 11:49:58.245603 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:49:58.245574 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4731170e-9d96-4544-a641-5fc8e1659d8e-metrics-certs\") pod \"network-metrics-daemon-hppk2\" (UID: \"4731170e-9d96-4544-a641-5fc8e1659d8e\") " pod="openshift-multus/network-metrics-daemon-hppk2" Jul 18 11:49:58.245736 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:49:58.245629 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/62e02fd7-1e2e-4f92-9989-08bb7b819cfa-original-pull-secret\") pod \"global-pull-secret-syncer-sp74l\" (UID: \"62e02fd7-1e2e-4f92-9989-08bb7b819cfa\") " pod="kube-system/global-pull-secret-syncer-sp74l" Jul 18 11:49:58.248032 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:49:58.248013 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 18 11:49:58.248115 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:49:58.248046 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 18 11:49:58.256187 ip-10-0-136-240 kubenswrapper[2585]: E0718 11:49:58.256169 2585 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Jul 18 11:49:58.256261 ip-10-0-136-240 kubenswrapper[2585]: E0718 11:49:58.256220 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4731170e-9d96-4544-a641-5fc8e1659d8e-metrics-certs podName:4731170e-9d96-4544-a641-5fc8e1659d8e nodeName:}" failed. No retries permitted until 2026-07-18 11:51:02.256205768 +0000 UTC m=+129.336412103 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4731170e-9d96-4544-a641-5fc8e1659d8e-metrics-certs") pod "network-metrics-daemon-hppk2" (UID: "4731170e-9d96-4544-a641-5fc8e1659d8e") : secret "metrics-daemon-secret" not found Jul 18 11:49:58.258229 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:49:58.258192 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/62e02fd7-1e2e-4f92-9989-08bb7b819cfa-original-pull-secret\") pod \"global-pull-secret-syncer-sp74l\" (UID: \"62e02fd7-1e2e-4f92-9989-08bb7b819cfa\") " pod="kube-system/global-pull-secret-syncer-sp74l" Jul 18 11:49:58.346937 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:49:58.346872 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-frt9b\" (UniqueName: \"kubernetes.io/projected/c1cdc202-852a-4bc3-a68c-8f5bf5baab20-kube-api-access-frt9b\") pod \"network-check-target-vd8wb\" (UID: \"c1cdc202-852a-4bc3-a68c-8f5bf5baab20\") " pod="openshift-network-diagnostics/network-check-target-vd8wb" Jul 18 11:49:58.349215 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:49:58.349199 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 18 11:49:58.359072 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:49:58.359053 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 18 11:49:58.370642 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:49:58.370618 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-frt9b\" (UniqueName: \"kubernetes.io/projected/c1cdc202-852a-4bc3-a68c-8f5bf5baab20-kube-api-access-frt9b\") pod \"network-check-target-vd8wb\" (UID: \"c1cdc202-852a-4bc3-a68c-8f5bf5baab20\") " pod="openshift-network-diagnostics/network-check-target-vd8wb" Jul 18 11:49:58.383498 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:49:58.383473 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sp74l" Jul 18 11:49:58.392818 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:49:58.392799 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-7qw87\"" Jul 18 11:49:58.400866 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:49:58.400837 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vd8wb" Jul 18 11:49:58.516502 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:49:58.516443 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-sp74l"] Jul 18 11:49:58.518941 ip-10-0-136-240 kubenswrapper[2585]: W0718 11:49:58.518912 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod62e02fd7_1e2e_4f92_9989_08bb7b819cfa.slice/crio-141a0192dfe120e417923a8b409885daf34dfc39ead1df3bfb7f0466213f1f94 WatchSource:0}: Error finding container 141a0192dfe120e417923a8b409885daf34dfc39ead1df3bfb7f0466213f1f94: Status 404 returned error can't find the container with id 141a0192dfe120e417923a8b409885daf34dfc39ead1df3bfb7f0466213f1f94 Jul 18 11:49:58.536898 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:49:58.536871 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-vd8wb"] Jul 18 11:49:58.540642 ip-10-0-136-240 kubenswrapper[2585]: W0718 11:49:58.540614 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc1cdc202_852a_4bc3_a68c_8f5bf5baab20.slice/crio-910383794fe6c49d1498cf925888a294ffa82043cc88b6be00683a6c33a92e86 WatchSource:0}: Error finding container 910383794fe6c49d1498cf925888a294ffa82043cc88b6be00683a6c33a92e86: Status 404 returned error can't find the container with id 910383794fe6c49d1498cf925888a294ffa82043cc88b6be00683a6c33a92e86 Jul 18 11:49:58.848907 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:49:58.848876 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-555986d5b4-8r9nw" event={"ID":"2a5a3d5e-ed10-4c8b-83b6-dc23ad399e00","Type":"ContainerStarted","Data":"048cafff27b351d25efedefae5330018e358645d71d5a04663586bad41aa44a8"} Jul 18 11:49:58.849361 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:49:58.848928 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-555986d5b4-8r9nw" event={"ID":"2a5a3d5e-ed10-4c8b-83b6-dc23ad399e00","Type":"ContainerStarted","Data":"6cfc3072a5dce4640f24fd6c74dc665dbd6c2a067057d4f9e9b0d5d6c9a81641"} Jul 18 11:49:58.849361 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:49:58.848969 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-555986d5b4-8r9nw" Jul 18 11:49:58.850076 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:49:58.850045 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-sp74l" event={"ID":"62e02fd7-1e2e-4f92-9989-08bb7b819cfa","Type":"ContainerStarted","Data":"141a0192dfe120e417923a8b409885daf34dfc39ead1df3bfb7f0466213f1f94"} Jul 18 11:49:58.850997 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:49:58.850974 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-vd8wb" event={"ID":"c1cdc202-852a-4bc3-a68c-8f5bf5baab20","Type":"ContainerStarted","Data":"910383794fe6c49d1498cf925888a294ffa82043cc88b6be00683a6c33a92e86"} Jul 18 11:50:01.173612 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:50:01.173575 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/42cb6807-1262-4919-856b-80c7b6d08ae2-metrics-tls\") pod \"dns-default-mrh2r\" (UID: \"42cb6807-1262-4919-856b-80c7b6d08ae2\") " pod="openshift-dns/dns-default-mrh2r" Jul 18 11:50:01.176371 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:50:01.176348 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/42cb6807-1262-4919-856b-80c7b6d08ae2-metrics-tls\") pod \"dns-default-mrh2r\" (UID: \"42cb6807-1262-4919-856b-80c7b6d08ae2\") " pod="openshift-dns/dns-default-mrh2r" Jul 18 11:50:01.397446 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:50:01.397416 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-qxztp\"" Jul 18 11:50:01.405404 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:50:01.405375 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-mrh2r" Jul 18 11:50:02.584003 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:50:02.583962 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/f44aedd4-16f8-4adb-a7b7-51719a187834-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-hhwkn\" (UID: \"f44aedd4-16f8-4adb-a7b7-51719a187834\") " pod="openshift-insights/insights-runtime-extractor-hhwkn" Jul 18 11:50:02.584477 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:50:02.584019 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f44aedd4-16f8-4adb-a7b7-51719a187834-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-hhwkn\" (UID: \"f44aedd4-16f8-4adb-a7b7-51719a187834\") " pod="openshift-insights/insights-runtime-extractor-hhwkn" Jul 18 11:50:02.584477 ip-10-0-136-240 kubenswrapper[2585]: E0718 11:50:02.584143 2585 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 18 11:50:02.584477 ip-10-0-136-240 kubenswrapper[2585]: E0718 11:50:02.584208 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f44aedd4-16f8-4adb-a7b7-51719a187834-insights-runtime-extractor-tls podName:f44aedd4-16f8-4adb-a7b7-51719a187834 nodeName:}" failed. No retries permitted until 2026-07-18 11:50:18.584187674 +0000 UTC m=+85.664394017 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/f44aedd4-16f8-4adb-a7b7-51719a187834-insights-runtime-extractor-tls") pod "insights-runtime-extractor-hhwkn" (UID: "f44aedd4-16f8-4adb-a7b7-51719a187834") : secret "insights-runtime-extractor-tls" not found Jul 18 11:50:02.584665 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:50:02.584645 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/f44aedd4-16f8-4adb-a7b7-51719a187834-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-hhwkn\" (UID: \"f44aedd4-16f8-4adb-a7b7-51719a187834\") " pod="openshift-insights/insights-runtime-extractor-hhwkn" Jul 18 11:50:03.570358 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:50:03.570271 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-555986d5b4-8r9nw" podStartSLOduration=22.570252247 podStartE2EDuration="22.570252247s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-18 11:49:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-18 11:49:58.870517027 +0000 UTC m=+65.950723385" watchObservedRunningTime="2026-07-18 11:50:03.570252247 +0000 UTC m=+70.650458608" Jul 18 11:50:03.571647 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:50:03.571626 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-mrh2r"] Jul 18 11:50:03.574873 ip-10-0-136-240 kubenswrapper[2585]: W0718 11:50:03.574853 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod42cb6807_1262_4919_856b_80c7b6d08ae2.slice/crio-49c5f3217d30ee154acb651edf60e5f83ad2a458c59a5b2337526065109ca9d6 WatchSource:0}: Error finding container 49c5f3217d30ee154acb651edf60e5f83ad2a458c59a5b2337526065109ca9d6: Status 404 returned error can't find the container with id 49c5f3217d30ee154acb651edf60e5f83ad2a458c59a5b2337526065109ca9d6 Jul 18 11:50:03.869437 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:50:03.869399 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-sp74l" event={"ID":"62e02fd7-1e2e-4f92-9989-08bb7b819cfa","Type":"ContainerStarted","Data":"ad6aee852ab112003fcf6e432a8573a527020fd6a4df89b82edb97b04e069cfa"} Jul 18 11:50:03.870931 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:50:03.870907 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-vd8wb" event={"ID":"c1cdc202-852a-4bc3-a68c-8f5bf5baab20","Type":"ContainerStarted","Data":"12763a706c780bf33a2f3620418bb1f892bd89332a1ae607fe3992cf3bf7bef8"} Jul 18 11:50:03.871057 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:50:03.870982 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-vd8wb" Jul 18 11:50:03.871958 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:50:03.871940 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-mrh2r" event={"ID":"42cb6807-1262-4919-856b-80c7b6d08ae2","Type":"ContainerStarted","Data":"49c5f3217d30ee154acb651edf60e5f83ad2a458c59a5b2337526065109ca9d6"} Jul 18 11:50:03.883707 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:50:03.883665 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-sp74l" podStartSLOduration=65.96248324 podStartE2EDuration="1m10.883649918s" totalImagesPullingTime="4.921166678s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-18 11:48:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-18 11:50:03.882995655 +0000 UTC m=+70.963202012" watchObservedRunningTime="2026-07-18 11:50:03.883649918 +0000 UTC m=+70.963856299" Jul 18 11:50:03.897274 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:50:03.897232 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-vd8wb" podStartSLOduration=66.000615085 podStartE2EDuration="1m10.897216834s" totalImagesPullingTime="4.896601749s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-18 11:48:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-18 11:50:03.896771003 +0000 UTC m=+70.976977358" watchObservedRunningTime="2026-07-18 11:50:03.897216834 +0000 UTC m=+70.977423191" Jul 18 11:50:05.463132 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:50:05.463106 2585 scope.go:117] "RemoveContainer" containerID="1f0859a9aac58c8bfd25851002fba1bcdb7504d48038449b980d1fb3bdf0442a" Jul 18 11:50:05.883111 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:50:05.883033 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-j9l7p_8ec47eff-5fbc-4130-baef-d8544aa4e786/console-operator/1.log" Jul 18 11:50:05.883262 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:50:05.883139 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-565b7b569d-j9l7p" event={"ID":"8ec47eff-5fbc-4130-baef-d8544aa4e786","Type":"ContainerStarted","Data":"8e58acfc6cbc1ecf1c738a87fa331d2910f80a8c5e9fef2e0d43dc6f48bf0da5"} Jul 18 11:50:05.883644 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:50:05.883597 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-565b7b569d-j9l7p" Jul 18 11:50:05.885393 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:50:05.885372 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-mrh2r" event={"ID":"42cb6807-1262-4919-856b-80c7b6d08ae2","Type":"ContainerStarted","Data":"807846cb694be04b0c1d8fb2541b9719f8bca3b3625dd08d733d0bb4a73e736b"} Jul 18 11:50:05.885476 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:50:05.885412 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-mrh2r" event={"ID":"42cb6807-1262-4919-856b-80c7b6d08ae2","Type":"ContainerStarted","Data":"8aa300ff376a9aed8603686da4d24210ee7c7653ee2239b35d6e2ccfc40d28c5"} Jul 18 11:50:05.885476 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:50:05.885438 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-mrh2r" Jul 18 11:50:05.901794 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:50:05.901751 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console-operator/console-operator-565b7b569d-j9l7p" podStartSLOduration=20.933086952 podStartE2EDuration="24.90173755s" totalImagesPullingTime="3.968650598s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-18 11:49:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-18 11:50:05.900934504 +0000 UTC m=+72.981140875" watchObservedRunningTime="2026-07-18 11:50:05.90173755 +0000 UTC m=+72.981943907" Jul 18 11:50:05.919362 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:50:05.919279 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-mrh2r" podStartSLOduration=35.320739364 podStartE2EDuration="36.919240554s" totalImagesPullingTime="1.59850119s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-18 11:49:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-18 11:50:05.917842544 +0000 UTC m=+72.998048902" watchObservedRunningTime="2026-07-18 11:50:05.919240554 +0000 UTC m=+72.999446902" Jul 18 11:50:06.003184 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:50:06.003157 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-565b7b569d-j9l7p" Jul 18 11:50:12.262012 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:50:12.261979 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-555986d5b4-8r9nw"] Jul 18 11:50:12.332961 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:50:12.332927 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-5bc67dcd46-rpxdk"] Jul 18 11:50:12.335822 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:50:12.335805 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-5bc67dcd46-rpxdk" Jul 18 11:50:12.338620 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:50:12.338594 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Jul 18 11:50:12.338755 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:50:12.338660 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Jul 18 11:50:12.338935 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:50:12.338919 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"default-dockercfg-fr2mp\"" Jul 18 11:50:12.352715 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:50:12.352689 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-5bc67dcd46-rpxdk"] Jul 18 11:50:12.424001 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:50:12.423969 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-6d5c77756-pkbzb"] Jul 18 11:50:12.427054 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:50:12.427033 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6d5c77756-pkbzb" Jul 18 11:50:12.439444 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:50:12.439421 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-6d5c77756-pkbzb"] Jul 18 11:50:12.463585 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:50:12.463536 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bbb7h\" (UniqueName: \"kubernetes.io/projected/cd834b16-b441-4739-a530-75ca9dacba75-kube-api-access-bbb7h\") pod \"downloads-5bc67dcd46-rpxdk\" (UID: \"cd834b16-b441-4739-a530-75ca9dacba75\") " pod="openshift-console/downloads-5bc67dcd46-rpxdk" Jul 18 11:50:12.564746 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:50:12.564663 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/b9ae0f78-c6f7-441c-a614-00c96d69dd39-trusted-ca\") pod \"image-registry-6d5c77756-pkbzb\" (UID: \"b9ae0f78-c6f7-441c-a614-00c96d69dd39\") " pod="openshift-image-registry/image-registry-6d5c77756-pkbzb" Jul 18 11:50:12.564746 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:50:12.564697 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/b9ae0f78-c6f7-441c-a614-00c96d69dd39-installation-pull-secrets\") pod \"image-registry-6d5c77756-pkbzb\" (UID: \"b9ae0f78-c6f7-441c-a614-00c96d69dd39\") " pod="openshift-image-registry/image-registry-6d5c77756-pkbzb" Jul 18 11:50:12.564746 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:50:12.564715 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b9ae0f78-c6f7-441c-a614-00c96d69dd39-bound-sa-token\") pod \"image-registry-6d5c77756-pkbzb\" (UID: \"b9ae0f78-c6f7-441c-a614-00c96d69dd39\") " pod="openshift-image-registry/image-registry-6d5c77756-pkbzb" Jul 18 11:50:12.564961 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:50:12.564761 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b9ae0f78-c6f7-441c-a614-00c96d69dd39-registry-tls\") pod \"image-registry-6d5c77756-pkbzb\" (UID: \"b9ae0f78-c6f7-441c-a614-00c96d69dd39\") " pod="openshift-image-registry/image-registry-6d5c77756-pkbzb" Jul 18 11:50:12.564961 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:50:12.564808 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8zgrs\" (UniqueName: \"kubernetes.io/projected/b9ae0f78-c6f7-441c-a614-00c96d69dd39-kube-api-access-8zgrs\") pod \"image-registry-6d5c77756-pkbzb\" (UID: \"b9ae0f78-c6f7-441c-a614-00c96d69dd39\") " pod="openshift-image-registry/image-registry-6d5c77756-pkbzb" Jul 18 11:50:12.564961 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:50:12.564840 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bbb7h\" (UniqueName: \"kubernetes.io/projected/cd834b16-b441-4739-a530-75ca9dacba75-kube-api-access-bbb7h\") pod \"downloads-5bc67dcd46-rpxdk\" (UID: \"cd834b16-b441-4739-a530-75ca9dacba75\") " pod="openshift-console/downloads-5bc67dcd46-rpxdk" Jul 18 11:50:12.564961 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:50:12.564863 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/b9ae0f78-c6f7-441c-a614-00c96d69dd39-registry-certificates\") pod \"image-registry-6d5c77756-pkbzb\" (UID: \"b9ae0f78-c6f7-441c-a614-00c96d69dd39\") " pod="openshift-image-registry/image-registry-6d5c77756-pkbzb" Jul 18 11:50:12.564961 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:50:12.564904 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/b9ae0f78-c6f7-441c-a614-00c96d69dd39-image-registry-private-configuration\") pod \"image-registry-6d5c77756-pkbzb\" (UID: \"b9ae0f78-c6f7-441c-a614-00c96d69dd39\") " pod="openshift-image-registry/image-registry-6d5c77756-pkbzb" Jul 18 11:50:12.564961 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:50:12.564927 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/b9ae0f78-c6f7-441c-a614-00c96d69dd39-ca-trust-extracted\") pod \"image-registry-6d5c77756-pkbzb\" (UID: \"b9ae0f78-c6f7-441c-a614-00c96d69dd39\") " pod="openshift-image-registry/image-registry-6d5c77756-pkbzb" Jul 18 11:50:12.575866 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:50:12.575840 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bbb7h\" (UniqueName: \"kubernetes.io/projected/cd834b16-b441-4739-a530-75ca9dacba75-kube-api-access-bbb7h\") pod \"downloads-5bc67dcd46-rpxdk\" (UID: \"cd834b16-b441-4739-a530-75ca9dacba75\") " pod="openshift-console/downloads-5bc67dcd46-rpxdk" Jul 18 11:50:12.644897 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:50:12.644859 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-5bc67dcd46-rpxdk" Jul 18 11:50:12.666272 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:50:12.666234 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/b9ae0f78-c6f7-441c-a614-00c96d69dd39-registry-certificates\") pod \"image-registry-6d5c77756-pkbzb\" (UID: \"b9ae0f78-c6f7-441c-a614-00c96d69dd39\") " pod="openshift-image-registry/image-registry-6d5c77756-pkbzb" Jul 18 11:50:12.666595 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:50:12.666301 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/b9ae0f78-c6f7-441c-a614-00c96d69dd39-image-registry-private-configuration\") pod \"image-registry-6d5c77756-pkbzb\" (UID: \"b9ae0f78-c6f7-441c-a614-00c96d69dd39\") " pod="openshift-image-registry/image-registry-6d5c77756-pkbzb" Jul 18 11:50:12.666595 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:50:12.666327 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/b9ae0f78-c6f7-441c-a614-00c96d69dd39-ca-trust-extracted\") pod \"image-registry-6d5c77756-pkbzb\" (UID: \"b9ae0f78-c6f7-441c-a614-00c96d69dd39\") " pod="openshift-image-registry/image-registry-6d5c77756-pkbzb" Jul 18 11:50:12.666595 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:50:12.666386 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/b9ae0f78-c6f7-441c-a614-00c96d69dd39-trusted-ca\") pod \"image-registry-6d5c77756-pkbzb\" (UID: \"b9ae0f78-c6f7-441c-a614-00c96d69dd39\") " pod="openshift-image-registry/image-registry-6d5c77756-pkbzb" Jul 18 11:50:12.666595 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:50:12.666408 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/b9ae0f78-c6f7-441c-a614-00c96d69dd39-installation-pull-secrets\") pod \"image-registry-6d5c77756-pkbzb\" (UID: \"b9ae0f78-c6f7-441c-a614-00c96d69dd39\") " pod="openshift-image-registry/image-registry-6d5c77756-pkbzb" Jul 18 11:50:12.666595 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:50:12.666433 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b9ae0f78-c6f7-441c-a614-00c96d69dd39-bound-sa-token\") pod \"image-registry-6d5c77756-pkbzb\" (UID: \"b9ae0f78-c6f7-441c-a614-00c96d69dd39\") " pod="openshift-image-registry/image-registry-6d5c77756-pkbzb" Jul 18 11:50:12.666595 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:50:12.666468 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b9ae0f78-c6f7-441c-a614-00c96d69dd39-registry-tls\") pod \"image-registry-6d5c77756-pkbzb\" (UID: \"b9ae0f78-c6f7-441c-a614-00c96d69dd39\") " pod="openshift-image-registry/image-registry-6d5c77756-pkbzb" Jul 18 11:50:12.667111 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:50:12.666939 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8zgrs\" (UniqueName: \"kubernetes.io/projected/b9ae0f78-c6f7-441c-a614-00c96d69dd39-kube-api-access-8zgrs\") pod \"image-registry-6d5c77756-pkbzb\" (UID: \"b9ae0f78-c6f7-441c-a614-00c96d69dd39\") " pod="openshift-image-registry/image-registry-6d5c77756-pkbzb" Jul 18 11:50:12.667111 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:50:12.666967 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/b9ae0f78-c6f7-441c-a614-00c96d69dd39-ca-trust-extracted\") pod \"image-registry-6d5c77756-pkbzb\" (UID: \"b9ae0f78-c6f7-441c-a614-00c96d69dd39\") " pod="openshift-image-registry/image-registry-6d5c77756-pkbzb" Jul 18 11:50:12.667416 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:50:12.667393 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/b9ae0f78-c6f7-441c-a614-00c96d69dd39-registry-certificates\") pod \"image-registry-6d5c77756-pkbzb\" (UID: \"b9ae0f78-c6f7-441c-a614-00c96d69dd39\") " pod="openshift-image-registry/image-registry-6d5c77756-pkbzb" Jul 18 11:50:12.667814 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:50:12.667765 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/b9ae0f78-c6f7-441c-a614-00c96d69dd39-trusted-ca\") pod \"image-registry-6d5c77756-pkbzb\" (UID: \"b9ae0f78-c6f7-441c-a614-00c96d69dd39\") " pod="openshift-image-registry/image-registry-6d5c77756-pkbzb" Jul 18 11:50:12.669258 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:50:12.669232 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/b9ae0f78-c6f7-441c-a614-00c96d69dd39-image-registry-private-configuration\") pod \"image-registry-6d5c77756-pkbzb\" (UID: \"b9ae0f78-c6f7-441c-a614-00c96d69dd39\") " pod="openshift-image-registry/image-registry-6d5c77756-pkbzb" Jul 18 11:50:12.669362 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:50:12.669283 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/b9ae0f78-c6f7-441c-a614-00c96d69dd39-installation-pull-secrets\") pod \"image-registry-6d5c77756-pkbzb\" (UID: \"b9ae0f78-c6f7-441c-a614-00c96d69dd39\") " pod="openshift-image-registry/image-registry-6d5c77756-pkbzb" Jul 18 11:50:12.669578 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:50:12.669541 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b9ae0f78-c6f7-441c-a614-00c96d69dd39-registry-tls\") pod \"image-registry-6d5c77756-pkbzb\" (UID: \"b9ae0f78-c6f7-441c-a614-00c96d69dd39\") " pod="openshift-image-registry/image-registry-6d5c77756-pkbzb" Jul 18 11:50:12.674516 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:50:12.674493 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b9ae0f78-c6f7-441c-a614-00c96d69dd39-bound-sa-token\") pod \"image-registry-6d5c77756-pkbzb\" (UID: \"b9ae0f78-c6f7-441c-a614-00c96d69dd39\") " pod="openshift-image-registry/image-registry-6d5c77756-pkbzb" Jul 18 11:50:12.674634 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:50:12.674593 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8zgrs\" (UniqueName: \"kubernetes.io/projected/b9ae0f78-c6f7-441c-a614-00c96d69dd39-kube-api-access-8zgrs\") pod \"image-registry-6d5c77756-pkbzb\" (UID: \"b9ae0f78-c6f7-441c-a614-00c96d69dd39\") " pod="openshift-image-registry/image-registry-6d5c77756-pkbzb" Jul 18 11:50:12.735888 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:50:12.735863 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6d5c77756-pkbzb" Jul 18 11:50:12.775076 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:50:12.775048 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-5bc67dcd46-rpxdk"] Jul 18 11:50:12.778930 ip-10-0-136-240 kubenswrapper[2585]: W0718 11:50:12.778905 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcd834b16_b441_4739_a530_75ca9dacba75.slice/crio-4f09ce3adea871ddd3e691836f0187b0d84ff5778bbb598869c6b0d8e3a4d79c WatchSource:0}: Error finding container 4f09ce3adea871ddd3e691836f0187b0d84ff5778bbb598869c6b0d8e3a4d79c: Status 404 returned error can't find the container with id 4f09ce3adea871ddd3e691836f0187b0d84ff5778bbb598869c6b0d8e3a4d79c Jul 18 11:50:12.865441 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:50:12.865407 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-6d5c77756-pkbzb"] Jul 18 11:50:12.867843 ip-10-0-136-240 kubenswrapper[2585]: W0718 11:50:12.867816 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb9ae0f78_c6f7_441c_a614_00c96d69dd39.slice/crio-de82aa77ac423b72f3156e822f1f4986492428f3d2e493407218621739c16532 WatchSource:0}: Error finding container de82aa77ac423b72f3156e822f1f4986492428f3d2e493407218621739c16532: Status 404 returned error can't find the container with id de82aa77ac423b72f3156e822f1f4986492428f3d2e493407218621739c16532 Jul 18 11:50:12.909539 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:50:12.909512 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6d5c77756-pkbzb" event={"ID":"b9ae0f78-c6f7-441c-a614-00c96d69dd39","Type":"ContainerStarted","Data":"de82aa77ac423b72f3156e822f1f4986492428f3d2e493407218621739c16532"} Jul 18 11:50:12.910504 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:50:12.910482 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-5bc67dcd46-rpxdk" event={"ID":"cd834b16-b441-4739-a530-75ca9dacba75","Type":"ContainerStarted","Data":"4f09ce3adea871ddd3e691836f0187b0d84ff5778bbb598869c6b0d8e3a4d79c"} Jul 18 11:50:13.775394 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:50:13.775349 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9477954a-e6a6-4bba-ad8f-3a8788b2b2ec-service-ca-bundle\") pod \"router-default-696b95fb6d-hx9nd\" (UID: \"9477954a-e6a6-4bba-ad8f-3a8788b2b2ec\") " pod="openshift-ingress/router-default-696b95fb6d-hx9nd" Jul 18 11:50:13.776045 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:50:13.776025 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9477954a-e6a6-4bba-ad8f-3a8788b2b2ec-service-ca-bundle\") pod \"router-default-696b95fb6d-hx9nd\" (UID: \"9477954a-e6a6-4bba-ad8f-3a8788b2b2ec\") " pod="openshift-ingress/router-default-696b95fb6d-hx9nd" Jul 18 11:50:13.915804 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:50:13.915763 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6d5c77756-pkbzb" event={"ID":"b9ae0f78-c6f7-441c-a614-00c96d69dd39","Type":"ContainerStarted","Data":"8f04aefc81bf66701f2b9e5928922660b0010d09fb3fc0336c344edb7949e122"} Jul 18 11:50:13.915996 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:50:13.915903 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-6d5c77756-pkbzb" Jul 18 11:50:13.937289 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:50:13.937233 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-6d5c77756-pkbzb" podStartSLOduration=1.93721414 podStartE2EDuration="1.93721414s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-18 11:50:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-18 11:50:13.935568478 +0000 UTC m=+81.015774829" watchObservedRunningTime="2026-07-18 11:50:13.93721414 +0000 UTC m=+81.017420498" Jul 18 11:50:14.007429 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:50:14.007391 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-nq2ft\"" Jul 18 11:50:14.015340 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:50:14.015319 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-696b95fb6d-hx9nd" Jul 18 11:50:14.167607 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:50:14.167581 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-696b95fb6d-hx9nd"] Jul 18 11:50:14.169320 ip-10-0-136-240 kubenswrapper[2585]: W0718 11:50:14.169289 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9477954a_e6a6_4bba_ad8f_3a8788b2b2ec.slice/crio-8322dc713d7aaf2387b18731d73e88ed29bd958eef75cca94f08d725c6fc049b WatchSource:0}: Error finding container 8322dc713d7aaf2387b18731d73e88ed29bd958eef75cca94f08d725c6fc049b: Status 404 returned error can't find the container with id 8322dc713d7aaf2387b18731d73e88ed29bd958eef75cca94f08d725c6fc049b Jul 18 11:50:14.922655 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:50:14.922613 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-696b95fb6d-hx9nd" event={"ID":"9477954a-e6a6-4bba-ad8f-3a8788b2b2ec","Type":"ContainerStarted","Data":"57a8bbeb8c257935b44a60731d8ffe702e5646367324770abc25c2c9cfe04025"} Jul 18 11:50:14.923093 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:50:14.922675 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-696b95fb6d-hx9nd" event={"ID":"9477954a-e6a6-4bba-ad8f-3a8788b2b2ec","Type":"ContainerStarted","Data":"8322dc713d7aaf2387b18731d73e88ed29bd958eef75cca94f08d725c6fc049b"} Jul 18 11:50:14.942974 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:50:14.942920 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/router-default-696b95fb6d-hx9nd" podStartSLOduration=33.942898587 podStartE2EDuration="33.942898587s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-18 11:49:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-18 11:50:14.94195919 +0000 UTC m=+82.022165573" watchObservedRunningTime="2026-07-18 11:50:14.942898587 +0000 UTC m=+82.023104945" Jul 18 11:50:15.016065 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:50:15.016029 2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-696b95fb6d-hx9nd" Jul 18 11:50:15.019030 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:50:15.019000 2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-696b95fb6d-hx9nd" Jul 18 11:50:15.891620 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:50:15.891586 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-mrh2r" Jul 18 11:50:15.926222 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:50:15.926193 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-696b95fb6d-hx9nd" Jul 18 11:50:15.927690 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:50:15.927664 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-696b95fb6d-hx9nd" Jul 18 11:50:18.617791 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:50:18.617752 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f44aedd4-16f8-4adb-a7b7-51719a187834-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-hhwkn\" (UID: \"f44aedd4-16f8-4adb-a7b7-51719a187834\") " pod="openshift-insights/insights-runtime-extractor-hhwkn" Jul 18 11:50:18.620465 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:50:18.620436 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f44aedd4-16f8-4adb-a7b7-51719a187834-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-hhwkn\" (UID: \"f44aedd4-16f8-4adb-a7b7-51719a187834\") " pod="openshift-insights/insights-runtime-extractor-hhwkn" Jul 18 11:50:18.652311 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:50:18.652145 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-29g54\"" Jul 18 11:50:18.659585 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:50:18.659543 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-hhwkn" Jul 18 11:50:18.825797 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:50:18.825666 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-hhwkn"] Jul 18 11:50:18.828753 ip-10-0-136-240 kubenswrapper[2585]: W0718 11:50:18.828727 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf44aedd4_16f8_4adb_a7b7_51719a187834.slice/crio-856698dafc5f099789c86931e4b4caca5df8709cfb6b9670ac0ac3ea59243297 WatchSource:0}: Error finding container 856698dafc5f099789c86931e4b4caca5df8709cfb6b9670ac0ac3ea59243297: Status 404 returned error can't find the container with id 856698dafc5f099789c86931e4b4caca5df8709cfb6b9670ac0ac3ea59243297 Jul 18 11:50:18.938325 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:50:18.938237 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-hhwkn" event={"ID":"f44aedd4-16f8-4adb-a7b7-51719a187834","Type":"ContainerStarted","Data":"856698dafc5f099789c86931e4b4caca5df8709cfb6b9670ac0ac3ea59243297"} Jul 18 11:50:22.268433 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:50:22.268397 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-555986d5b4-8r9nw" Jul 18 11:50:28.187147 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:50:28.187113 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-79f6958cdf-dfvf2"] Jul 18 11:50:28.192046 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:50:28.192021 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-dfvf2" Jul 18 11:50:28.195013 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:50:28.194982 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Jul 18 11:50:28.195013 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:50:28.195007 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Jul 18 11:50:28.195333 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:50:28.195316 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-zp6tm\"" Jul 18 11:50:28.195928 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:50:28.195907 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\"" Jul 18 11:50:28.196035 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:50:28.195907 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\"" Jul 18 11:50:28.200924 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:50:28.200903 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Jul 18 11:50:28.217698 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:50:28.217676 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-79f6958cdf-dfvf2"] Jul 18 11:50:28.259132 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:50:28.259101 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-56c5594674-z2dzf"] Jul 18 11:50:28.262661 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:50:28.262639 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-56c5594674-z2dzf" Jul 18 11:50:28.265277 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:50:28.265255 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\"" Jul 18 11:50:28.265395 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:50:28.265303 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\"" Jul 18 11:50:28.265715 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:50:28.265694 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\"" Jul 18 11:50:28.269308 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:50:28.269291 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-s67gg\"" Jul 18 11:50:28.272649 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:50:28.272631 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-5qrx7"] Jul 18 11:50:28.276056 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:50:28.275956 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-5qrx7" Jul 18 11:50:28.277404 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:50:28.277385 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-56c5594674-z2dzf"] Jul 18 11:50:28.280741 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:50:28.280505 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Jul 18 11:50:28.280851 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:50:28.280818 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Jul 18 11:50:28.281114 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:50:28.281092 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Jul 18 11:50:28.281544 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:50:28.281386 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-grnx4\"" Jul 18 11:50:28.303841 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:50:28.303814 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k4t92\" (UniqueName: \"kubernetes.io/projected/02675456-e24b-4ca5-aa20-55083a8495f5-kube-api-access-k4t92\") pod \"openshift-state-metrics-79f6958cdf-dfvf2\" (UID: \"02675456-e24b-4ca5-aa20-55083a8495f5\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-dfvf2" Jul 18 11:50:28.303976 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:50:28.303857 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/02675456-e24b-4ca5-aa20-55083a8495f5-metrics-client-ca\") pod \"openshift-state-metrics-79f6958cdf-dfvf2\" (UID: \"02675456-e24b-4ca5-aa20-55083a8495f5\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-dfvf2" Jul 18 11:50:28.303976 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:50:28.303907 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/02675456-e24b-4ca5-aa20-55083a8495f5-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-79f6958cdf-dfvf2\" (UID: \"02675456-e24b-4ca5-aa20-55083a8495f5\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-dfvf2" Jul 18 11:50:28.304065 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:50:28.304005 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/02675456-e24b-4ca5-aa20-55083a8495f5-openshift-state-metrics-tls\") pod \"openshift-state-metrics-79f6958cdf-dfvf2\" (UID: \"02675456-e24b-4ca5-aa20-55083a8495f5\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-dfvf2" Jul 18 11:50:28.404673 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:50:28.404637 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/2c36ca40-cd3b-4b4d-a0d0-b725909d65ca-kube-state-metrics-tls\") pod \"kube-state-metrics-56c5594674-z2dzf\" (UID: \"2c36ca40-cd3b-4b4d-a0d0-b725909d65ca\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-z2dzf" Jul 18 11:50:28.404844 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:50:28.404686 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/60e1cfe5-abff-405b-9ed9-083ec6ab7b73-node-exporter-accelerators-collector-config\") pod \"node-exporter-5qrx7\" (UID: \"60e1cfe5-abff-405b-9ed9-083ec6ab7b73\") " pod="openshift-monitoring/node-exporter-5qrx7" Jul 18 11:50:28.404844 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:50:28.404713 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nbths\" (UniqueName: \"kubernetes.io/projected/2c36ca40-cd3b-4b4d-a0d0-b725909d65ca-kube-api-access-nbths\") pod \"kube-state-metrics-56c5594674-z2dzf\" (UID: \"2c36ca40-cd3b-4b4d-a0d0-b725909d65ca\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-z2dzf" Jul 18 11:50:28.404844 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:50:28.404754 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/02675456-e24b-4ca5-aa20-55083a8495f5-openshift-state-metrics-tls\") pod \"openshift-state-metrics-79f6958cdf-dfvf2\" (UID: \"02675456-e24b-4ca5-aa20-55083a8495f5\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-dfvf2" Jul 18 11:50:28.404844 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:50:28.404815 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/2c36ca40-cd3b-4b4d-a0d0-b725909d65ca-volume-directive-shadow\") pod \"kube-state-metrics-56c5594674-z2dzf\" (UID: \"2c36ca40-cd3b-4b4d-a0d0-b725909d65ca\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-z2dzf" Jul 18 11:50:28.405037 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:50:28.404878 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2c36ca40-cd3b-4b4d-a0d0-b725909d65ca-metrics-client-ca\") pod \"kube-state-metrics-56c5594674-z2dzf\" (UID: \"2c36ca40-cd3b-4b4d-a0d0-b725909d65ca\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-z2dzf" Jul 18 11:50:28.405037 ip-10-0-136-240 kubenswrapper[2585]: E0718 11:50:28.404890 2585 secret.go:189] Couldn't get secret openshift-monitoring/openshift-state-metrics-tls: secret "openshift-state-metrics-tls" not found Jul 18 11:50:28.405037 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:50:28.404906 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/2c36ca40-cd3b-4b4d-a0d0-b725909d65ca-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-56c5594674-z2dzf\" (UID: \"2c36ca40-cd3b-4b4d-a0d0-b725909d65ca\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-z2dzf" Jul 18 11:50:28.405037 ip-10-0-136-240 kubenswrapper[2585]: E0718 11:50:28.404951 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/02675456-e24b-4ca5-aa20-55083a8495f5-openshift-state-metrics-tls podName:02675456-e24b-4ca5-aa20-55083a8495f5 nodeName:}" failed. No retries permitted until 2026-07-18 11:50:28.904934004 +0000 UTC m=+95.985140339 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "openshift-state-metrics-tls" (UniqueName: "kubernetes.io/secret/02675456-e24b-4ca5-aa20-55083a8495f5-openshift-state-metrics-tls") pod "openshift-state-metrics-79f6958cdf-dfvf2" (UID: "02675456-e24b-4ca5-aa20-55083a8495f5") : secret "openshift-state-metrics-tls" not found Jul 18 11:50:28.405037 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:50:28.405007 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k4t92\" (UniqueName: \"kubernetes.io/projected/02675456-e24b-4ca5-aa20-55083a8495f5-kube-api-access-k4t92\") pod \"openshift-state-metrics-79f6958cdf-dfvf2\" (UID: \"02675456-e24b-4ca5-aa20-55083a8495f5\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-dfvf2" Jul 18 11:50:28.405250 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:50:28.405044 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/02675456-e24b-4ca5-aa20-55083a8495f5-metrics-client-ca\") pod \"openshift-state-metrics-79f6958cdf-dfvf2\" (UID: \"02675456-e24b-4ca5-aa20-55083a8495f5\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-dfvf2" Jul 18 11:50:28.405250 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:50:28.405070 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/60e1cfe5-abff-405b-9ed9-083ec6ab7b73-root\") pod \"node-exporter-5qrx7\" (UID: \"60e1cfe5-abff-405b-9ed9-083ec6ab7b73\") " pod="openshift-monitoring/node-exporter-5qrx7" Jul 18 11:50:28.405250 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:50:28.405183 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/2c36ca40-cd3b-4b4d-a0d0-b725909d65ca-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-56c5594674-z2dzf\" (UID: \"2c36ca40-cd3b-4b4d-a0d0-b725909d65ca\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-z2dzf" Jul 18 11:50:28.405400 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:50:28.405321 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/60e1cfe5-abff-405b-9ed9-083ec6ab7b73-node-exporter-wtmp\") pod \"node-exporter-5qrx7\" (UID: \"60e1cfe5-abff-405b-9ed9-083ec6ab7b73\") " pod="openshift-monitoring/node-exporter-5qrx7" Jul 18 11:50:28.405400 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:50:28.405352 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/60e1cfe5-abff-405b-9ed9-083ec6ab7b73-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-5qrx7\" (UID: \"60e1cfe5-abff-405b-9ed9-083ec6ab7b73\") " pod="openshift-monitoring/node-exporter-5qrx7" Jul 18 11:50:28.405462 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:50:28.405410 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/02675456-e24b-4ca5-aa20-55083a8495f5-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-79f6958cdf-dfvf2\" (UID: \"02675456-e24b-4ca5-aa20-55083a8495f5\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-dfvf2" Jul 18 11:50:28.405462 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:50:28.405435 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/60e1cfe5-abff-405b-9ed9-083ec6ab7b73-metrics-client-ca\") pod \"node-exporter-5qrx7\" (UID: \"60e1cfe5-abff-405b-9ed9-083ec6ab7b73\") " pod="openshift-monitoring/node-exporter-5qrx7" Jul 18 11:50:28.405524 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:50:28.405472 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9vjrd\" (UniqueName: \"kubernetes.io/projected/60e1cfe5-abff-405b-9ed9-083ec6ab7b73-kube-api-access-9vjrd\") pod \"node-exporter-5qrx7\" (UID: \"60e1cfe5-abff-405b-9ed9-083ec6ab7b73\") " pod="openshift-monitoring/node-exporter-5qrx7" Jul 18 11:50:28.405524 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:50:28.405504 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/60e1cfe5-abff-405b-9ed9-083ec6ab7b73-node-exporter-textfile\") pod \"node-exporter-5qrx7\" (UID: \"60e1cfe5-abff-405b-9ed9-083ec6ab7b73\") " pod="openshift-monitoring/node-exporter-5qrx7" Jul 18 11:50:28.405524 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:50:28.405518 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/60e1cfe5-abff-405b-9ed9-083ec6ab7b73-node-exporter-tls\") pod \"node-exporter-5qrx7\" (UID: \"60e1cfe5-abff-405b-9ed9-083ec6ab7b73\") " pod="openshift-monitoring/node-exporter-5qrx7" Jul 18 11:50:28.405654 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:50:28.405534 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/60e1cfe5-abff-405b-9ed9-083ec6ab7b73-sys\") pod \"node-exporter-5qrx7\" (UID: \"60e1cfe5-abff-405b-9ed9-083ec6ab7b73\") " pod="openshift-monitoring/node-exporter-5qrx7" Jul 18 11:50:28.405701 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:50:28.405672 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/02675456-e24b-4ca5-aa20-55083a8495f5-metrics-client-ca\") pod \"openshift-state-metrics-79f6958cdf-dfvf2\" (UID: \"02675456-e24b-4ca5-aa20-55083a8495f5\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-dfvf2" Jul 18 11:50:28.408280 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:50:28.408254 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/02675456-e24b-4ca5-aa20-55083a8495f5-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-79f6958cdf-dfvf2\" (UID: \"02675456-e24b-4ca5-aa20-55083a8495f5\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-dfvf2" Jul 18 11:50:28.420358 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:50:28.420333 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k4t92\" (UniqueName: \"kubernetes.io/projected/02675456-e24b-4ca5-aa20-55083a8495f5-kube-api-access-k4t92\") pod \"openshift-state-metrics-79f6958cdf-dfvf2\" (UID: \"02675456-e24b-4ca5-aa20-55083a8495f5\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-dfvf2" Jul 18 11:50:28.506399 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:50:28.506366 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/2c36ca40-cd3b-4b4d-a0d0-b725909d65ca-kube-state-metrics-tls\") pod \"kube-state-metrics-56c5594674-z2dzf\" (UID: \"2c36ca40-cd3b-4b4d-a0d0-b725909d65ca\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-z2dzf" Jul 18 11:50:28.506599 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:50:28.506413 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/60e1cfe5-abff-405b-9ed9-083ec6ab7b73-node-exporter-accelerators-collector-config\") pod \"node-exporter-5qrx7\" (UID: \"60e1cfe5-abff-405b-9ed9-083ec6ab7b73\") " pod="openshift-monitoring/node-exporter-5qrx7" Jul 18 11:50:28.506599 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:50:28.506438 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nbths\" (UniqueName: \"kubernetes.io/projected/2c36ca40-cd3b-4b4d-a0d0-b725909d65ca-kube-api-access-nbths\") pod \"kube-state-metrics-56c5594674-z2dzf\" (UID: \"2c36ca40-cd3b-4b4d-a0d0-b725909d65ca\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-z2dzf" Jul 18 11:50:28.506599 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:50:28.506479 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/2c36ca40-cd3b-4b4d-a0d0-b725909d65ca-volume-directive-shadow\") pod \"kube-state-metrics-56c5594674-z2dzf\" (UID: \"2c36ca40-cd3b-4b4d-a0d0-b725909d65ca\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-z2dzf" Jul 18 11:50:28.506599 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:50:28.506509 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2c36ca40-cd3b-4b4d-a0d0-b725909d65ca-metrics-client-ca\") pod \"kube-state-metrics-56c5594674-z2dzf\" (UID: \"2c36ca40-cd3b-4b4d-a0d0-b725909d65ca\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-z2dzf" Jul 18 11:50:28.506599 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:50:28.506535 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/2c36ca40-cd3b-4b4d-a0d0-b725909d65ca-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-56c5594674-z2dzf\" (UID: \"2c36ca40-cd3b-4b4d-a0d0-b725909d65ca\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-z2dzf" Jul 18 11:50:28.506599 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:50:28.506597 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/60e1cfe5-abff-405b-9ed9-083ec6ab7b73-root\") pod \"node-exporter-5qrx7\" (UID: \"60e1cfe5-abff-405b-9ed9-083ec6ab7b73\") " pod="openshift-monitoring/node-exporter-5qrx7" Jul 18 11:50:28.506920 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:50:28.506628 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/2c36ca40-cd3b-4b4d-a0d0-b725909d65ca-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-56c5594674-z2dzf\" (UID: \"2c36ca40-cd3b-4b4d-a0d0-b725909d65ca\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-z2dzf" Jul 18 11:50:28.506920 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:50:28.506654 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/60e1cfe5-abff-405b-9ed9-083ec6ab7b73-node-exporter-wtmp\") pod \"node-exporter-5qrx7\" (UID: \"60e1cfe5-abff-405b-9ed9-083ec6ab7b73\") " pod="openshift-monitoring/node-exporter-5qrx7" Jul 18 11:50:28.506920 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:50:28.506681 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/60e1cfe5-abff-405b-9ed9-083ec6ab7b73-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-5qrx7\" (UID: \"60e1cfe5-abff-405b-9ed9-083ec6ab7b73\") " pod="openshift-monitoring/node-exporter-5qrx7" Jul 18 11:50:28.506920 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:50:28.506723 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/60e1cfe5-abff-405b-9ed9-083ec6ab7b73-metrics-client-ca\") pod \"node-exporter-5qrx7\" (UID: \"60e1cfe5-abff-405b-9ed9-083ec6ab7b73\") " pod="openshift-monitoring/node-exporter-5qrx7" Jul 18 11:50:28.506920 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:50:28.506763 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9vjrd\" (UniqueName: \"kubernetes.io/projected/60e1cfe5-abff-405b-9ed9-083ec6ab7b73-kube-api-access-9vjrd\") pod \"node-exporter-5qrx7\" (UID: \"60e1cfe5-abff-405b-9ed9-083ec6ab7b73\") " pod="openshift-monitoring/node-exporter-5qrx7" Jul 18 11:50:28.506920 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:50:28.506806 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/60e1cfe5-abff-405b-9ed9-083ec6ab7b73-node-exporter-textfile\") pod \"node-exporter-5qrx7\" (UID: \"60e1cfe5-abff-405b-9ed9-083ec6ab7b73\") " pod="openshift-monitoring/node-exporter-5qrx7" Jul 18 11:50:28.506920 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:50:28.506720 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/60e1cfe5-abff-405b-9ed9-083ec6ab7b73-root\") pod \"node-exporter-5qrx7\" (UID: \"60e1cfe5-abff-405b-9ed9-083ec6ab7b73\") " pod="openshift-monitoring/node-exporter-5qrx7" Jul 18 11:50:28.506920 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:50:28.506829 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/60e1cfe5-abff-405b-9ed9-083ec6ab7b73-node-exporter-tls\") pod \"node-exporter-5qrx7\" (UID: \"60e1cfe5-abff-405b-9ed9-083ec6ab7b73\") " pod="openshift-monitoring/node-exporter-5qrx7" Jul 18 11:50:28.506920 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:50:28.506885 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/60e1cfe5-abff-405b-9ed9-083ec6ab7b73-sys\") pod \"node-exporter-5qrx7\" (UID: \"60e1cfe5-abff-405b-9ed9-083ec6ab7b73\") " pod="openshift-monitoring/node-exporter-5qrx7" Jul 18 11:50:28.507334 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:50:28.506968 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/60e1cfe5-abff-405b-9ed9-083ec6ab7b73-node-exporter-wtmp\") pod \"node-exporter-5qrx7\" (UID: \"60e1cfe5-abff-405b-9ed9-083ec6ab7b73\") " pod="openshift-monitoring/node-exporter-5qrx7" Jul 18 11:50:28.507334 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:50:28.506991 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/60e1cfe5-abff-405b-9ed9-083ec6ab7b73-sys\") pod \"node-exporter-5qrx7\" (UID: \"60e1cfe5-abff-405b-9ed9-083ec6ab7b73\") " pod="openshift-monitoring/node-exporter-5qrx7" Jul 18 11:50:28.507334 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:50:28.507124 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/60e1cfe5-abff-405b-9ed9-083ec6ab7b73-node-exporter-accelerators-collector-config\") pod \"node-exporter-5qrx7\" (UID: \"60e1cfe5-abff-405b-9ed9-083ec6ab7b73\") " pod="openshift-monitoring/node-exporter-5qrx7" Jul 18 11:50:28.507334 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:50:28.507255 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/60e1cfe5-abff-405b-9ed9-083ec6ab7b73-node-exporter-textfile\") pod \"node-exporter-5qrx7\" (UID: \"60e1cfe5-abff-405b-9ed9-083ec6ab7b73\") " pod="openshift-monitoring/node-exporter-5qrx7" Jul 18 11:50:28.507334 ip-10-0-136-240 kubenswrapper[2585]: E0718 11:50:28.507269 2585 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found Jul 18 11:50:28.507334 ip-10-0-136-240 kubenswrapper[2585]: E0718 11:50:28.507326 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/60e1cfe5-abff-405b-9ed9-083ec6ab7b73-node-exporter-tls podName:60e1cfe5-abff-405b-9ed9-083ec6ab7b73 nodeName:}" failed. No retries permitted until 2026-07-18 11:50:29.007307134 +0000 UTC m=+96.087513473 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/60e1cfe5-abff-405b-9ed9-083ec6ab7b73-node-exporter-tls") pod "node-exporter-5qrx7" (UID: "60e1cfe5-abff-405b-9ed9-083ec6ab7b73") : secret "node-exporter-tls" not found Jul 18 11:50:28.507674 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:50:28.507517 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/2c36ca40-cd3b-4b4d-a0d0-b725909d65ca-volume-directive-shadow\") pod \"kube-state-metrics-56c5594674-z2dzf\" (UID: \"2c36ca40-cd3b-4b4d-a0d0-b725909d65ca\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-z2dzf" Jul 18 11:50:28.507674 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:50:28.507657 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/2c36ca40-cd3b-4b4d-a0d0-b725909d65ca-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-56c5594674-z2dzf\" (UID: \"2c36ca40-cd3b-4b4d-a0d0-b725909d65ca\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-z2dzf" Jul 18 11:50:28.507854 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:50:28.507827 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2c36ca40-cd3b-4b4d-a0d0-b725909d65ca-metrics-client-ca\") pod \"kube-state-metrics-56c5594674-z2dzf\" (UID: \"2c36ca40-cd3b-4b4d-a0d0-b725909d65ca\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-z2dzf" Jul 18 11:50:28.507983 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:50:28.507968 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/60e1cfe5-abff-405b-9ed9-083ec6ab7b73-metrics-client-ca\") pod \"node-exporter-5qrx7\" (UID: \"60e1cfe5-abff-405b-9ed9-083ec6ab7b73\") " pod="openshift-monitoring/node-exporter-5qrx7" Jul 18 11:50:28.509419 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:50:28.509372 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/60e1cfe5-abff-405b-9ed9-083ec6ab7b73-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-5qrx7\" (UID: \"60e1cfe5-abff-405b-9ed9-083ec6ab7b73\") " pod="openshift-monitoring/node-exporter-5qrx7" Jul 18 11:50:28.509520 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:50:28.509483 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/2c36ca40-cd3b-4b4d-a0d0-b725909d65ca-kube-state-metrics-tls\") pod \"kube-state-metrics-56c5594674-z2dzf\" (UID: \"2c36ca40-cd3b-4b4d-a0d0-b725909d65ca\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-z2dzf" Jul 18 11:50:28.510117 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:50:28.510093 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/2c36ca40-cd3b-4b4d-a0d0-b725909d65ca-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-56c5594674-z2dzf\" (UID: \"2c36ca40-cd3b-4b4d-a0d0-b725909d65ca\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-z2dzf" Jul 18 11:50:28.518990 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:50:28.518930 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nbths\" (UniqueName: \"kubernetes.io/projected/2c36ca40-cd3b-4b4d-a0d0-b725909d65ca-kube-api-access-nbths\") pod \"kube-state-metrics-56c5594674-z2dzf\" (UID: \"2c36ca40-cd3b-4b4d-a0d0-b725909d65ca\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-z2dzf" Jul 18 11:50:28.519725 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:50:28.519701 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9vjrd\" (UniqueName: \"kubernetes.io/projected/60e1cfe5-abff-405b-9ed9-083ec6ab7b73-kube-api-access-9vjrd\") pod \"node-exporter-5qrx7\" (UID: \"60e1cfe5-abff-405b-9ed9-083ec6ab7b73\") " pod="openshift-monitoring/node-exporter-5qrx7" Jul 18 11:50:28.573123 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:50:28.573090 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-56c5594674-z2dzf" Jul 18 11:50:28.910150 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:50:28.910067 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/02675456-e24b-4ca5-aa20-55083a8495f5-openshift-state-metrics-tls\") pod \"openshift-state-metrics-79f6958cdf-dfvf2\" (UID: \"02675456-e24b-4ca5-aa20-55083a8495f5\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-dfvf2" Jul 18 11:50:28.912874 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:50:28.912848 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/02675456-e24b-4ca5-aa20-55083a8495f5-openshift-state-metrics-tls\") pod \"openshift-state-metrics-79f6958cdf-dfvf2\" (UID: \"02675456-e24b-4ca5-aa20-55083a8495f5\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-dfvf2" Jul 18 11:50:29.010584 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:50:29.010534 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/60e1cfe5-abff-405b-9ed9-083ec6ab7b73-node-exporter-tls\") pod \"node-exporter-5qrx7\" (UID: \"60e1cfe5-abff-405b-9ed9-083ec6ab7b73\") " pod="openshift-monitoring/node-exporter-5qrx7" Jul 18 11:50:29.010768 ip-10-0-136-240 kubenswrapper[2585]: E0718 11:50:29.010707 2585 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found Jul 18 11:50:29.010823 ip-10-0-136-240 kubenswrapper[2585]: E0718 11:50:29.010775 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/60e1cfe5-abff-405b-9ed9-083ec6ab7b73-node-exporter-tls podName:60e1cfe5-abff-405b-9ed9-083ec6ab7b73 nodeName:}" failed. No retries permitted until 2026-07-18 11:50:30.010757279 +0000 UTC m=+97.090963614 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/60e1cfe5-abff-405b-9ed9-083ec6ab7b73-node-exporter-tls") pod "node-exporter-5qrx7" (UID: "60e1cfe5-abff-405b-9ed9-083ec6ab7b73") : secret "node-exporter-tls" not found Jul 18 11:50:29.102953 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:50:29.102918 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-dfvf2" Jul 18 11:50:29.283643 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:50:29.283604 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 18 11:50:29.290097 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:50:29.290072 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 18 11:50:29.292536 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:50:29.292515 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Jul 18 11:50:29.292686 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:50:29.292517 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Jul 18 11:50:29.292686 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:50:29.292646 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Jul 18 11:50:29.293454 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:50:29.293433 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Jul 18 11:50:29.293662 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:50:29.293639 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Jul 18 11:50:29.293775 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:50:29.293669 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Jul 18 11:50:29.293853 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:50:29.293799 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Jul 18 11:50:29.294053 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:50:29.294034 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Jul 18 11:50:29.294053 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:50:29.294050 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Jul 18 11:50:29.294159 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:50:29.294088 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-l699l\"" Jul 18 11:50:29.306443 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:50:29.306423 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 18 11:50:29.414539 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:50:29.414395 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/c6ca8f97-d51f-425d-9a79-58d7de301827-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"c6ca8f97-d51f-425d-9a79-58d7de301827\") " pod="openshift-monitoring/alertmanager-main-0" Jul 18 11:50:29.414539 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:50:29.414447 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c6ca8f97-d51f-425d-9a79-58d7de301827-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"c6ca8f97-d51f-425d-9a79-58d7de301827\") " pod="openshift-monitoring/alertmanager-main-0" Jul 18 11:50:29.414539 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:50:29.414477 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sxpl5\" (UniqueName: \"kubernetes.io/projected/c6ca8f97-d51f-425d-9a79-58d7de301827-kube-api-access-sxpl5\") pod \"alertmanager-main-0\" (UID: \"c6ca8f97-d51f-425d-9a79-58d7de301827\") " pod="openshift-monitoring/alertmanager-main-0" Jul 18 11:50:29.414863 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:50:29.414568 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/c6ca8f97-d51f-425d-9a79-58d7de301827-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"c6ca8f97-d51f-425d-9a79-58d7de301827\") " pod="openshift-monitoring/alertmanager-main-0" Jul 18 11:50:29.414863 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:50:29.414616 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/c6ca8f97-d51f-425d-9a79-58d7de301827-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"c6ca8f97-d51f-425d-9a79-58d7de301827\") " pod="openshift-monitoring/alertmanager-main-0" Jul 18 11:50:29.414863 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:50:29.414687 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/c6ca8f97-d51f-425d-9a79-58d7de301827-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"c6ca8f97-d51f-425d-9a79-58d7de301827\") " pod="openshift-monitoring/alertmanager-main-0" Jul 18 11:50:29.414863 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:50:29.414738 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/c6ca8f97-d51f-425d-9a79-58d7de301827-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"c6ca8f97-d51f-425d-9a79-58d7de301827\") " pod="openshift-monitoring/alertmanager-main-0" Jul 18 11:50:29.414863 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:50:29.414819 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/c6ca8f97-d51f-425d-9a79-58d7de301827-config-volume\") pod \"alertmanager-main-0\" (UID: \"c6ca8f97-d51f-425d-9a79-58d7de301827\") " pod="openshift-monitoring/alertmanager-main-0" Jul 18 11:50:29.415119 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:50:29.414873 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/c6ca8f97-d51f-425d-9a79-58d7de301827-config-out\") pod \"alertmanager-main-0\" (UID: \"c6ca8f97-d51f-425d-9a79-58d7de301827\") " pod="openshift-monitoring/alertmanager-main-0" Jul 18 11:50:29.415119 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:50:29.414901 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/c6ca8f97-d51f-425d-9a79-58d7de301827-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"c6ca8f97-d51f-425d-9a79-58d7de301827\") " pod="openshift-monitoring/alertmanager-main-0" Jul 18 11:50:29.415119 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:50:29.414927 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/c6ca8f97-d51f-425d-9a79-58d7de301827-web-config\") pod \"alertmanager-main-0\" (UID: \"c6ca8f97-d51f-425d-9a79-58d7de301827\") " pod="openshift-monitoring/alertmanager-main-0" Jul 18 11:50:29.415119 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:50:29.414954 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c6ca8f97-d51f-425d-9a79-58d7de301827-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"c6ca8f97-d51f-425d-9a79-58d7de301827\") " pod="openshift-monitoring/alertmanager-main-0" Jul 18 11:50:29.415119 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:50:29.414980 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/c6ca8f97-d51f-425d-9a79-58d7de301827-tls-assets\") pod \"alertmanager-main-0\" (UID: \"c6ca8f97-d51f-425d-9a79-58d7de301827\") " pod="openshift-monitoring/alertmanager-main-0" Jul 18 11:50:29.516314 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:50:29.516270 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/c6ca8f97-d51f-425d-9a79-58d7de301827-web-config\") pod \"alertmanager-main-0\" (UID: \"c6ca8f97-d51f-425d-9a79-58d7de301827\") " pod="openshift-monitoring/alertmanager-main-0" Jul 18 11:50:29.516314 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:50:29.516318 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c6ca8f97-d51f-425d-9a79-58d7de301827-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"c6ca8f97-d51f-425d-9a79-58d7de301827\") " pod="openshift-monitoring/alertmanager-main-0" Jul 18 11:50:29.516587 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:50:29.516344 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/c6ca8f97-d51f-425d-9a79-58d7de301827-tls-assets\") pod \"alertmanager-main-0\" (UID: \"c6ca8f97-d51f-425d-9a79-58d7de301827\") " pod="openshift-monitoring/alertmanager-main-0" Jul 18 11:50:29.516587 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:50:29.516406 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/c6ca8f97-d51f-425d-9a79-58d7de301827-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"c6ca8f97-d51f-425d-9a79-58d7de301827\") " pod="openshift-monitoring/alertmanager-main-0" Jul 18 11:50:29.516587 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:50:29.516430 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c6ca8f97-d51f-425d-9a79-58d7de301827-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"c6ca8f97-d51f-425d-9a79-58d7de301827\") " pod="openshift-monitoring/alertmanager-main-0" Jul 18 11:50:29.516587 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:50:29.516447 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sxpl5\" (UniqueName: \"kubernetes.io/projected/c6ca8f97-d51f-425d-9a79-58d7de301827-kube-api-access-sxpl5\") pod \"alertmanager-main-0\" (UID: \"c6ca8f97-d51f-425d-9a79-58d7de301827\") " pod="openshift-monitoring/alertmanager-main-0" Jul 18 11:50:29.516587 ip-10-0-136-240 kubenswrapper[2585]: E0718 11:50:29.516482 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/c6ca8f97-d51f-425d-9a79-58d7de301827-alertmanager-trusted-ca-bundle podName:c6ca8f97-d51f-425d-9a79-58d7de301827 nodeName:}" failed. No retries permitted until 2026-07-18 11:50:30.016456997 +0000 UTC m=+97.096663358 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "alertmanager-trusted-ca-bundle" (UniqueName: "kubernetes.io/configmap/c6ca8f97-d51f-425d-9a79-58d7de301827-alertmanager-trusted-ca-bundle") pod "alertmanager-main-0" (UID: "c6ca8f97-d51f-425d-9a79-58d7de301827") : configmap references non-existent config key: ca-bundle.crt Jul 18 11:50:29.516587 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:50:29.516533 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/c6ca8f97-d51f-425d-9a79-58d7de301827-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"c6ca8f97-d51f-425d-9a79-58d7de301827\") " pod="openshift-monitoring/alertmanager-main-0" Jul 18 11:50:29.516587 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:50:29.516587 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/c6ca8f97-d51f-425d-9a79-58d7de301827-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"c6ca8f97-d51f-425d-9a79-58d7de301827\") " pod="openshift-monitoring/alertmanager-main-0" Jul 18 11:50:29.516991 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:50:29.516627 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/c6ca8f97-d51f-425d-9a79-58d7de301827-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"c6ca8f97-d51f-425d-9a79-58d7de301827\") " pod="openshift-monitoring/alertmanager-main-0" Jul 18 11:50:29.516991 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:50:29.516659 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/c6ca8f97-d51f-425d-9a79-58d7de301827-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"c6ca8f97-d51f-425d-9a79-58d7de301827\") " pod="openshift-monitoring/alertmanager-main-0" Jul 18 11:50:29.516991 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:50:29.516723 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/c6ca8f97-d51f-425d-9a79-58d7de301827-config-volume\") pod \"alertmanager-main-0\" (UID: \"c6ca8f97-d51f-425d-9a79-58d7de301827\") " pod="openshift-monitoring/alertmanager-main-0" Jul 18 11:50:29.516991 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:50:29.516769 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/c6ca8f97-d51f-425d-9a79-58d7de301827-config-out\") pod \"alertmanager-main-0\" (UID: \"c6ca8f97-d51f-425d-9a79-58d7de301827\") " pod="openshift-monitoring/alertmanager-main-0" Jul 18 11:50:29.516991 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:50:29.516787 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/c6ca8f97-d51f-425d-9a79-58d7de301827-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"c6ca8f97-d51f-425d-9a79-58d7de301827\") " pod="openshift-monitoring/alertmanager-main-0" Jul 18 11:50:29.516991 ip-10-0-136-240 kubenswrapper[2585]: E0718 11:50:29.516894 2585 secret.go:189] Couldn't get secret openshift-monitoring/alertmanager-main-tls: secret "alertmanager-main-tls" not found Jul 18 11:50:29.516991 ip-10-0-136-240 kubenswrapper[2585]: E0718 11:50:29.516965 2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c6ca8f97-d51f-425d-9a79-58d7de301827-secret-alertmanager-main-tls podName:c6ca8f97-d51f-425d-9a79-58d7de301827 nodeName:}" failed. No retries permitted until 2026-07-18 11:50:30.016946661 +0000 UTC m=+97.097153011 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "secret-alertmanager-main-tls" (UniqueName: "kubernetes.io/secret/c6ca8f97-d51f-425d-9a79-58d7de301827-secret-alertmanager-main-tls") pod "alertmanager-main-0" (UID: "c6ca8f97-d51f-425d-9a79-58d7de301827") : secret "alertmanager-main-tls" not found Jul 18 11:50:29.518700 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:50:29.518091 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c6ca8f97-d51f-425d-9a79-58d7de301827-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"c6ca8f97-d51f-425d-9a79-58d7de301827\") " pod="openshift-monitoring/alertmanager-main-0" Jul 18 11:50:29.518700 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:50:29.518653 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/c6ca8f97-d51f-425d-9a79-58d7de301827-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"c6ca8f97-d51f-425d-9a79-58d7de301827\") " pod="openshift-monitoring/alertmanager-main-0" Jul 18 11:50:29.520846 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:50:29.520797 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/c6ca8f97-d51f-425d-9a79-58d7de301827-config-volume\") pod \"alertmanager-main-0\" (UID: \"c6ca8f97-d51f-425d-9a79-58d7de301827\") " pod="openshift-monitoring/alertmanager-main-0" Jul 18 11:50:29.521055 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:50:29.521034 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/c6ca8f97-d51f-425d-9a79-58d7de301827-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"c6ca8f97-d51f-425d-9a79-58d7de301827\") " pod="openshift-monitoring/alertmanager-main-0" Jul 18 11:50:29.521138 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:50:29.521115 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/c6ca8f97-d51f-425d-9a79-58d7de301827-config-out\") pod \"alertmanager-main-0\" (UID: \"c6ca8f97-d51f-425d-9a79-58d7de301827\") " pod="openshift-monitoring/alertmanager-main-0" Jul 18 11:50:29.521626 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:50:29.521579 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/c6ca8f97-d51f-425d-9a79-58d7de301827-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"c6ca8f97-d51f-425d-9a79-58d7de301827\") " pod="openshift-monitoring/alertmanager-main-0" Jul 18 11:50:29.521729 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:50:29.521683 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/c6ca8f97-d51f-425d-9a79-58d7de301827-tls-assets\") pod \"alertmanager-main-0\" (UID: \"c6ca8f97-d51f-425d-9a79-58d7de301827\") " pod="openshift-monitoring/alertmanager-main-0" Jul 18 11:50:29.522205 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:50:29.522184 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/c6ca8f97-d51f-425d-9a79-58d7de301827-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"c6ca8f97-d51f-425d-9a79-58d7de301827\") " pod="openshift-monitoring/alertmanager-main-0" Jul 18 11:50:29.522911 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:50:29.522884 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/c6ca8f97-d51f-425d-9a79-58d7de301827-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"c6ca8f97-d51f-425d-9a79-58d7de301827\") " pod="openshift-monitoring/alertmanager-main-0" Jul 18 11:50:29.524066 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:50:29.524048 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/c6ca8f97-d51f-425d-9a79-58d7de301827-web-config\") pod \"alertmanager-main-0\" (UID: \"c6ca8f97-d51f-425d-9a79-58d7de301827\") " pod="openshift-monitoring/alertmanager-main-0" Jul 18 11:50:29.527672 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:50:29.527649 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sxpl5\" (UniqueName: \"kubernetes.io/projected/c6ca8f97-d51f-425d-9a79-58d7de301827-kube-api-access-sxpl5\") pod \"alertmanager-main-0\" (UID: \"c6ca8f97-d51f-425d-9a79-58d7de301827\") " pod="openshift-monitoring/alertmanager-main-0" Jul 18 11:50:30.021081 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:50:30.021049 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/c6ca8f97-d51f-425d-9a79-58d7de301827-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"c6ca8f97-d51f-425d-9a79-58d7de301827\") " pod="openshift-monitoring/alertmanager-main-0" Jul 18 11:50:30.021271 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:50:30.021117 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/60e1cfe5-abff-405b-9ed9-083ec6ab7b73-node-exporter-tls\") pod \"node-exporter-5qrx7\" (UID: \"60e1cfe5-abff-405b-9ed9-083ec6ab7b73\") " pod="openshift-monitoring/node-exporter-5qrx7" Jul 18 11:50:30.021271 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:50:30.021161 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c6ca8f97-d51f-425d-9a79-58d7de301827-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"c6ca8f97-d51f-425d-9a79-58d7de301827\") " pod="openshift-monitoring/alertmanager-main-0" Jul 18 11:50:30.022132 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:50:30.022106 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c6ca8f97-d51f-425d-9a79-58d7de301827-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"c6ca8f97-d51f-425d-9a79-58d7de301827\") " pod="openshift-monitoring/alertmanager-main-0" Jul 18 11:50:30.023807 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:50:30.023786 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/60e1cfe5-abff-405b-9ed9-083ec6ab7b73-node-exporter-tls\") pod \"node-exporter-5qrx7\" (UID: \"60e1cfe5-abff-405b-9ed9-083ec6ab7b73\") " pod="openshift-monitoring/node-exporter-5qrx7" Jul 18 11:50:30.024125 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:50:30.024102 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/c6ca8f97-d51f-425d-9a79-58d7de301827-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"c6ca8f97-d51f-425d-9a79-58d7de301827\") " pod="openshift-monitoring/alertmanager-main-0" Jul 18 11:50:30.088066 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:50:30.088031 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-5qrx7" Jul 18 11:50:30.202139 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:50:30.202097 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 18 11:50:31.659958 ip-10-0-136-240 kubenswrapper[2585]: W0718 11:50:31.659927 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod60e1cfe5_abff_405b_9ed9_083ec6ab7b73.slice/crio-e1ee976d0bd62c29d74166cd6bb05262d7ee71c8ebafada37d6145cb3e1be724 WatchSource:0}: Error finding container e1ee976d0bd62c29d74166cd6bb05262d7ee71c8ebafada37d6145cb3e1be724: Status 404 returned error can't find the container with id e1ee976d0bd62c29d74166cd6bb05262d7ee71c8ebafada37d6145cb3e1be724 Jul 18 11:50:31.826915 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:50:31.826630 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-79f6958cdf-dfvf2"] Jul 18 11:50:31.830130 ip-10-0-136-240 kubenswrapper[2585]: W0718 11:50:31.830094 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod02675456_e24b_4ca5_aa20_55083a8495f5.slice/crio-16ed3a6f6d71f106c5c658f4cf64b49b5cff827fbd5a192529db31c58a00450e WatchSource:0}: Error finding container 16ed3a6f6d71f106c5c658f4cf64b49b5cff827fbd5a192529db31c58a00450e: Status 404 returned error can't find the container with id 16ed3a6f6d71f106c5c658f4cf64b49b5cff827fbd5a192529db31c58a00450e Jul 18 11:50:31.842182 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:50:31.842153 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-56c5594674-z2dzf"] Jul 18 11:50:31.873934 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:50:31.873912 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 18 11:50:31.885675 ip-10-0-136-240 kubenswrapper[2585]: W0718 11:50:31.885651 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc6ca8f97_d51f_425d_9a79_58d7de301827.slice/crio-88846bba07831830792f51e5d9e97d482669be64a10a6da8f0a4140769e297bb WatchSource:0}: Error finding container 88846bba07831830792f51e5d9e97d482669be64a10a6da8f0a4140769e297bb: Status 404 returned error can't find the container with id 88846bba07831830792f51e5d9e97d482669be64a10a6da8f0a4140769e297bb Jul 18 11:50:32.004845 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:50:32.004797 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-5bc67dcd46-rpxdk" event={"ID":"cd834b16-b441-4739-a530-75ca9dacba75","Type":"ContainerStarted","Data":"7d04d0fef907642766746257316ac1789bc164c55d53bbe9c826944494f6bbd4"} Jul 18 11:50:32.005770 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:50:32.005744 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-5bc67dcd46-rpxdk" Jul 18 11:50:32.008527 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:50:32.008489 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c6ca8f97-d51f-425d-9a79-58d7de301827","Type":"ContainerStarted","Data":"88846bba07831830792f51e5d9e97d482669be64a10a6da8f0a4140769e297bb"} Jul 18 11:50:32.010328 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:50:32.010266 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-hhwkn" event={"ID":"f44aedd4-16f8-4adb-a7b7-51719a187834","Type":"ContainerStarted","Data":"42f945373542d5812e2c7ebf498c52bd1f17e270f981468034210eec1cc8d57b"} Jul 18 11:50:32.011817 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:50:32.011749 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-5qrx7" event={"ID":"60e1cfe5-abff-405b-9ed9-083ec6ab7b73","Type":"ContainerStarted","Data":"e1ee976d0bd62c29d74166cd6bb05262d7ee71c8ebafada37d6145cb3e1be724"} Jul 18 11:50:32.012888 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:50:32.012835 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-56c5594674-z2dzf" event={"ID":"2c36ca40-cd3b-4b4d-a0d0-b725909d65ca","Type":"ContainerStarted","Data":"829db29a81d1d8743152e4d86608225ee308a76e95f708d98f84f31cac70aae3"} Jul 18 11:50:32.014177 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:50:32.014120 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-5bc67dcd46-rpxdk" Jul 18 11:50:32.015001 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:50:32.014971 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-dfvf2" event={"ID":"02675456-e24b-4ca5-aa20-55083a8495f5","Type":"ContainerStarted","Data":"d679d7f44b120f34fa7c36234caec1e0dc4361f1e4983598d77b8e3a857850b9"} Jul 18 11:50:32.015088 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:50:32.015026 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-dfvf2" event={"ID":"02675456-e24b-4ca5-aa20-55083a8495f5","Type":"ContainerStarted","Data":"924c8781e235cae90522049abe2ed071e536ecf7b7950c1700037116f9dfc8e1"} Jul 18 11:50:32.015088 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:50:32.015056 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-dfvf2" event={"ID":"02675456-e24b-4ca5-aa20-55083a8495f5","Type":"ContainerStarted","Data":"16ed3a6f6d71f106c5c658f4cf64b49b5cff827fbd5a192529db31c58a00450e"} Jul 18 11:50:32.025137 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:50:32.025095 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/downloads-5bc67dcd46-rpxdk" podStartSLOduration=1.123824625 podStartE2EDuration="20.025080402s" totalImagesPullingTime="18.901255777s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-18 11:50:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-18 11:50:32.02365346 +0000 UTC m=+99.103859817" watchObservedRunningTime="2026-07-18 11:50:32.025080402 +0000 UTC m=+99.105286758" Jul 18 11:50:32.576646 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:50:32.576267 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-5f695cc87f-9rppb"] Jul 18 11:50:32.583097 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:50:32.582670 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-5f695cc87f-9rppb" Jul 18 11:50:32.587494 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:50:32.587052 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-2eh93hv2nhvn1\"" Jul 18 11:50:32.587494 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:50:32.587363 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\"" Jul 18 11:50:32.605966 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:50:32.602483 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Jul 18 11:50:32.605966 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:50:32.602707 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\"" Jul 18 11:50:32.605966 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:50:32.603072 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\"" Jul 18 11:50:32.605966 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:50:32.605052 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-t9c2w\"" Jul 18 11:50:32.605966 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:50:32.605743 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-5f695cc87f-9rppb"] Jul 18 11:50:32.748174 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:50:32.747834 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/456885f6-d25a-4ae8-8ce2-d80c621900af-secret-metrics-server-tls\") pod \"metrics-server-5f695cc87f-9rppb\" (UID: \"456885f6-d25a-4ae8-8ce2-d80c621900af\") " pod="openshift-monitoring/metrics-server-5f695cc87f-9rppb" Jul 18 11:50:32.748174 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:50:32.747896 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/456885f6-d25a-4ae8-8ce2-d80c621900af-client-ca-bundle\") pod \"metrics-server-5f695cc87f-9rppb\" (UID: \"456885f6-d25a-4ae8-8ce2-d80c621900af\") " pod="openshift-monitoring/metrics-server-5f695cc87f-9rppb" Jul 18 11:50:32.748174 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:50:32.747926 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hd8m8\" (UniqueName: \"kubernetes.io/projected/456885f6-d25a-4ae8-8ce2-d80c621900af-kube-api-access-hd8m8\") pod \"metrics-server-5f695cc87f-9rppb\" (UID: \"456885f6-d25a-4ae8-8ce2-d80c621900af\") " pod="openshift-monitoring/metrics-server-5f695cc87f-9rppb" Jul 18 11:50:32.748174 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:50:32.747994 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/456885f6-d25a-4ae8-8ce2-d80c621900af-audit-log\") pod \"metrics-server-5f695cc87f-9rppb\" (UID: \"456885f6-d25a-4ae8-8ce2-d80c621900af\") " pod="openshift-monitoring/metrics-server-5f695cc87f-9rppb" Jul 18 11:50:32.748174 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:50:32.748067 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/456885f6-d25a-4ae8-8ce2-d80c621900af-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-5f695cc87f-9rppb\" (UID: \"456885f6-d25a-4ae8-8ce2-d80c621900af\") " pod="openshift-monitoring/metrics-server-5f695cc87f-9rppb" Jul 18 11:50:32.748174 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:50:32.748122 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/456885f6-d25a-4ae8-8ce2-d80c621900af-secret-metrics-server-client-certs\") pod \"metrics-server-5f695cc87f-9rppb\" (UID: \"456885f6-d25a-4ae8-8ce2-d80c621900af\") " pod="openshift-monitoring/metrics-server-5f695cc87f-9rppb" Jul 18 11:50:32.748174 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:50:32.748150 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/456885f6-d25a-4ae8-8ce2-d80c621900af-metrics-server-audit-profiles\") pod \"metrics-server-5f695cc87f-9rppb\" (UID: \"456885f6-d25a-4ae8-8ce2-d80c621900af\") " pod="openshift-monitoring/metrics-server-5f695cc87f-9rppb" Jul 18 11:50:32.851360 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:50:32.850457 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/456885f6-d25a-4ae8-8ce2-d80c621900af-client-ca-bundle\") pod \"metrics-server-5f695cc87f-9rppb\" (UID: \"456885f6-d25a-4ae8-8ce2-d80c621900af\") " pod="openshift-monitoring/metrics-server-5f695cc87f-9rppb" Jul 18 11:50:32.851360 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:50:32.850504 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hd8m8\" (UniqueName: \"kubernetes.io/projected/456885f6-d25a-4ae8-8ce2-d80c621900af-kube-api-access-hd8m8\") pod \"metrics-server-5f695cc87f-9rppb\" (UID: \"456885f6-d25a-4ae8-8ce2-d80c621900af\") " pod="openshift-monitoring/metrics-server-5f695cc87f-9rppb" Jul 18 11:50:32.851360 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:50:32.850574 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/456885f6-d25a-4ae8-8ce2-d80c621900af-audit-log\") pod \"metrics-server-5f695cc87f-9rppb\" (UID: \"456885f6-d25a-4ae8-8ce2-d80c621900af\") " pod="openshift-monitoring/metrics-server-5f695cc87f-9rppb" Jul 18 11:50:32.851360 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:50:32.850638 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/456885f6-d25a-4ae8-8ce2-d80c621900af-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-5f695cc87f-9rppb\" (UID: \"456885f6-d25a-4ae8-8ce2-d80c621900af\") " pod="openshift-monitoring/metrics-server-5f695cc87f-9rppb" Jul 18 11:50:32.851360 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:50:32.850696 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/456885f6-d25a-4ae8-8ce2-d80c621900af-secret-metrics-server-client-certs\") pod \"metrics-server-5f695cc87f-9rppb\" (UID: \"456885f6-d25a-4ae8-8ce2-d80c621900af\") " pod="openshift-monitoring/metrics-server-5f695cc87f-9rppb" Jul 18 11:50:32.851360 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:50:32.850722 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/456885f6-d25a-4ae8-8ce2-d80c621900af-metrics-server-audit-profiles\") pod \"metrics-server-5f695cc87f-9rppb\" (UID: \"456885f6-d25a-4ae8-8ce2-d80c621900af\") " pod="openshift-monitoring/metrics-server-5f695cc87f-9rppb" Jul 18 11:50:32.851360 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:50:32.850804 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/456885f6-d25a-4ae8-8ce2-d80c621900af-secret-metrics-server-tls\") pod \"metrics-server-5f695cc87f-9rppb\" (UID: \"456885f6-d25a-4ae8-8ce2-d80c621900af\") " pod="openshift-monitoring/metrics-server-5f695cc87f-9rppb" Jul 18 11:50:32.853879 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:50:32.851947 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/456885f6-d25a-4ae8-8ce2-d80c621900af-audit-log\") pod \"metrics-server-5f695cc87f-9rppb\" (UID: \"456885f6-d25a-4ae8-8ce2-d80c621900af\") " pod="openshift-monitoring/metrics-server-5f695cc87f-9rppb" Jul 18 11:50:32.853879 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:50:32.853185 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/456885f6-d25a-4ae8-8ce2-d80c621900af-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-5f695cc87f-9rppb\" (UID: \"456885f6-d25a-4ae8-8ce2-d80c621900af\") " pod="openshift-monitoring/metrics-server-5f695cc87f-9rppb" Jul 18 11:50:32.853879 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:50:32.853467 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/456885f6-d25a-4ae8-8ce2-d80c621900af-metrics-server-audit-profiles\") pod \"metrics-server-5f695cc87f-9rppb\" (UID: \"456885f6-d25a-4ae8-8ce2-d80c621900af\") " pod="openshift-monitoring/metrics-server-5f695cc87f-9rppb" Jul 18 11:50:32.898574 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:50:32.875258 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hd8m8\" (UniqueName: \"kubernetes.io/projected/456885f6-d25a-4ae8-8ce2-d80c621900af-kube-api-access-hd8m8\") pod \"metrics-server-5f695cc87f-9rppb\" (UID: \"456885f6-d25a-4ae8-8ce2-d80c621900af\") " pod="openshift-monitoring/metrics-server-5f695cc87f-9rppb" Jul 18 11:50:32.898574 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:50:32.878351 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/456885f6-d25a-4ae8-8ce2-d80c621900af-secret-metrics-server-tls\") pod \"metrics-server-5f695cc87f-9rppb\" (UID: \"456885f6-d25a-4ae8-8ce2-d80c621900af\") " pod="openshift-monitoring/metrics-server-5f695cc87f-9rppb" Jul 18 11:50:32.898574 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:50:32.879706 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/456885f6-d25a-4ae8-8ce2-d80c621900af-secret-metrics-server-client-certs\") pod \"metrics-server-5f695cc87f-9rppb\" (UID: \"456885f6-d25a-4ae8-8ce2-d80c621900af\") " pod="openshift-monitoring/metrics-server-5f695cc87f-9rppb" Jul 18 11:50:32.898574 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:50:32.879828 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/456885f6-d25a-4ae8-8ce2-d80c621900af-client-ca-bundle\") pod \"metrics-server-5f695cc87f-9rppb\" (UID: \"456885f6-d25a-4ae8-8ce2-d80c621900af\") " pod="openshift-monitoring/metrics-server-5f695cc87f-9rppb" Jul 18 11:50:32.940164 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:50:32.940119 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-5f695cc87f-9rppb" Jul 18 11:50:34.878741 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:50:34.878698 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-vd8wb" Jul 18 11:50:34.930159 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:50:34.929280 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-6d5c77756-pkbzb" Jul 18 11:50:37.281885 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:50:37.281841 2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-555986d5b4-8r9nw" podUID="2a5a3d5e-ed10-4c8b-83b6-dc23ad399e00" containerName="registry" containerID="cri-o://048cafff27b351d25efedefae5330018e358645d71d5a04663586bad41aa44a8" gracePeriod=30 Jul 18 11:50:38.047064 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:50:38.047023 2585 generic.go:356] "Generic (PLEG): container finished" podID="2a5a3d5e-ed10-4c8b-83b6-dc23ad399e00" containerID="048cafff27b351d25efedefae5330018e358645d71d5a04663586bad41aa44a8" exitCode=0 Jul 18 11:50:38.047324 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:50:38.047110 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-555986d5b4-8r9nw" event={"ID":"2a5a3d5e-ed10-4c8b-83b6-dc23ad399e00","Type":"ContainerDied","Data":"048cafff27b351d25efedefae5330018e358645d71d5a04663586bad41aa44a8"} Jul 18 11:50:38.562235 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:50:38.562185 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-555986d5b4-8r9nw" Jul 18 11:50:38.611632 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:50:38.610147 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/2a5a3d5e-ed10-4c8b-83b6-dc23ad399e00-installation-pull-secrets\") pod \"2a5a3d5e-ed10-4c8b-83b6-dc23ad399e00\" (UID: \"2a5a3d5e-ed10-4c8b-83b6-dc23ad399e00\") " Jul 18 11:50:38.611632 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:50:38.610193 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/2a5a3d5e-ed10-4c8b-83b6-dc23ad399e00-registry-certificates\") pod \"2a5a3d5e-ed10-4c8b-83b6-dc23ad399e00\" (UID: \"2a5a3d5e-ed10-4c8b-83b6-dc23ad399e00\") " Jul 18 11:50:38.611632 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:50:38.610582 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2a5a3d5e-ed10-4c8b-83b6-dc23ad399e00-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "2a5a3d5e-ed10-4c8b-83b6-dc23ad399e00" (UID: "2a5a3d5e-ed10-4c8b-83b6-dc23ad399e00"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 18 11:50:38.611632 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:50:38.610739 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/2a5a3d5e-ed10-4c8b-83b6-dc23ad399e00-ca-trust-extracted\") pod \"2a5a3d5e-ed10-4c8b-83b6-dc23ad399e00\" (UID: \"2a5a3d5e-ed10-4c8b-83b6-dc23ad399e00\") " Jul 18 11:50:38.611632 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:50:38.610804 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/2a5a3d5e-ed10-4c8b-83b6-dc23ad399e00-bound-sa-token\") pod \"2a5a3d5e-ed10-4c8b-83b6-dc23ad399e00\" (UID: \"2a5a3d5e-ed10-4c8b-83b6-dc23ad399e00\") " Jul 18 11:50:38.611632 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:50:38.610833 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/2a5a3d5e-ed10-4c8b-83b6-dc23ad399e00-registry-tls\") pod \"2a5a3d5e-ed10-4c8b-83b6-dc23ad399e00\" (UID: \"2a5a3d5e-ed10-4c8b-83b6-dc23ad399e00\") " Jul 18 11:50:38.611632 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:50:38.610889 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4g27h\" (UniqueName: \"kubernetes.io/projected/2a5a3d5e-ed10-4c8b-83b6-dc23ad399e00-kube-api-access-4g27h\") pod \"2a5a3d5e-ed10-4c8b-83b6-dc23ad399e00\" (UID: \"2a5a3d5e-ed10-4c8b-83b6-dc23ad399e00\") " Jul 18 11:50:38.611632 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:50:38.610932 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/2a5a3d5e-ed10-4c8b-83b6-dc23ad399e00-image-registry-private-configuration\") pod \"2a5a3d5e-ed10-4c8b-83b6-dc23ad399e00\" (UID: \"2a5a3d5e-ed10-4c8b-83b6-dc23ad399e00\") " Jul 18 11:50:38.611632 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:50:38.610973 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/2a5a3d5e-ed10-4c8b-83b6-dc23ad399e00-trusted-ca\") pod \"2a5a3d5e-ed10-4c8b-83b6-dc23ad399e00\" (UID: \"2a5a3d5e-ed10-4c8b-83b6-dc23ad399e00\") " Jul 18 11:50:38.611632 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:50:38.611223 2585 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/2a5a3d5e-ed10-4c8b-83b6-dc23ad399e00-registry-certificates\") on node \"ip-10-0-136-240.ec2.internal\" DevicePath \"\"" Jul 18 11:50:38.611632 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:50:38.611608 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2a5a3d5e-ed10-4c8b-83b6-dc23ad399e00-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "2a5a3d5e-ed10-4c8b-83b6-dc23ad399e00" (UID: "2a5a3d5e-ed10-4c8b-83b6-dc23ad399e00"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 18 11:50:38.613463 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:50:38.613314 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2a5a3d5e-ed10-4c8b-83b6-dc23ad399e00-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "2a5a3d5e-ed10-4c8b-83b6-dc23ad399e00" (UID: "2a5a3d5e-ed10-4c8b-83b6-dc23ad399e00"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 18 11:50:38.616341 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:50:38.615655 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2a5a3d5e-ed10-4c8b-83b6-dc23ad399e00-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "2a5a3d5e-ed10-4c8b-83b6-dc23ad399e00" (UID: "2a5a3d5e-ed10-4c8b-83b6-dc23ad399e00"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 18 11:50:38.616341 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:50:38.615751 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2a5a3d5e-ed10-4c8b-83b6-dc23ad399e00-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "2a5a3d5e-ed10-4c8b-83b6-dc23ad399e00" (UID: "2a5a3d5e-ed10-4c8b-83b6-dc23ad399e00"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 18 11:50:38.616341 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:50:38.615964 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2a5a3d5e-ed10-4c8b-83b6-dc23ad399e00-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "2a5a3d5e-ed10-4c8b-83b6-dc23ad399e00" (UID: "2a5a3d5e-ed10-4c8b-83b6-dc23ad399e00"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 18 11:50:38.617845 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:50:38.617824 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2a5a3d5e-ed10-4c8b-83b6-dc23ad399e00-kube-api-access-4g27h" (OuterVolumeSpecName: "kube-api-access-4g27h") pod "2a5a3d5e-ed10-4c8b-83b6-dc23ad399e00" (UID: "2a5a3d5e-ed10-4c8b-83b6-dc23ad399e00"). InnerVolumeSpecName "kube-api-access-4g27h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 18 11:50:38.622620 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:50:38.622595 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a5a3d5e-ed10-4c8b-83b6-dc23ad399e00-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "2a5a3d5e-ed10-4c8b-83b6-dc23ad399e00" (UID: "2a5a3d5e-ed10-4c8b-83b6-dc23ad399e00"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 18 11:50:38.679175 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:50:38.679145 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-5f695cc87f-9rppb"] Jul 18 11:50:38.687283 ip-10-0-136-240 kubenswrapper[2585]: W0718 11:50:38.687237 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod456885f6_d25a_4ae8_8ce2_d80c621900af.slice/crio-ffe81cbf35d8ce48c57ff378cfbc7c153a1381fcbcbef85f0d3418531363d169 WatchSource:0}: Error finding container ffe81cbf35d8ce48c57ff378cfbc7c153a1381fcbcbef85f0d3418531363d169: Status 404 returned error can't find the container with id ffe81cbf35d8ce48c57ff378cfbc7c153a1381fcbcbef85f0d3418531363d169 Jul 18 11:50:38.711874 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:50:38.711740 2585 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/2a5a3d5e-ed10-4c8b-83b6-dc23ad399e00-trusted-ca\") on node \"ip-10-0-136-240.ec2.internal\" DevicePath \"\"" Jul 18 11:50:38.711874 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:50:38.711772 2585 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/2a5a3d5e-ed10-4c8b-83b6-dc23ad399e00-installation-pull-secrets\") on node \"ip-10-0-136-240.ec2.internal\" DevicePath \"\"" Jul 18 11:50:38.711874 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:50:38.711790 2585 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/2a5a3d5e-ed10-4c8b-83b6-dc23ad399e00-ca-trust-extracted\") on node \"ip-10-0-136-240.ec2.internal\" DevicePath \"\"" Jul 18 11:50:38.711874 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:50:38.711806 2585 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/2a5a3d5e-ed10-4c8b-83b6-dc23ad399e00-bound-sa-token\") on node \"ip-10-0-136-240.ec2.internal\" DevicePath \"\"" Jul 18 11:50:38.711874 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:50:38.711819 2585 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/2a5a3d5e-ed10-4c8b-83b6-dc23ad399e00-registry-tls\") on node \"ip-10-0-136-240.ec2.internal\" DevicePath \"\"" Jul 18 11:50:38.711874 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:50:38.711833 2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4g27h\" (UniqueName: \"kubernetes.io/projected/2a5a3d5e-ed10-4c8b-83b6-dc23ad399e00-kube-api-access-4g27h\") on node \"ip-10-0-136-240.ec2.internal\" DevicePath \"\"" Jul 18 11:50:38.711874 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:50:38.711847 2585 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/2a5a3d5e-ed10-4c8b-83b6-dc23ad399e00-image-registry-private-configuration\") on node \"ip-10-0-136-240.ec2.internal\" DevicePath \"\"" Jul 18 11:50:39.053869 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:50:39.053748 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-555986d5b4-8r9nw" event={"ID":"2a5a3d5e-ed10-4c8b-83b6-dc23ad399e00","Type":"ContainerDied","Data":"6cfc3072a5dce4640f24fd6c74dc665dbd6c2a067057d4f9e9b0d5d6c9a81641"} Jul 18 11:50:39.053869 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:50:39.053809 2585 scope.go:117] "RemoveContainer" containerID="048cafff27b351d25efedefae5330018e358645d71d5a04663586bad41aa44a8" Jul 18 11:50:39.054003 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:50:39.053977 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-555986d5b4-8r9nw" Jul 18 11:50:39.058719 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:50:39.057971 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-dfvf2" event={"ID":"02675456-e24b-4ca5-aa20-55083a8495f5","Type":"ContainerStarted","Data":"de1783cfb450eb248b34b8d663e42ff0cbd6be79f924194d739078dae871bd93"} Jul 18 11:50:39.063960 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:50:39.063935 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-hhwkn" event={"ID":"f44aedd4-16f8-4adb-a7b7-51719a187834","Type":"ContainerStarted","Data":"09aedb88de2e4bea1015f81942254454f2327a6964999a56e63506ad13537d0b"} Jul 18 11:50:39.066326 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:50:39.066305 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-56c5594674-z2dzf" event={"ID":"2c36ca40-cd3b-4b4d-a0d0-b725909d65ca","Type":"ContainerStarted","Data":"9bb7f48a60784da70ff5ee4323af5ca8ef931789a6ae3132ecc82ff7cb2eec86"} Jul 18 11:50:39.066930 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:50:39.066901 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-56c5594674-z2dzf" event={"ID":"2c36ca40-cd3b-4b4d-a0d0-b725909d65ca","Type":"ContainerStarted","Data":"0c7dee2e70d08fe9a0756d54865c3cb6ee00078624555f51b49f51248a2970ca"} Jul 18 11:50:39.069174 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:50:39.068255 2585 generic.go:356] "Generic (PLEG): container finished" podID="c6ca8f97-d51f-425d-9a79-58d7de301827" containerID="a330e7b7367e8a66c3ab41ca82022386acc7dd1bfc4a48cd781c1e91fe2152f7" exitCode=0 Jul 18 11:50:39.069174 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:50:39.068318 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c6ca8f97-d51f-425d-9a79-58d7de301827","Type":"ContainerDied","Data":"a330e7b7367e8a66c3ab41ca82022386acc7dd1bfc4a48cd781c1e91fe2152f7"} Jul 18 11:50:39.073405 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:50:39.073384 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-5f695cc87f-9rppb" event={"ID":"456885f6-d25a-4ae8-8ce2-d80c621900af","Type":"ContainerStarted","Data":"ffe81cbf35d8ce48c57ff378cfbc7c153a1381fcbcbef85f0d3418531363d169"} Jul 18 11:50:39.075882 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:50:39.075859 2585 generic.go:356] "Generic (PLEG): container finished" podID="60e1cfe5-abff-405b-9ed9-083ec6ab7b73" containerID="ceaf5b072e65fc9a786411ec89c46f692fa6f766005aa2822d65b8b3fd4bcdd7" exitCode=0 Jul 18 11:50:39.076006 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:50:39.075991 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-5qrx7" event={"ID":"60e1cfe5-abff-405b-9ed9-083ec6ab7b73","Type":"ContainerDied","Data":"ceaf5b072e65fc9a786411ec89c46f692fa6f766005aa2822d65b8b3fd4bcdd7"} Jul 18 11:50:39.078994 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:50:39.078270 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-dfvf2" podStartSLOduration=4.541241225 podStartE2EDuration="11.078252045s" totalImagesPullingTime="6.53701082s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-18 11:50:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-18 11:50:39.075942445 +0000 UTC m=+106.156148806" watchObservedRunningTime="2026-07-18 11:50:39.078252045 +0000 UTC m=+106.158458403" Jul 18 11:50:39.093967 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:50:39.093842 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-555986d5b4-8r9nw"] Jul 18 11:50:39.099122 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:50:39.099055 2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-555986d5b4-8r9nw"] Jul 18 11:50:39.471209 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:50:39.471170 2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2a5a3d5e-ed10-4c8b-83b6-dc23ad399e00" path="/var/lib/kubelet/pods/2a5a3d5e-ed10-4c8b-83b6-dc23ad399e00/volumes" Jul 18 11:50:40.094579 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:50:40.094241 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-5qrx7" event={"ID":"60e1cfe5-abff-405b-9ed9-083ec6ab7b73","Type":"ContainerStarted","Data":"79d03aa2aeac5e601d2899ebfa723250f6e460d71b9c7cfd756f19c558c3cce3"} Jul 18 11:50:40.094579 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:50:40.094315 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-5qrx7" event={"ID":"60e1cfe5-abff-405b-9ed9-083ec6ab7b73","Type":"ContainerStarted","Data":"4bc78dae700177b4d7e48d912d684274fe5dba57e0d8c3b7efc83a6e2fc7d009"} Jul 18 11:50:40.105855 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:50:40.105776 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-56c5594674-z2dzf" event={"ID":"2c36ca40-cd3b-4b4d-a0d0-b725909d65ca","Type":"ContainerStarted","Data":"9cc2ec823edb011efec44ae5a3237e3760e46e94fb98452978479665ee882649"} Jul 18 11:50:40.117406 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:50:40.117349 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-5qrx7" podStartSLOduration=6.166536319 podStartE2EDuration="12.117328695s" totalImagesPullingTime="5.830636634s" totalInitContainerRuntime="120.155742ms" isStatefulPod=true podCreationTimestamp="2026-07-18 11:50:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-18 11:50:40.115383473 +0000 UTC m=+107.195590022" watchObservedRunningTime="2026-07-18 11:50:40.117328695 +0000 UTC m=+107.197535033" Jul 18 11:50:40.137196 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:50:40.137102 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-56c5594674-z2dzf" podStartSLOduration=5.469700538 podStartE2EDuration="12.137054394s" totalImagesPullingTime="6.667353856s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-18 11:50:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-18 11:50:40.13443391 +0000 UTC m=+107.214640267" watchObservedRunningTime="2026-07-18 11:50:40.137054394 +0000 UTC m=+107.217260751" Jul 18 11:50:44.131307 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:50:44.130785 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-hhwkn" event={"ID":"f44aedd4-16f8-4adb-a7b7-51719a187834","Type":"ContainerStarted","Data":"f8423645ff6a29f11f03de715adf2c83d54256797897864f23a3763b0d154987"} Jul 18 11:50:44.134412 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:50:44.134365 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c6ca8f97-d51f-425d-9a79-58d7de301827","Type":"ContainerStarted","Data":"a2bcc3b9545ecaea143a4faaf0426d4d201a03836c8aaec724652b1c627911d3"} Jul 18 11:50:44.134523 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:50:44.134440 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c6ca8f97-d51f-425d-9a79-58d7de301827","Type":"ContainerStarted","Data":"ede1d4011172c191d304250d33ab432e447c2e304f9247e66423be9dd2933905"} Jul 18 11:50:44.134523 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:50:44.134483 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c6ca8f97-d51f-425d-9a79-58d7de301827","Type":"ContainerStarted","Data":"094b0d373fe232aab9c5e25af46c31a499294e42343c438ff2f592ec253040a7"} Jul 18 11:50:44.134523 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:50:44.134522 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c6ca8f97-d51f-425d-9a79-58d7de301827","Type":"ContainerStarted","Data":"bd98009f88d1bca9757902ead251ad6f52791c541a99d451b51159c981ad0101"} Jul 18 11:50:44.136024 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:50:44.136001 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-5f695cc87f-9rppb" event={"ID":"456885f6-d25a-4ae8-8ce2-d80c621900af","Type":"ContainerStarted","Data":"e966637e17ac4ac3e28dade7c36b2b57ac4d19f9f45b75939c9767d190d3370f"} Jul 18 11:50:44.153184 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:50:44.153130 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-hhwkn" podStartSLOduration=33.321894935 podStartE2EDuration="58.153111152s" totalImagesPullingTime="24.831216217s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-18 11:49:46 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-18 11:50:44.150939522 +0000 UTC m=+111.231145881" watchObservedRunningTime="2026-07-18 11:50:44.153111152 +0000 UTC m=+111.233317510" Jul 18 11:50:44.172885 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:50:44.172657 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-5f695cc87f-9rppb" podStartSLOduration=7.295958321 podStartE2EDuration="12.172630574s" totalImagesPullingTime="4.876672253s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-18 11:50:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-18 11:50:44.171737115 +0000 UTC m=+111.251943474" watchObservedRunningTime="2026-07-18 11:50:44.172630574 +0000 UTC m=+111.252836935" Jul 18 11:50:45.143223 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:50:45.143186 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c6ca8f97-d51f-425d-9a79-58d7de301827","Type":"ContainerStarted","Data":"c6a2b3bbe0f511a080fe8adfebd97e4e4d116f696499dd0764a1675cd7bba11b"} Jul 18 11:50:46.149910 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:50:46.149877 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c6ca8f97-d51f-425d-9a79-58d7de301827","Type":"ContainerStarted","Data":"f8e7904558f7c187e381f6f2fdfa752d5300458ab47f9ce040276a855fa58e0d"} Jul 18 11:50:46.175709 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:50:46.175659 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=4.942644468 podStartE2EDuration="17.1756429s" totalImagesPullingTime="12.205144852s" totalInitContainerRuntime="27.85358ms" isStatefulPod=false podCreationTimestamp="2026-07-18 11:50:29 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-18 11:50:46.174266441 +0000 UTC m=+113.254472810" watchObservedRunningTime="2026-07-18 11:50:46.1756429 +0000 UTC m=+113.255849258" Jul 18 11:50:52.941224 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:50:52.941187 2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-5f695cc87f-9rppb" Jul 18 11:50:52.941712 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:50:52.941318 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-5f695cc87f-9rppb" Jul 18 11:50:58.197792 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:50:58.197747 2585 generic.go:356] "Generic (PLEG): container finished" podID="45cc3bd5-b19f-4ac1-9257-6c39802b8e2f" containerID="0aad6b21f4690d70307e6bfe3ffe63e0614b5a15c4361c1c8f55261f44ef95ff" exitCode=0 Jul 18 11:50:58.198249 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:50:58.197878 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-29ngm" event={"ID":"45cc3bd5-b19f-4ac1-9257-6c39802b8e2f","Type":"ContainerDied","Data":"0aad6b21f4690d70307e6bfe3ffe63e0614b5a15c4361c1c8f55261f44ef95ff"} Jul 18 11:50:58.198351 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:50:58.198330 2585 scope.go:117] "RemoveContainer" containerID="0aad6b21f4690d70307e6bfe3ffe63e0614b5a15c4361c1c8f55261f44ef95ff" Jul 18 11:50:59.203741 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:50:59.203710 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-29ngm" event={"ID":"45cc3bd5-b19f-4ac1-9257-6c39802b8e2f","Type":"ContainerStarted","Data":"2eb0c1e1492c06643e9986eb3bcec1883450e030602b5464bfd44319fd06e60e"} Jul 18 11:51:02.326464 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:51:02.326425 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4731170e-9d96-4544-a641-5fc8e1659d8e-metrics-certs\") pod \"network-metrics-daemon-hppk2\" (UID: \"4731170e-9d96-4544-a641-5fc8e1659d8e\") " pod="openshift-multus/network-metrics-daemon-hppk2" Jul 18 11:51:02.328935 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:51:02.328912 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4731170e-9d96-4544-a641-5fc8e1659d8e-metrics-certs\") pod \"network-metrics-daemon-hppk2\" (UID: \"4731170e-9d96-4544-a641-5fc8e1659d8e\") " pod="openshift-multus/network-metrics-daemon-hppk2" Jul 18 11:51:02.577372 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:51:02.577291 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-6q4p9\"" Jul 18 11:51:02.585237 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:51:02.585220 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hppk2" Jul 18 11:51:02.737507 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:51:02.737480 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-hppk2"] Jul 18 11:51:02.739324 ip-10-0-136-240 kubenswrapper[2585]: W0718 11:51:02.739296 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4731170e_9d96_4544_a641_5fc8e1659d8e.slice/crio-12969d1ad2e3e9ec4fc65e699432c85eff4d26dfb877c0ae3d37e7e56810a28f WatchSource:0}: Error finding container 12969d1ad2e3e9ec4fc65e699432c85eff4d26dfb877c0ae3d37e7e56810a28f: Status 404 returned error can't find the container with id 12969d1ad2e3e9ec4fc65e699432c85eff4d26dfb877c0ae3d37e7e56810a28f Jul 18 11:51:03.221003 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:51:03.220969 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-hppk2" event={"ID":"4731170e-9d96-4544-a641-5fc8e1659d8e","Type":"ContainerStarted","Data":"12969d1ad2e3e9ec4fc65e699432c85eff4d26dfb877c0ae3d37e7e56810a28f"} Jul 18 11:51:05.231519 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:51:05.231482 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-hppk2" event={"ID":"4731170e-9d96-4544-a641-5fc8e1659d8e","Type":"ContainerStarted","Data":"f69f1cf09324205c6e91778b6bbb3b00866a2a261f07fe7228dd0efd2537ab8f"} Jul 18 11:51:05.231987 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:51:05.231542 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-hppk2" event={"ID":"4731170e-9d96-4544-a641-5fc8e1659d8e","Type":"ContainerStarted","Data":"7e3e3c693a2169e17a0729071e8b8d62955ae8436566b680e552d351695cfd28"} Jul 18 11:51:05.247854 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:51:05.247807 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-hppk2" podStartSLOduration=130.87820686 podStartE2EDuration="2m12.247790744s" totalImagesPullingTime="1.369583884s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-18 11:48:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-18 11:51:05.245815696 +0000 UTC m=+132.326022071" watchObservedRunningTime="2026-07-18 11:51:05.247790744 +0000 UTC m=+132.327997100" Jul 18 11:51:12.945670 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:51:12.945635 2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-5f695cc87f-9rppb" Jul 18 11:51:12.949476 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:51:12.949446 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-5f695cc87f-9rppb" Jul 18 11:51:17.278588 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:51:17.278534 2585 generic.go:356] "Generic (PLEG): container finished" podID="43f38029-3002-493c-a718-2fe047c3cd99" containerID="2d2217de63ea65770ab2586b286489496b4872c915f4a2155fbd913035bd376f" exitCode=0 Jul 18 11:51:17.279036 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:51:17.278611 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-555d9d7885-44h2s" event={"ID":"43f38029-3002-493c-a718-2fe047c3cd99","Type":"ContainerDied","Data":"2d2217de63ea65770ab2586b286489496b4872c915f4a2155fbd913035bd376f"} Jul 18 11:51:17.279036 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:51:17.278944 2585 scope.go:117] "RemoveContainer" containerID="2d2217de63ea65770ab2586b286489496b4872c915f4a2155fbd913035bd376f" Jul 18 11:51:18.284643 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:51:18.284607 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-555d9d7885-44h2s" event={"ID":"43f38029-3002-493c-a718-2fe047c3cd99","Type":"ContainerStarted","Data":"9ab39e1418c38b9fbc7d1568bb7df896175b676e37a1538ed3ca30e791f30bb5"} Jul 18 11:51:52.441915 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:51:52.441721 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-7f6cccbddb-lztkh"] Jul 18 11:51:52.442419 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:51:52.442079 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2a5a3d5e-ed10-4c8b-83b6-dc23ad399e00" containerName="registry" Jul 18 11:51:52.442419 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:51:52.442096 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="2a5a3d5e-ed10-4c8b-83b6-dc23ad399e00" containerName="registry" Jul 18 11:51:52.442419 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:51:52.442183 2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="2a5a3d5e-ed10-4c8b-83b6-dc23ad399e00" containerName="registry" Jul 18 11:51:52.445587 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:51:52.445570 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-7f6cccbddb-lztkh" Jul 18 11:51:52.447902 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:51:52.447880 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\"" Jul 18 11:51:52.448010 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:51:52.447883 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\"" Jul 18 11:51:52.448010 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:51:52.447942 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\"" Jul 18 11:51:52.448226 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:51:52.448203 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\"" Jul 18 11:51:52.448296 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:51:52.448230 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\"" Jul 18 11:51:52.448369 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:51:52.448332 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-smh29\"" Jul 18 11:51:52.452964 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:51:52.452823 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-8i12ta5c71j38\"" Jul 18 11:51:52.454930 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:51:52.454912 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-7f6cccbddb-lztkh"] Jul 18 11:51:52.548911 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:51:52.548881 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/8d8d0932-1f5f-4fdb-aaae-7bd83340950f-secret-telemeter-client\") pod \"telemeter-client-7f6cccbddb-lztkh\" (UID: \"8d8d0932-1f5f-4fdb-aaae-7bd83340950f\") " pod="openshift-monitoring/telemeter-client-7f6cccbddb-lztkh" Jul 18 11:51:52.549046 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:51:52.548913 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8d8d0932-1f5f-4fdb-aaae-7bd83340950f-metrics-client-ca\") pod \"telemeter-client-7f6cccbddb-lztkh\" (UID: \"8d8d0932-1f5f-4fdb-aaae-7bd83340950f\") " pod="openshift-monitoring/telemeter-client-7f6cccbddb-lztkh" Jul 18 11:51:52.549046 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:51:52.548944 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/8d8d0932-1f5f-4fdb-aaae-7bd83340950f-federate-client-tls\") pod \"telemeter-client-7f6cccbddb-lztkh\" (UID: \"8d8d0932-1f5f-4fdb-aaae-7bd83340950f\") " pod="openshift-monitoring/telemeter-client-7f6cccbddb-lztkh" Jul 18 11:51:52.549046 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:51:52.548965 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/8d8d0932-1f5f-4fdb-aaae-7bd83340950f-telemeter-client-tls\") pod \"telemeter-client-7f6cccbddb-lztkh\" (UID: \"8d8d0932-1f5f-4fdb-aaae-7bd83340950f\") " pod="openshift-monitoring/telemeter-client-7f6cccbddb-lztkh" Jul 18 11:51:52.549171 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:51:52.549064 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/8d8d0932-1f5f-4fdb-aaae-7bd83340950f-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-7f6cccbddb-lztkh\" (UID: \"8d8d0932-1f5f-4fdb-aaae-7bd83340950f\") " pod="openshift-monitoring/telemeter-client-7f6cccbddb-lztkh" Jul 18 11:51:52.549171 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:51:52.549098 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8d8d0932-1f5f-4fdb-aaae-7bd83340950f-serving-certs-ca-bundle\") pod \"telemeter-client-7f6cccbddb-lztkh\" (UID: \"8d8d0932-1f5f-4fdb-aaae-7bd83340950f\") " pod="openshift-monitoring/telemeter-client-7f6cccbddb-lztkh" Jul 18 11:51:52.549171 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:51:52.549120 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f568z\" (UniqueName: \"kubernetes.io/projected/8d8d0932-1f5f-4fdb-aaae-7bd83340950f-kube-api-access-f568z\") pod \"telemeter-client-7f6cccbddb-lztkh\" (UID: \"8d8d0932-1f5f-4fdb-aaae-7bd83340950f\") " pod="openshift-monitoring/telemeter-client-7f6cccbddb-lztkh" Jul 18 11:51:52.549171 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:51:52.549167 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8d8d0932-1f5f-4fdb-aaae-7bd83340950f-telemeter-trusted-ca-bundle\") pod \"telemeter-client-7f6cccbddb-lztkh\" (UID: \"8d8d0932-1f5f-4fdb-aaae-7bd83340950f\") " pod="openshift-monitoring/telemeter-client-7f6cccbddb-lztkh" Jul 18 11:51:52.650289 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:51:52.650258 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8d8d0932-1f5f-4fdb-aaae-7bd83340950f-telemeter-trusted-ca-bundle\") pod \"telemeter-client-7f6cccbddb-lztkh\" (UID: \"8d8d0932-1f5f-4fdb-aaae-7bd83340950f\") " pod="openshift-monitoring/telemeter-client-7f6cccbddb-lztkh" Jul 18 11:51:52.650289 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:51:52.650291 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/8d8d0932-1f5f-4fdb-aaae-7bd83340950f-secret-telemeter-client\") pod \"telemeter-client-7f6cccbddb-lztkh\" (UID: \"8d8d0932-1f5f-4fdb-aaae-7bd83340950f\") " pod="openshift-monitoring/telemeter-client-7f6cccbddb-lztkh" Jul 18 11:51:52.650481 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:51:52.650307 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8d8d0932-1f5f-4fdb-aaae-7bd83340950f-metrics-client-ca\") pod \"telemeter-client-7f6cccbddb-lztkh\" (UID: \"8d8d0932-1f5f-4fdb-aaae-7bd83340950f\") " pod="openshift-monitoring/telemeter-client-7f6cccbddb-lztkh" Jul 18 11:51:52.650481 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:51:52.650335 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/8d8d0932-1f5f-4fdb-aaae-7bd83340950f-federate-client-tls\") pod \"telemeter-client-7f6cccbddb-lztkh\" (UID: \"8d8d0932-1f5f-4fdb-aaae-7bd83340950f\") " pod="openshift-monitoring/telemeter-client-7f6cccbddb-lztkh" Jul 18 11:51:52.650481 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:51:52.650363 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/8d8d0932-1f5f-4fdb-aaae-7bd83340950f-telemeter-client-tls\") pod \"telemeter-client-7f6cccbddb-lztkh\" (UID: \"8d8d0932-1f5f-4fdb-aaae-7bd83340950f\") " pod="openshift-monitoring/telemeter-client-7f6cccbddb-lztkh" Jul 18 11:51:52.650481 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:51:52.650397 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/8d8d0932-1f5f-4fdb-aaae-7bd83340950f-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-7f6cccbddb-lztkh\" (UID: \"8d8d0932-1f5f-4fdb-aaae-7bd83340950f\") " pod="openshift-monitoring/telemeter-client-7f6cccbddb-lztkh" Jul 18 11:51:52.650481 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:51:52.650427 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8d8d0932-1f5f-4fdb-aaae-7bd83340950f-serving-certs-ca-bundle\") pod \"telemeter-client-7f6cccbddb-lztkh\" (UID: \"8d8d0932-1f5f-4fdb-aaae-7bd83340950f\") " pod="openshift-monitoring/telemeter-client-7f6cccbddb-lztkh" Jul 18 11:51:52.650481 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:51:52.650458 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f568z\" (UniqueName: \"kubernetes.io/projected/8d8d0932-1f5f-4fdb-aaae-7bd83340950f-kube-api-access-f568z\") pod \"telemeter-client-7f6cccbddb-lztkh\" (UID: \"8d8d0932-1f5f-4fdb-aaae-7bd83340950f\") " pod="openshift-monitoring/telemeter-client-7f6cccbddb-lztkh" Jul 18 11:51:52.651112 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:51:52.651089 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8d8d0932-1f5f-4fdb-aaae-7bd83340950f-metrics-client-ca\") pod \"telemeter-client-7f6cccbddb-lztkh\" (UID: \"8d8d0932-1f5f-4fdb-aaae-7bd83340950f\") " pod="openshift-monitoring/telemeter-client-7f6cccbddb-lztkh" Jul 18 11:51:52.651455 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:51:52.651421 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8d8d0932-1f5f-4fdb-aaae-7bd83340950f-telemeter-trusted-ca-bundle\") pod \"telemeter-client-7f6cccbddb-lztkh\" (UID: \"8d8d0932-1f5f-4fdb-aaae-7bd83340950f\") " pod="openshift-monitoring/telemeter-client-7f6cccbddb-lztkh" Jul 18 11:51:52.651575 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:51:52.651492 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8d8d0932-1f5f-4fdb-aaae-7bd83340950f-serving-certs-ca-bundle\") pod \"telemeter-client-7f6cccbddb-lztkh\" (UID: \"8d8d0932-1f5f-4fdb-aaae-7bd83340950f\") " pod="openshift-monitoring/telemeter-client-7f6cccbddb-lztkh" Jul 18 11:51:52.653133 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:51:52.653098 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/8d8d0932-1f5f-4fdb-aaae-7bd83340950f-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-7f6cccbddb-lztkh\" (UID: \"8d8d0932-1f5f-4fdb-aaae-7bd83340950f\") " pod="openshift-monitoring/telemeter-client-7f6cccbddb-lztkh" Jul 18 11:51:52.653234 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:51:52.653142 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/8d8d0932-1f5f-4fdb-aaae-7bd83340950f-federate-client-tls\") pod \"telemeter-client-7f6cccbddb-lztkh\" (UID: \"8d8d0932-1f5f-4fdb-aaae-7bd83340950f\") " pod="openshift-monitoring/telemeter-client-7f6cccbddb-lztkh" Jul 18 11:51:52.653288 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:51:52.653227 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/8d8d0932-1f5f-4fdb-aaae-7bd83340950f-secret-telemeter-client\") pod \"telemeter-client-7f6cccbddb-lztkh\" (UID: \"8d8d0932-1f5f-4fdb-aaae-7bd83340950f\") " pod="openshift-monitoring/telemeter-client-7f6cccbddb-lztkh" Jul 18 11:51:52.653288 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:51:52.653264 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/8d8d0932-1f5f-4fdb-aaae-7bd83340950f-telemeter-client-tls\") pod \"telemeter-client-7f6cccbddb-lztkh\" (UID: \"8d8d0932-1f5f-4fdb-aaae-7bd83340950f\") " pod="openshift-monitoring/telemeter-client-7f6cccbddb-lztkh" Jul 18 11:51:52.658157 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:51:52.658135 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f568z\" (UniqueName: \"kubernetes.io/projected/8d8d0932-1f5f-4fdb-aaae-7bd83340950f-kube-api-access-f568z\") pod \"telemeter-client-7f6cccbddb-lztkh\" (UID: \"8d8d0932-1f5f-4fdb-aaae-7bd83340950f\") " pod="openshift-monitoring/telemeter-client-7f6cccbddb-lztkh" Jul 18 11:51:52.756540 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:51:52.756519 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-7f6cccbddb-lztkh" Jul 18 11:51:52.885876 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:51:52.885853 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-7f6cccbddb-lztkh"] Jul 18 11:51:52.888247 ip-10-0-136-240 kubenswrapper[2585]: W0718 11:51:52.888204 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8d8d0932_1f5f_4fdb_aaae_7bd83340950f.slice/crio-02ed7c9afcac03d217d5d3b41fdecd7a3d1c2a4d89884ab5144bcce0154599bc WatchSource:0}: Error finding container 02ed7c9afcac03d217d5d3b41fdecd7a3d1c2a4d89884ab5144bcce0154599bc: Status 404 returned error can't find the container with id 02ed7c9afcac03d217d5d3b41fdecd7a3d1c2a4d89884ab5144bcce0154599bc Jul 18 11:51:53.424395 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:51:53.424364 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-7f6cccbddb-lztkh" event={"ID":"8d8d0932-1f5f-4fdb-aaae-7bd83340950f","Type":"ContainerStarted","Data":"02ed7c9afcac03d217d5d3b41fdecd7a3d1c2a4d89884ab5144bcce0154599bc"} Jul 18 11:51:55.434134 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:51:55.434096 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-7f6cccbddb-lztkh" event={"ID":"8d8d0932-1f5f-4fdb-aaae-7bd83340950f","Type":"ContainerStarted","Data":"82ff03fa524bf662a578edc96f3f54e1fc192a3210f9a8d099e8cd4562c0dfd0"} Jul 18 11:51:55.434502 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:51:55.434161 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-7f6cccbddb-lztkh" event={"ID":"8d8d0932-1f5f-4fdb-aaae-7bd83340950f","Type":"ContainerStarted","Data":"e6c7ae3e826242d8042878ce58dad26cef093b6882fe8e61cbeb2813d99c04cd"} Jul 18 11:51:55.434502 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:51:55.434190 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-7f6cccbddb-lztkh" event={"ID":"8d8d0932-1f5f-4fdb-aaae-7bd83340950f","Type":"ContainerStarted","Data":"47336d57138b64867bfa05c83a88619ee027aeae4f807aa6b682f83c75309742"} Jul 18 11:51:55.457808 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:51:55.457758 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-7f6cccbddb-lztkh" podStartSLOduration=1.49896499 podStartE2EDuration="3.457739514s" totalImagesPullingTime="1.958774524s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-18 11:51:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-18 11:51:55.455625416 +0000 UTC m=+182.535831784" watchObservedRunningTime="2026-07-18 11:51:55.457739514 +0000 UTC m=+182.537945873" Jul 18 11:53:21.128833 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:53:21.128755 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-56pt5"] Jul 18 11:53:21.131987 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:53:21.131971 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-56pt5" Jul 18 11:53:21.134436 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:53:21.134412 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Jul 18 11:53:21.134568 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:53:21.134436 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Jul 18 11:53:21.135390 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:53:21.135375 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-cainjector-dockercfg-gkgms\"" Jul 18 11:53:21.138244 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:53:21.138226 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-56pt5"] Jul 18 11:53:21.227427 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:53:21.227399 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x82zx\" (UniqueName: \"kubernetes.io/projected/6fff81dd-550f-4be0-a1e7-bb7efe919e29-kube-api-access-x82zx\") pod \"cert-manager-cainjector-559cdd8c87-56pt5\" (UID: \"6fff81dd-550f-4be0-a1e7-bb7efe919e29\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-56pt5" Jul 18 11:53:21.227539 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:53:21.227439 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/6fff81dd-550f-4be0-a1e7-bb7efe919e29-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-56pt5\" (UID: \"6fff81dd-550f-4be0-a1e7-bb7efe919e29\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-56pt5" Jul 18 11:53:21.327888 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:53:21.327862 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x82zx\" (UniqueName: \"kubernetes.io/projected/6fff81dd-550f-4be0-a1e7-bb7efe919e29-kube-api-access-x82zx\") pod \"cert-manager-cainjector-559cdd8c87-56pt5\" (UID: \"6fff81dd-550f-4be0-a1e7-bb7efe919e29\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-56pt5" Jul 18 11:53:21.327986 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:53:21.327901 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/6fff81dd-550f-4be0-a1e7-bb7efe919e29-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-56pt5\" (UID: \"6fff81dd-550f-4be0-a1e7-bb7efe919e29\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-56pt5" Jul 18 11:53:21.335673 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:53:21.335648 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/6fff81dd-550f-4be0-a1e7-bb7efe919e29-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-56pt5\" (UID: \"6fff81dd-550f-4be0-a1e7-bb7efe919e29\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-56pt5" Jul 18 11:53:21.335784 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:53:21.335769 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x82zx\" (UniqueName: \"kubernetes.io/projected/6fff81dd-550f-4be0-a1e7-bb7efe919e29-kube-api-access-x82zx\") pod \"cert-manager-cainjector-559cdd8c87-56pt5\" (UID: \"6fff81dd-550f-4be0-a1e7-bb7efe919e29\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-56pt5" Jul 18 11:53:21.452545 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:53:21.452486 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-56pt5" Jul 18 11:53:21.578480 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:53:21.578456 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-56pt5"] Jul 18 11:53:21.580834 ip-10-0-136-240 kubenswrapper[2585]: W0718 11:53:21.580804 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6fff81dd_550f_4be0_a1e7_bb7efe919e29.slice/crio-80f758e70936216041e7e6b7e432211f04dd33bab1248d53ff62b678cd395401 WatchSource:0}: Error finding container 80f758e70936216041e7e6b7e432211f04dd33bab1248d53ff62b678cd395401: Status 404 returned error can't find the container with id 80f758e70936216041e7e6b7e432211f04dd33bab1248d53ff62b678cd395401 Jul 18 11:53:21.757188 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:53:21.757158 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-56pt5" event={"ID":"6fff81dd-550f-4be0-a1e7-bb7efe919e29","Type":"ContainerStarted","Data":"80f758e70936216041e7e6b7e432211f04dd33bab1248d53ff62b678cd395401"} Jul 18 11:53:24.777371 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:53:24.777335 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-56pt5" event={"ID":"6fff81dd-550f-4be0-a1e7-bb7efe919e29","Type":"ContainerStarted","Data":"bc918a8fd03c83520c5621d39e3c52f5a083663a5a2015567fec9db72d28b41a"} Jul 18 11:53:24.794834 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:53:24.794768 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-cainjector-559cdd8c87-56pt5" podStartSLOduration=0.995203664 podStartE2EDuration="3.794752156s" totalImagesPullingTime="2.799548492s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-18 11:53:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-18 11:53:24.793608123 +0000 UTC m=+271.873814481" watchObservedRunningTime="2026-07-18 11:53:24.794752156 +0000 UTC m=+271.874958512" Jul 18 11:53:30.238016 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:53:30.237967 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-c7gc6"] Jul 18 11:53:30.241190 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:53:30.241174 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-c7gc6" Jul 18 11:53:30.247582 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:53:30.244031 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\"" Jul 18 11:53:30.247582 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:53:30.244735 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\"" Jul 18 11:53:30.247582 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:53:30.245057 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"openshift-lws-operator-dockercfg-lhj92\"" Jul 18 11:53:30.252755 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:53:30.252728 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-c7gc6"] Jul 18 11:53:30.301370 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:53:30.301338 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/cb99d1b5-ebd0-4feb-8aca-0c6b827c0c4a-tmp\") pod \"openshift-lws-operator-fd8ccff4c-c7gc6\" (UID: \"cb99d1b5-ebd0-4feb-8aca-0c6b827c0c4a\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-c7gc6" Jul 18 11:53:30.301524 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:53:30.301380 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l4kkm\" (UniqueName: \"kubernetes.io/projected/cb99d1b5-ebd0-4feb-8aca-0c6b827c0c4a-kube-api-access-l4kkm\") pod \"openshift-lws-operator-fd8ccff4c-c7gc6\" (UID: \"cb99d1b5-ebd0-4feb-8aca-0c6b827c0c4a\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-c7gc6" Jul 18 11:53:30.402925 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:53:30.402886 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/cb99d1b5-ebd0-4feb-8aca-0c6b827c0c4a-tmp\") pod \"openshift-lws-operator-fd8ccff4c-c7gc6\" (UID: \"cb99d1b5-ebd0-4feb-8aca-0c6b827c0c4a\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-c7gc6" Jul 18 11:53:30.403094 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:53:30.402934 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l4kkm\" (UniqueName: \"kubernetes.io/projected/cb99d1b5-ebd0-4feb-8aca-0c6b827c0c4a-kube-api-access-l4kkm\") pod \"openshift-lws-operator-fd8ccff4c-c7gc6\" (UID: \"cb99d1b5-ebd0-4feb-8aca-0c6b827c0c4a\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-c7gc6" Jul 18 11:53:30.403295 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:53:30.403275 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/cb99d1b5-ebd0-4feb-8aca-0c6b827c0c4a-tmp\") pod \"openshift-lws-operator-fd8ccff4c-c7gc6\" (UID: \"cb99d1b5-ebd0-4feb-8aca-0c6b827c0c4a\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-c7gc6" Jul 18 11:53:30.411381 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:53:30.411350 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l4kkm\" (UniqueName: \"kubernetes.io/projected/cb99d1b5-ebd0-4feb-8aca-0c6b827c0c4a-kube-api-access-l4kkm\") pod \"openshift-lws-operator-fd8ccff4c-c7gc6\" (UID: \"cb99d1b5-ebd0-4feb-8aca-0c6b827c0c4a\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-c7gc6" Jul 18 11:53:30.555387 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:53:30.555317 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-c7gc6" Jul 18 11:53:30.683414 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:53:30.683381 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-c7gc6"] Jul 18 11:53:30.686229 ip-10-0-136-240 kubenswrapper[2585]: W0718 11:53:30.686197 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podcb99d1b5_ebd0_4feb_8aca_0c6b827c0c4a.slice/crio-a850f5e7951fc0366b4667ae16eddbc0f3dd490672bc45428b7bd9aff507cbd3 WatchSource:0}: Error finding container a850f5e7951fc0366b4667ae16eddbc0f3dd490672bc45428b7bd9aff507cbd3: Status 404 returned error can't find the container with id a850f5e7951fc0366b4667ae16eddbc0f3dd490672bc45428b7bd9aff507cbd3 Jul 18 11:53:30.802946 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:53:30.802919 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-c7gc6" event={"ID":"cb99d1b5-ebd0-4feb-8aca-0c6b827c0c4a","Type":"ContainerStarted","Data":"a850f5e7951fc0366b4667ae16eddbc0f3dd490672bc45428b7bd9aff507cbd3"} Jul 18 11:53:33.818845 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:53:33.818808 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-c7gc6" event={"ID":"cb99d1b5-ebd0-4feb-8aca-0c6b827c0c4a","Type":"ContainerStarted","Data":"23fef4d10b22e89b764461162717286687171eee00e7585c38b360039248d2aa"} Jul 18 11:53:33.835845 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:53:33.835798 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-c7gc6" podStartSLOduration=1.467092707 podStartE2EDuration="3.835783094s" totalImagesPullingTime="2.368690387s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-18 11:53:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-18 11:53:33.833945606 +0000 UTC m=+280.914151967" watchObservedRunningTime="2026-07-18 11:53:33.835783094 +0000 UTC m=+280.915989450" Jul 18 11:53:46.694674 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:53:46.694638 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-77c8bd676b-lsmbc"] Jul 18 11:53:46.697799 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:53:46.697783 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-77c8bd676b-lsmbc" Jul 18 11:53:46.701489 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:53:46.701457 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\"" Jul 18 11:53:46.701629 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:53:46.701462 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-tgpq8\"" Jul 18 11:53:46.701629 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:53:46.701503 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\"" Jul 18 11:53:46.701629 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:53:46.701462 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\"" Jul 18 11:53:46.709349 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:53:46.709325 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-77c8bd676b-lsmbc"] Jul 18 11:53:46.834762 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:53:46.834729 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/98206ec9-6aca-4f15-8186-3b6b2c4aec08-manager-config\") pod \"lws-controller-manager-77c8bd676b-lsmbc\" (UID: \"98206ec9-6aca-4f15-8186-3b6b2c4aec08\") " pod="openshift-lws-operator/lws-controller-manager-77c8bd676b-lsmbc" Jul 18 11:53:46.834924 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:53:46.834771 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/98206ec9-6aca-4f15-8186-3b6b2c4aec08-cert\") pod \"lws-controller-manager-77c8bd676b-lsmbc\" (UID: \"98206ec9-6aca-4f15-8186-3b6b2c4aec08\") " pod="openshift-lws-operator/lws-controller-manager-77c8bd676b-lsmbc" Jul 18 11:53:46.834924 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:53:46.834796 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-679wk\" (UniqueName: \"kubernetes.io/projected/98206ec9-6aca-4f15-8186-3b6b2c4aec08-kube-api-access-679wk\") pod \"lws-controller-manager-77c8bd676b-lsmbc\" (UID: \"98206ec9-6aca-4f15-8186-3b6b2c4aec08\") " pod="openshift-lws-operator/lws-controller-manager-77c8bd676b-lsmbc" Jul 18 11:53:46.834924 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:53:46.834866 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/98206ec9-6aca-4f15-8186-3b6b2c4aec08-metrics-cert\") pod \"lws-controller-manager-77c8bd676b-lsmbc\" (UID: \"98206ec9-6aca-4f15-8186-3b6b2c4aec08\") " pod="openshift-lws-operator/lws-controller-manager-77c8bd676b-lsmbc" Jul 18 11:53:46.935911 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:53:46.935880 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/98206ec9-6aca-4f15-8186-3b6b2c4aec08-metrics-cert\") pod \"lws-controller-manager-77c8bd676b-lsmbc\" (UID: \"98206ec9-6aca-4f15-8186-3b6b2c4aec08\") " pod="openshift-lws-operator/lws-controller-manager-77c8bd676b-lsmbc" Jul 18 11:53:46.936045 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:53:46.935939 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/98206ec9-6aca-4f15-8186-3b6b2c4aec08-manager-config\") pod \"lws-controller-manager-77c8bd676b-lsmbc\" (UID: \"98206ec9-6aca-4f15-8186-3b6b2c4aec08\") " pod="openshift-lws-operator/lws-controller-manager-77c8bd676b-lsmbc" Jul 18 11:53:46.936045 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:53:46.935961 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/98206ec9-6aca-4f15-8186-3b6b2c4aec08-cert\") pod \"lws-controller-manager-77c8bd676b-lsmbc\" (UID: \"98206ec9-6aca-4f15-8186-3b6b2c4aec08\") " pod="openshift-lws-operator/lws-controller-manager-77c8bd676b-lsmbc" Jul 18 11:53:46.936045 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:53:46.935982 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-679wk\" (UniqueName: \"kubernetes.io/projected/98206ec9-6aca-4f15-8186-3b6b2c4aec08-kube-api-access-679wk\") pod \"lws-controller-manager-77c8bd676b-lsmbc\" (UID: \"98206ec9-6aca-4f15-8186-3b6b2c4aec08\") " pod="openshift-lws-operator/lws-controller-manager-77c8bd676b-lsmbc" Jul 18 11:53:46.936754 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:53:46.936690 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/98206ec9-6aca-4f15-8186-3b6b2c4aec08-manager-config\") pod \"lws-controller-manager-77c8bd676b-lsmbc\" (UID: \"98206ec9-6aca-4f15-8186-3b6b2c4aec08\") " pod="openshift-lws-operator/lws-controller-manager-77c8bd676b-lsmbc" Jul 18 11:53:46.938582 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:53:46.938544 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/98206ec9-6aca-4f15-8186-3b6b2c4aec08-metrics-cert\") pod \"lws-controller-manager-77c8bd676b-lsmbc\" (UID: \"98206ec9-6aca-4f15-8186-3b6b2c4aec08\") " pod="openshift-lws-operator/lws-controller-manager-77c8bd676b-lsmbc" Jul 18 11:53:46.938659 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:53:46.938639 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/98206ec9-6aca-4f15-8186-3b6b2c4aec08-cert\") pod \"lws-controller-manager-77c8bd676b-lsmbc\" (UID: \"98206ec9-6aca-4f15-8186-3b6b2c4aec08\") " pod="openshift-lws-operator/lws-controller-manager-77c8bd676b-lsmbc" Jul 18 11:53:46.944488 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:53:46.944470 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-679wk\" (UniqueName: \"kubernetes.io/projected/98206ec9-6aca-4f15-8186-3b6b2c4aec08-kube-api-access-679wk\") pod \"lws-controller-manager-77c8bd676b-lsmbc\" (UID: \"98206ec9-6aca-4f15-8186-3b6b2c4aec08\") " pod="openshift-lws-operator/lws-controller-manager-77c8bd676b-lsmbc" Jul 18 11:53:47.007411 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:53:47.007386 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-77c8bd676b-lsmbc" Jul 18 11:53:47.137253 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:53:47.137224 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-77c8bd676b-lsmbc"] Jul 18 11:53:47.138652 ip-10-0-136-240 kubenswrapper[2585]: W0718 11:53:47.138622 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod98206ec9_6aca_4f15_8186_3b6b2c4aec08.slice/crio-392b8e793de02b3f0d2ac15ec695e7a10bda859261c76a8fb27fab59c019560e WatchSource:0}: Error finding container 392b8e793de02b3f0d2ac15ec695e7a10bda859261c76a8fb27fab59c019560e: Status 404 returned error can't find the container with id 392b8e793de02b3f0d2ac15ec695e7a10bda859261c76a8fb27fab59c019560e Jul 18 11:53:47.875118 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:53:47.875084 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-77c8bd676b-lsmbc" event={"ID":"98206ec9-6aca-4f15-8186-3b6b2c4aec08","Type":"ContainerStarted","Data":"392b8e793de02b3f0d2ac15ec695e7a10bda859261c76a8fb27fab59c019560e"} Jul 18 11:53:52.932874 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:53:52.932844 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-fbb59dc99-9rkcw"] Jul 18 11:53:52.937395 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:53:52.937380 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-fbb59dc99-9rkcw" Jul 18 11:53:52.939718 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:53:52.939695 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Jul 18 11:53:52.939914 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:53:52.939899 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Jul 18 11:53:52.940040 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:53:52.940023 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Jul 18 11:53:52.940239 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:53:52.940217 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-m7xcv\"" Jul 18 11:53:52.940573 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:53:52.940536 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Jul 18 11:53:52.961210 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:53:52.961186 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-fbb59dc99-9rkcw"] Jul 18 11:53:53.083786 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:53:53.083757 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fqfvk\" (UniqueName: \"kubernetes.io/projected/d46e8b83-a5c7-44b0-aa32-7c68ac538c32-kube-api-access-fqfvk\") pod \"opendatahub-operator-controller-manager-fbb59dc99-9rkcw\" (UID: \"d46e8b83-a5c7-44b0-aa32-7c68ac538c32\") " pod="opendatahub/opendatahub-operator-controller-manager-fbb59dc99-9rkcw" Jul 18 11:53:53.083941 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:53:53.083794 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/d46e8b83-a5c7-44b0-aa32-7c68ac538c32-webhook-cert\") pod \"opendatahub-operator-controller-manager-fbb59dc99-9rkcw\" (UID: \"d46e8b83-a5c7-44b0-aa32-7c68ac538c32\") " pod="opendatahub/opendatahub-operator-controller-manager-fbb59dc99-9rkcw" Jul 18 11:53:53.083941 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:53:53.083828 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/d46e8b83-a5c7-44b0-aa32-7c68ac538c32-manifests\") pod \"opendatahub-operator-controller-manager-fbb59dc99-9rkcw\" (UID: \"d46e8b83-a5c7-44b0-aa32-7c68ac538c32\") " pod="opendatahub/opendatahub-operator-controller-manager-fbb59dc99-9rkcw" Jul 18 11:53:53.083941 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:53:53.083896 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/d46e8b83-a5c7-44b0-aa32-7c68ac538c32-apiservice-cert\") pod \"opendatahub-operator-controller-manager-fbb59dc99-9rkcw\" (UID: \"d46e8b83-a5c7-44b0-aa32-7c68ac538c32\") " pod="opendatahub/opendatahub-operator-controller-manager-fbb59dc99-9rkcw" Jul 18 11:53:53.084056 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:53:53.083958 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d46e8b83-a5c7-44b0-aa32-7c68ac538c32-tmp\") pod \"opendatahub-operator-controller-manager-fbb59dc99-9rkcw\" (UID: \"d46e8b83-a5c7-44b0-aa32-7c68ac538c32\") " pod="opendatahub/opendatahub-operator-controller-manager-fbb59dc99-9rkcw" Jul 18 11:53:53.184732 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:53:53.184654 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/d46e8b83-a5c7-44b0-aa32-7c68ac538c32-manifests\") pod \"opendatahub-operator-controller-manager-fbb59dc99-9rkcw\" (UID: \"d46e8b83-a5c7-44b0-aa32-7c68ac538c32\") " pod="opendatahub/opendatahub-operator-controller-manager-fbb59dc99-9rkcw" Jul 18 11:53:53.184732 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:53:53.184696 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/d46e8b83-a5c7-44b0-aa32-7c68ac538c32-apiservice-cert\") pod \"opendatahub-operator-controller-manager-fbb59dc99-9rkcw\" (UID: \"d46e8b83-a5c7-44b0-aa32-7c68ac538c32\") " pod="opendatahub/opendatahub-operator-controller-manager-fbb59dc99-9rkcw" Jul 18 11:53:53.184732 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:53:53.184726 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d46e8b83-a5c7-44b0-aa32-7c68ac538c32-tmp\") pod \"opendatahub-operator-controller-manager-fbb59dc99-9rkcw\" (UID: \"d46e8b83-a5c7-44b0-aa32-7c68ac538c32\") " pod="opendatahub/opendatahub-operator-controller-manager-fbb59dc99-9rkcw" Jul 18 11:53:53.184914 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:53:53.184768 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fqfvk\" (UniqueName: \"kubernetes.io/projected/d46e8b83-a5c7-44b0-aa32-7c68ac538c32-kube-api-access-fqfvk\") pod \"opendatahub-operator-controller-manager-fbb59dc99-9rkcw\" (UID: \"d46e8b83-a5c7-44b0-aa32-7c68ac538c32\") " pod="opendatahub/opendatahub-operator-controller-manager-fbb59dc99-9rkcw" Jul 18 11:53:53.184914 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:53:53.184789 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/d46e8b83-a5c7-44b0-aa32-7c68ac538c32-webhook-cert\") pod \"opendatahub-operator-controller-manager-fbb59dc99-9rkcw\" (UID: \"d46e8b83-a5c7-44b0-aa32-7c68ac538c32\") " pod="opendatahub/opendatahub-operator-controller-manager-fbb59dc99-9rkcw" Jul 18 11:53:53.185117 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:53:53.185092 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/d46e8b83-a5c7-44b0-aa32-7c68ac538c32-manifests\") pod \"opendatahub-operator-controller-manager-fbb59dc99-9rkcw\" (UID: \"d46e8b83-a5c7-44b0-aa32-7c68ac538c32\") " pod="opendatahub/opendatahub-operator-controller-manager-fbb59dc99-9rkcw" Jul 18 11:53:53.187320 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:53:53.187300 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/d46e8b83-a5c7-44b0-aa32-7c68ac538c32-apiservice-cert\") pod \"opendatahub-operator-controller-manager-fbb59dc99-9rkcw\" (UID: \"d46e8b83-a5c7-44b0-aa32-7c68ac538c32\") " pod="opendatahub/opendatahub-operator-controller-manager-fbb59dc99-9rkcw" Jul 18 11:53:53.187320 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:53:53.187303 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d46e8b83-a5c7-44b0-aa32-7c68ac538c32-tmp\") pod \"opendatahub-operator-controller-manager-fbb59dc99-9rkcw\" (UID: \"d46e8b83-a5c7-44b0-aa32-7c68ac538c32\") " pod="opendatahub/opendatahub-operator-controller-manager-fbb59dc99-9rkcw" Jul 18 11:53:53.187460 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:53:53.187366 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/d46e8b83-a5c7-44b0-aa32-7c68ac538c32-webhook-cert\") pod \"opendatahub-operator-controller-manager-fbb59dc99-9rkcw\" (UID: \"d46e8b83-a5c7-44b0-aa32-7c68ac538c32\") " pod="opendatahub/opendatahub-operator-controller-manager-fbb59dc99-9rkcw" Jul 18 11:53:53.193173 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:53:53.193154 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fqfvk\" (UniqueName: \"kubernetes.io/projected/d46e8b83-a5c7-44b0-aa32-7c68ac538c32-kube-api-access-fqfvk\") pod \"opendatahub-operator-controller-manager-fbb59dc99-9rkcw\" (UID: \"d46e8b83-a5c7-44b0-aa32-7c68ac538c32\") " pod="opendatahub/opendatahub-operator-controller-manager-fbb59dc99-9rkcw" Jul 18 11:53:53.248217 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:53:53.248199 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-fbb59dc99-9rkcw" Jul 18 11:53:53.341173 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:53:53.341150 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-j9l7p_8ec47eff-5fbc-4130-baef-d8544aa4e786/console-operator/1.log" Jul 18 11:53:53.349711 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:53:53.349685 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-j9l7p_8ec47eff-5fbc-4130-baef-d8544aa4e786/console-operator/1.log" Jul 18 11:53:53.357098 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:53:53.356418 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-wq4lg_247c97c8-982b-4176-ae10-afc396745360/ovn-acl-logging/0.log" Jul 18 11:53:53.362021 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:53:53.361999 2585 kubelet.go:1648] "Image garbage collection succeeded" Jul 18 11:53:53.363398 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:53:53.363383 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-wq4lg_247c97c8-982b-4176-ae10-afc396745360/ovn-acl-logging/0.log" Jul 18 11:53:53.418196 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:53:53.402656 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-fbb59dc99-9rkcw"] Jul 18 11:53:53.418196 ip-10-0-136-240 kubenswrapper[2585]: W0718 11:53:53.404911 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd46e8b83_a5c7_44b0_aa32_7c68ac538c32.slice/crio-c7d07b4c701fc5f6ca065adb8290009c434742704d2c5a7a847dfb97380d90bb WatchSource:0}: Error finding container c7d07b4c701fc5f6ca065adb8290009c434742704d2c5a7a847dfb97380d90bb: Status 404 returned error can't find the container with id c7d07b4c701fc5f6ca065adb8290009c434742704d2c5a7a847dfb97380d90bb Jul 18 11:53:53.902804 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:53:53.902757 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-fbb59dc99-9rkcw" event={"ID":"d46e8b83-a5c7-44b0-aa32-7c68ac538c32","Type":"ContainerStarted","Data":"c7d07b4c701fc5f6ca065adb8290009c434742704d2c5a7a847dfb97380d90bb"} Jul 18 11:53:56.918743 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:53:56.918709 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-fbb59dc99-9rkcw" event={"ID":"d46e8b83-a5c7-44b0-aa32-7c68ac538c32","Type":"ContainerStarted","Data":"4ccdfbb328e562fcfbe818b5b96396f632d6312ee79cb1e64291608910b7ccb9"} Jul 18 11:53:57.925673 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:53:57.925634 2585 generic.go:356] "Generic (PLEG): container finished" podID="d46e8b83-a5c7-44b0-aa32-7c68ac538c32" containerID="4ccdfbb328e562fcfbe818b5b96396f632d6312ee79cb1e64291608910b7ccb9" exitCode=0 Jul 18 11:53:57.926153 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:53:57.925705 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-fbb59dc99-9rkcw" event={"ID":"d46e8b83-a5c7-44b0-aa32-7c68ac538c32","Type":"ContainerDied","Data":"4ccdfbb328e562fcfbe818b5b96396f632d6312ee79cb1e64291608910b7ccb9"} Jul 18 11:53:57.926382 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:53:57.926362 2585 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 18 11:53:58.931974 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:53:58.931936 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-fbb59dc99-9rkcw" event={"ID":"d46e8b83-a5c7-44b0-aa32-7c68ac538c32","Type":"ContainerStarted","Data":"e25ad8c069694b81ae85b9d832ca305aad233ec6d776adf2f8d23682659dbb35"} Jul 18 11:53:58.932345 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:53:58.932077 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-fbb59dc99-9rkcw" Jul 18 11:53:58.958799 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:53:58.958733 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-fbb59dc99-9rkcw" podStartSLOduration=2.54351453 podStartE2EDuration="6.958710745s" totalImagesPullingTime="3.372118186s" totalInitContainerRuntime="1.043078029s" isStatefulPod=false podCreationTimestamp="2026-07-18 11:53:52 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-18 11:53:58.954398852 +0000 UTC m=+306.034605210" watchObservedRunningTime="2026-07-18 11:53:58.958710745 +0000 UTC m=+306.038917108" Jul 18 11:54:04.961138 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:54:04.961106 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-77c8bd676b-lsmbc" event={"ID":"98206ec9-6aca-4f15-8186-3b6b2c4aec08","Type":"ContainerStarted","Data":"66575c6985d85c963443412d7372f3dbca7b31de5fe2ede367ec4df3d4579af4"} Jul 18 11:54:04.961542 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:54:04.961211 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-77c8bd676b-lsmbc" Jul 18 11:54:04.979251 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:54:04.979207 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-77c8bd676b-lsmbc" podStartSLOduration=1.626279238 podStartE2EDuration="18.979191586s" totalImagesPullingTime="17.352912348s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-18 11:53:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-18 11:54:04.977446127 +0000 UTC m=+312.057652475" watchObservedRunningTime="2026-07-18 11:54:04.979191586 +0000 UTC m=+312.059397943" Jul 18 11:54:09.938154 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:54:09.938127 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-fbb59dc99-9rkcw" Jul 18 11:54:15.966628 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:54:15.966594 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-77c8bd676b-lsmbc" Jul 18 11:54:29.968722 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:54:29.968679 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf49clx6"] Jul 18 11:54:29.979277 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:54:29.979254 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf49clx6" Jul 18 11:54:29.982595 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:54:29.982573 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"data-science-gateway-data-science-gateway-class-dockercfg-hbps9\"" Jul 18 11:54:29.983031 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:54:29.983013 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Jul 18 11:54:29.989200 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:54:29.989176 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf49clx6"] Jul 18 11:54:30.084310 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:54:30.084284 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d999136c-b948-4ed1-a61a-ad0cdf3e339d-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-6576859cf49clx6\" (UID: \"d999136c-b948-4ed1-a61a-ad0cdf3e339d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf49clx6" Jul 18 11:54:30.084464 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:54:30.084323 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d999136c-b948-4ed1-a61a-ad0cdf3e339d-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-6576859cf49clx6\" (UID: \"d999136c-b948-4ed1-a61a-ad0cdf3e339d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf49clx6" Jul 18 11:54:30.084464 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:54:30.084343 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d999136c-b948-4ed1-a61a-ad0cdf3e339d-istio-token\") pod \"data-science-gateway-data-science-gateway-class-6576859cf49clx6\" (UID: \"d999136c-b948-4ed1-a61a-ad0cdf3e339d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf49clx6" Jul 18 11:54:30.084464 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:54:30.084381 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d999136c-b948-4ed1-a61a-ad0cdf3e339d-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-6576859cf49clx6\" (UID: \"d999136c-b948-4ed1-a61a-ad0cdf3e339d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf49clx6" Jul 18 11:54:30.084464 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:54:30.084400 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d999136c-b948-4ed1-a61a-ad0cdf3e339d-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-6576859cf49clx6\" (UID: \"d999136c-b948-4ed1-a61a-ad0cdf3e339d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf49clx6" Jul 18 11:54:30.084464 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:54:30.084427 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d999136c-b948-4ed1-a61a-ad0cdf3e339d-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-6576859cf49clx6\" (UID: \"d999136c-b948-4ed1-a61a-ad0cdf3e339d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf49clx6" Jul 18 11:54:30.084464 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:54:30.084452 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d999136c-b948-4ed1-a61a-ad0cdf3e339d-istio-data\") pod \"data-science-gateway-data-science-gateway-class-6576859cf49clx6\" (UID: \"d999136c-b948-4ed1-a61a-ad0cdf3e339d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf49clx6" Jul 18 11:54:30.084696 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:54:30.084515 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p6ntr\" (UniqueName: \"kubernetes.io/projected/d999136c-b948-4ed1-a61a-ad0cdf3e339d-kube-api-access-p6ntr\") pod \"data-science-gateway-data-science-gateway-class-6576859cf49clx6\" (UID: \"d999136c-b948-4ed1-a61a-ad0cdf3e339d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf49clx6" Jul 18 11:54:30.084696 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:54:30.084581 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d999136c-b948-4ed1-a61a-ad0cdf3e339d-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-6576859cf49clx6\" (UID: \"d999136c-b948-4ed1-a61a-ad0cdf3e339d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf49clx6" Jul 18 11:54:30.185196 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:54:30.185173 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d999136c-b948-4ed1-a61a-ad0cdf3e339d-istio-token\") pod \"data-science-gateway-data-science-gateway-class-6576859cf49clx6\" (UID: \"d999136c-b948-4ed1-a61a-ad0cdf3e339d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf49clx6" Jul 18 11:54:30.185336 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:54:30.185204 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d999136c-b948-4ed1-a61a-ad0cdf3e339d-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-6576859cf49clx6\" (UID: \"d999136c-b948-4ed1-a61a-ad0cdf3e339d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf49clx6" Jul 18 11:54:30.185336 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:54:30.185223 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d999136c-b948-4ed1-a61a-ad0cdf3e339d-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-6576859cf49clx6\" (UID: \"d999136c-b948-4ed1-a61a-ad0cdf3e339d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf49clx6" Jul 18 11:54:30.185336 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:54:30.185247 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d999136c-b948-4ed1-a61a-ad0cdf3e339d-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-6576859cf49clx6\" (UID: \"d999136c-b948-4ed1-a61a-ad0cdf3e339d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf49clx6" Jul 18 11:54:30.185483 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:54:30.185345 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d999136c-b948-4ed1-a61a-ad0cdf3e339d-istio-data\") pod \"data-science-gateway-data-science-gateway-class-6576859cf49clx6\" (UID: \"d999136c-b948-4ed1-a61a-ad0cdf3e339d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf49clx6" Jul 18 11:54:30.185483 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:54:30.185418 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p6ntr\" (UniqueName: \"kubernetes.io/projected/d999136c-b948-4ed1-a61a-ad0cdf3e339d-kube-api-access-p6ntr\") pod \"data-science-gateway-data-science-gateway-class-6576859cf49clx6\" (UID: \"d999136c-b948-4ed1-a61a-ad0cdf3e339d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf49clx6" Jul 18 11:54:30.185483 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:54:30.185467 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d999136c-b948-4ed1-a61a-ad0cdf3e339d-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-6576859cf49clx6\" (UID: \"d999136c-b948-4ed1-a61a-ad0cdf3e339d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf49clx6" Jul 18 11:54:30.185677 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:54:30.185514 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d999136c-b948-4ed1-a61a-ad0cdf3e339d-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-6576859cf49clx6\" (UID: \"d999136c-b948-4ed1-a61a-ad0cdf3e339d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf49clx6" Jul 18 11:54:30.185677 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:54:30.185592 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d999136c-b948-4ed1-a61a-ad0cdf3e339d-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-6576859cf49clx6\" (UID: \"d999136c-b948-4ed1-a61a-ad0cdf3e339d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf49clx6" Jul 18 11:54:30.185677 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:54:30.185636 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d999136c-b948-4ed1-a61a-ad0cdf3e339d-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-6576859cf49clx6\" (UID: \"d999136c-b948-4ed1-a61a-ad0cdf3e339d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf49clx6" Jul 18 11:54:30.185838 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:54:30.185773 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d999136c-b948-4ed1-a61a-ad0cdf3e339d-istio-data\") pod \"data-science-gateway-data-science-gateway-class-6576859cf49clx6\" (UID: \"d999136c-b948-4ed1-a61a-ad0cdf3e339d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf49clx6" Jul 18 11:54:30.185915 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:54:30.185893 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d999136c-b948-4ed1-a61a-ad0cdf3e339d-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-6576859cf49clx6\" (UID: \"d999136c-b948-4ed1-a61a-ad0cdf3e339d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf49clx6" Jul 18 11:54:30.185956 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:54:30.185917 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d999136c-b948-4ed1-a61a-ad0cdf3e339d-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-6576859cf49clx6\" (UID: \"d999136c-b948-4ed1-a61a-ad0cdf3e339d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf49clx6" Jul 18 11:54:30.186587 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:54:30.186536 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d999136c-b948-4ed1-a61a-ad0cdf3e339d-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-6576859cf49clx6\" (UID: \"d999136c-b948-4ed1-a61a-ad0cdf3e339d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf49clx6" Jul 18 11:54:30.187816 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:54:30.187793 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d999136c-b948-4ed1-a61a-ad0cdf3e339d-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-6576859cf49clx6\" (UID: \"d999136c-b948-4ed1-a61a-ad0cdf3e339d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf49clx6" Jul 18 11:54:30.188038 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:54:30.188018 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d999136c-b948-4ed1-a61a-ad0cdf3e339d-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-6576859cf49clx6\" (UID: \"d999136c-b948-4ed1-a61a-ad0cdf3e339d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf49clx6" Jul 18 11:54:30.195423 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:54:30.195405 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d999136c-b948-4ed1-a61a-ad0cdf3e339d-istio-token\") pod \"data-science-gateway-data-science-gateway-class-6576859cf49clx6\" (UID: \"d999136c-b948-4ed1-a61a-ad0cdf3e339d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf49clx6" Jul 18 11:54:30.195534 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:54:30.195513 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p6ntr\" (UniqueName: \"kubernetes.io/projected/d999136c-b948-4ed1-a61a-ad0cdf3e339d-kube-api-access-p6ntr\") pod \"data-science-gateway-data-science-gateway-class-6576859cf49clx6\" (UID: \"d999136c-b948-4ed1-a61a-ad0cdf3e339d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf49clx6" Jul 18 11:54:30.290768 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:54:30.290690 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf49clx6" Jul 18 11:54:30.427171 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:54:30.427143 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf49clx6"] Jul 18 11:54:30.428800 ip-10-0-136-240 kubenswrapper[2585]: W0718 11:54:30.428770 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd999136c_b948_4ed1_a61a_ad0cdf3e339d.slice/crio-0ef3498f647aa68d785f909962bf22e637dc80a9f53cbcac9dc8baaf3b439996 WatchSource:0}: Error finding container 0ef3498f647aa68d785f909962bf22e637dc80a9f53cbcac9dc8baaf3b439996: Status 404 returned error can't find the container with id 0ef3498f647aa68d785f909962bf22e637dc80a9f53cbcac9dc8baaf3b439996 Jul 18 11:54:31.076453 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:54:31.076420 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf49clx6" event={"ID":"d999136c-b948-4ed1-a61a-ad0cdf3e339d","Type":"ContainerStarted","Data":"0ef3498f647aa68d785f909962bf22e637dc80a9f53cbcac9dc8baaf3b439996"} Jul 18 11:54:33.032040 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:54:33.032007 2585 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 18 11:54:33.032282 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:54:33.032076 2585 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 18 11:54:33.032282 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:54:33.032109 2585 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 18 11:54:34.095585 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:54:34.095531 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf49clx6" event={"ID":"d999136c-b948-4ed1-a61a-ad0cdf3e339d","Type":"ContainerStarted","Data":"9e2d901b7f97aa75ad69b182994adffea9bc4cc0301b7c3a1aa751eec6b6dacc"} Jul 18 11:54:34.119341 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:54:34.119295 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf49clx6" podStartSLOduration=2.518583457 podStartE2EDuration="5.119279099s" totalImagesPullingTime="2.600695642s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-18 11:54:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-18 11:54:34.117967191 +0000 UTC m=+341.198173547" watchObservedRunningTime="2026-07-18 11:54:34.119279099 +0000 UTC m=+341.199485455" Jul 18 11:54:34.291142 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:54:34.291108 2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf49clx6" Jul 18 11:54:34.295506 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:54:34.295482 2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf49clx6" Jul 18 11:54:35.099701 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:54:35.099673 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf49clx6" Jul 18 11:54:35.100667 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:54:35.100647 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf49clx6" Jul 18 11:55:13.273193 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:55:13.273120 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-sgtl8"] Jul 18 11:55:13.282714 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:55:13.282691 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-sgtl8" Jul 18 11:55:13.285367 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:55:13.285341 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Jul 18 11:55:13.286984 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:55:13.286766 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Jul 18 11:55:13.286984 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:55:13.286826 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-catalog-dockercfg-qclpd\"" Jul 18 11:55:13.287720 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:55:13.287693 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-sgtl8"] Jul 18 11:55:13.447396 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:55:13.447363 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7ksq9\" (UniqueName: \"kubernetes.io/projected/b2cb7f1d-4bb4-4cd2-abd4-e0bf8bf750fe-kube-api-access-7ksq9\") pod \"kuadrant-operator-catalog-sgtl8\" (UID: \"b2cb7f1d-4bb4-4cd2-abd4-e0bf8bf750fe\") " pod="kuadrant-system/kuadrant-operator-catalog-sgtl8" Jul 18 11:55:13.548621 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:55:13.548515 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7ksq9\" (UniqueName: \"kubernetes.io/projected/b2cb7f1d-4bb4-4cd2-abd4-e0bf8bf750fe-kube-api-access-7ksq9\") pod \"kuadrant-operator-catalog-sgtl8\" (UID: \"b2cb7f1d-4bb4-4cd2-abd4-e0bf8bf750fe\") " pod="kuadrant-system/kuadrant-operator-catalog-sgtl8" Jul 18 11:55:13.556829 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:55:13.556797 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7ksq9\" (UniqueName: \"kubernetes.io/projected/b2cb7f1d-4bb4-4cd2-abd4-e0bf8bf750fe-kube-api-access-7ksq9\") pod \"kuadrant-operator-catalog-sgtl8\" (UID: \"b2cb7f1d-4bb4-4cd2-abd4-e0bf8bf750fe\") " pod="kuadrant-system/kuadrant-operator-catalog-sgtl8" Jul 18 11:55:13.595267 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:55:13.595247 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-sgtl8" Jul 18 11:55:13.630900 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:55:13.630875 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-sgtl8"] Jul 18 11:55:13.726612 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:55:13.726490 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-sgtl8"] Jul 18 11:55:13.730110 ip-10-0-136-240 kubenswrapper[2585]: W0718 11:55:13.730066 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb2cb7f1d_4bb4_4cd2_abd4_e0bf8bf750fe.slice/crio-efc22ec63f202d071ea410e29195d25521906409310fd044fce1f6edb13c6cf4 WatchSource:0}: Error finding container efc22ec63f202d071ea410e29195d25521906409310fd044fce1f6edb13c6cf4: Status 404 returned error can't find the container with id efc22ec63f202d071ea410e29195d25521906409310fd044fce1f6edb13c6cf4 Jul 18 11:55:13.841494 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:55:13.841418 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-75bjg"] Jul 18 11:55:13.844740 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:55:13.844723 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-75bjg" Jul 18 11:55:13.852287 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:55:13.852147 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-75bjg"] Jul 18 11:55:13.952012 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:55:13.951976 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k6gcg\" (UniqueName: \"kubernetes.io/projected/72eecc76-acef-47ea-b4f2-d0fa11afb272-kube-api-access-k6gcg\") pod \"kuadrant-operator-catalog-75bjg\" (UID: \"72eecc76-acef-47ea-b4f2-d0fa11afb272\") " pod="kuadrant-system/kuadrant-operator-catalog-75bjg" Jul 18 11:55:14.053080 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:55:14.052989 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k6gcg\" (UniqueName: \"kubernetes.io/projected/72eecc76-acef-47ea-b4f2-d0fa11afb272-kube-api-access-k6gcg\") pod \"kuadrant-operator-catalog-75bjg\" (UID: \"72eecc76-acef-47ea-b4f2-d0fa11afb272\") " pod="kuadrant-system/kuadrant-operator-catalog-75bjg" Jul 18 11:55:14.061614 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:55:14.061588 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k6gcg\" (UniqueName: \"kubernetes.io/projected/72eecc76-acef-47ea-b4f2-d0fa11afb272-kube-api-access-k6gcg\") pod \"kuadrant-operator-catalog-75bjg\" (UID: \"72eecc76-acef-47ea-b4f2-d0fa11afb272\") " pod="kuadrant-system/kuadrant-operator-catalog-75bjg" Jul 18 11:55:14.155324 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:55:14.155245 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-75bjg" Jul 18 11:55:14.278828 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:55:14.278794 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-sgtl8" event={"ID":"b2cb7f1d-4bb4-4cd2-abd4-e0bf8bf750fe","Type":"ContainerStarted","Data":"efc22ec63f202d071ea410e29195d25521906409310fd044fce1f6edb13c6cf4"} Jul 18 11:55:14.291855 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:55:14.291823 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-75bjg"] Jul 18 11:55:14.294166 ip-10-0-136-240 kubenswrapper[2585]: W0718 11:55:14.294136 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod72eecc76_acef_47ea_b4f2_d0fa11afb272.slice/crio-f1612e24cc48bfa3379fbe338f3db9f63ae0a7c25333ffdceafa522adece3026 WatchSource:0}: Error finding container f1612e24cc48bfa3379fbe338f3db9f63ae0a7c25333ffdceafa522adece3026: Status 404 returned error can't find the container with id f1612e24cc48bfa3379fbe338f3db9f63ae0a7c25333ffdceafa522adece3026 Jul 18 11:55:15.287197 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:55:15.287160 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-75bjg" event={"ID":"72eecc76-acef-47ea-b4f2-d0fa11afb272","Type":"ContainerStarted","Data":"f1612e24cc48bfa3379fbe338f3db9f63ae0a7c25333ffdceafa522adece3026"} Jul 18 11:55:16.293469 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:55:16.293432 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-75bjg" event={"ID":"72eecc76-acef-47ea-b4f2-d0fa11afb272","Type":"ContainerStarted","Data":"25a5a1f71cc36830fdb0b06701804954c85cca0eef320cfacac3fe8f79dacdb6"} Jul 18 11:55:16.295126 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:55:16.295098 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-sgtl8" event={"ID":"b2cb7f1d-4bb4-4cd2-abd4-e0bf8bf750fe","Type":"ContainerStarted","Data":"0801302a0e108f936180c49dc88372804af12dea92027f64bfdf2b48cf4b24f2"} Jul 18 11:55:16.295126 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:55:16.295113 2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-catalog-sgtl8" podUID="b2cb7f1d-4bb4-4cd2-abd4-e0bf8bf750fe" containerName="registry-server" containerID="cri-o://0801302a0e108f936180c49dc88372804af12dea92027f64bfdf2b48cf4b24f2" gracePeriod=2 Jul 18 11:55:16.310605 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:55:16.310543 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-catalog-75bjg" podStartSLOduration=1.659001343 podStartE2EDuration="3.310528179s" totalImagesPullingTime="1.651526836s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-18 11:55:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-18 11:55:16.308604536 +0000 UTC m=+383.388810893" watchObservedRunningTime="2026-07-18 11:55:16.310528179 +0000 UTC m=+383.390734536" Jul 18 11:55:16.325221 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:55:16.325178 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-catalog-sgtl8" podStartSLOduration=1.1267821040000001 podStartE2EDuration="3.325164182s" totalImagesPullingTime="2.198382078s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-18 11:55:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-18 11:55:16.323763689 +0000 UTC m=+383.403970046" watchObservedRunningTime="2026-07-18 11:55:16.325164182 +0000 UTC m=+383.405370539" Jul 18 11:55:16.539433 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:55:16.539409 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-sgtl8" Jul 18 11:55:16.575781 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:55:16.575716 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7ksq9\" (UniqueName: \"kubernetes.io/projected/b2cb7f1d-4bb4-4cd2-abd4-e0bf8bf750fe-kube-api-access-7ksq9\") pod \"b2cb7f1d-4bb4-4cd2-abd4-e0bf8bf750fe\" (UID: \"b2cb7f1d-4bb4-4cd2-abd4-e0bf8bf750fe\") " Jul 18 11:55:16.577968 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:55:16.577942 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b2cb7f1d-4bb4-4cd2-abd4-e0bf8bf750fe-kube-api-access-7ksq9" (OuterVolumeSpecName: "kube-api-access-7ksq9") pod "b2cb7f1d-4bb4-4cd2-abd4-e0bf8bf750fe" (UID: "b2cb7f1d-4bb4-4cd2-abd4-e0bf8bf750fe"). InnerVolumeSpecName "kube-api-access-7ksq9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 18 11:55:16.676964 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:55:16.676936 2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7ksq9\" (UniqueName: \"kubernetes.io/projected/b2cb7f1d-4bb4-4cd2-abd4-e0bf8bf750fe-kube-api-access-7ksq9\") on node \"ip-10-0-136-240.ec2.internal\" DevicePath \"\"" Jul 18 11:55:17.300538 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:55:17.300458 2585 generic.go:356] "Generic (PLEG): container finished" podID="b2cb7f1d-4bb4-4cd2-abd4-e0bf8bf750fe" containerID="0801302a0e108f936180c49dc88372804af12dea92027f64bfdf2b48cf4b24f2" exitCode=0 Jul 18 11:55:17.300538 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:55:17.300504 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-sgtl8" Jul 18 11:55:17.301048 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:55:17.300541 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-sgtl8" event={"ID":"b2cb7f1d-4bb4-4cd2-abd4-e0bf8bf750fe","Type":"ContainerDied","Data":"0801302a0e108f936180c49dc88372804af12dea92027f64bfdf2b48cf4b24f2"} Jul 18 11:55:17.301048 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:55:17.300658 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-sgtl8" event={"ID":"b2cb7f1d-4bb4-4cd2-abd4-e0bf8bf750fe","Type":"ContainerDied","Data":"efc22ec63f202d071ea410e29195d25521906409310fd044fce1f6edb13c6cf4"} Jul 18 11:55:17.301048 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:55:17.300692 2585 scope.go:117] "RemoveContainer" containerID="0801302a0e108f936180c49dc88372804af12dea92027f64bfdf2b48cf4b24f2" Jul 18 11:55:17.310622 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:55:17.310605 2585 scope.go:117] "RemoveContainer" containerID="0801302a0e108f936180c49dc88372804af12dea92027f64bfdf2b48cf4b24f2" Jul 18 11:55:17.310905 ip-10-0-136-240 kubenswrapper[2585]: E0718 11:55:17.310886 2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0801302a0e108f936180c49dc88372804af12dea92027f64bfdf2b48cf4b24f2\": container with ID starting with 0801302a0e108f936180c49dc88372804af12dea92027f64bfdf2b48cf4b24f2 not found: ID does not exist" containerID="0801302a0e108f936180c49dc88372804af12dea92027f64bfdf2b48cf4b24f2" Jul 18 11:55:17.310955 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:55:17.310913 2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0801302a0e108f936180c49dc88372804af12dea92027f64bfdf2b48cf4b24f2"} err="failed to get container status \"0801302a0e108f936180c49dc88372804af12dea92027f64bfdf2b48cf4b24f2\": rpc error: code = NotFound desc = could not find container \"0801302a0e108f936180c49dc88372804af12dea92027f64bfdf2b48cf4b24f2\": container with ID starting with 0801302a0e108f936180c49dc88372804af12dea92027f64bfdf2b48cf4b24f2 not found: ID does not exist" Jul 18 11:55:17.320761 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:55:17.320708 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-sgtl8"] Jul 18 11:55:17.322625 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:55:17.322605 2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-sgtl8"] Jul 18 11:55:17.468424 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:55:17.468389 2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b2cb7f1d-4bb4-4cd2-abd4-e0bf8bf750fe" path="/var/lib/kubelet/pods/b2cb7f1d-4bb4-4cd2-abd4-e0bf8bf750fe/volumes" Jul 18 11:55:24.155430 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:55:24.155391 2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kuadrant-system/kuadrant-operator-catalog-75bjg" Jul 18 11:55:24.155932 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:55:24.155497 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-catalog-75bjg" Jul 18 11:55:24.179957 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:55:24.179933 2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kuadrant-system/kuadrant-operator-catalog-75bjg" Jul 18 11:55:24.355380 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:55:24.355347 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-catalog-75bjg" Jul 18 11:55:59.287344 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:55:59.287314 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-vbvt6"] Jul 18 11:55:59.287804 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:55:59.287672 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b2cb7f1d-4bb4-4cd2-abd4-e0bf8bf750fe" containerName="registry-server" Jul 18 11:55:59.287804 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:55:59.287684 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="b2cb7f1d-4bb4-4cd2-abd4-e0bf8bf750fe" containerName="registry-server" Jul 18 11:55:59.287804 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:55:59.287742 2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="b2cb7f1d-4bb4-4cd2-abd4-e0bf8bf750fe" containerName="registry-server" Jul 18 11:55:59.293863 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:55:59.293847 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-75cc8d5cb7-vbvt6" Jul 18 11:55:59.296559 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:55:59.296522 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-operator-dockercfg-274jx\"" Jul 18 11:55:59.301383 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:55:59.301360 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-vbvt6"] Jul 18 11:55:59.417212 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:55:59.417175 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7nzrd\" (UniqueName: \"kubernetes.io/projected/3a1aa8e5-bd4a-4cf2-b4d4-25c0dae5c37e-kube-api-access-7nzrd\") pod \"authorino-operator-75cc8d5cb7-vbvt6\" (UID: \"3a1aa8e5-bd4a-4cf2-b4d4-25c0dae5c37e\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-vbvt6" Jul 18 11:55:59.518103 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:55:59.518075 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7nzrd\" (UniqueName: \"kubernetes.io/projected/3a1aa8e5-bd4a-4cf2-b4d4-25c0dae5c37e-kube-api-access-7nzrd\") pod \"authorino-operator-75cc8d5cb7-vbvt6\" (UID: \"3a1aa8e5-bd4a-4cf2-b4d4-25c0dae5c37e\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-vbvt6" Jul 18 11:55:59.530569 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:55:59.530524 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7nzrd\" (UniqueName: \"kubernetes.io/projected/3a1aa8e5-bd4a-4cf2-b4d4-25c0dae5c37e-kube-api-access-7nzrd\") pod \"authorino-operator-75cc8d5cb7-vbvt6\" (UID: \"3a1aa8e5-bd4a-4cf2-b4d4-25c0dae5c37e\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-vbvt6" Jul 18 11:55:59.605077 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:55:59.605015 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-75cc8d5cb7-vbvt6" Jul 18 11:55:59.737658 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:55:59.737635 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-vbvt6"] Jul 18 11:55:59.739907 ip-10-0-136-240 kubenswrapper[2585]: W0718 11:55:59.739883 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3a1aa8e5_bd4a_4cf2_b4d4_25c0dae5c37e.slice/crio-7e91799977cfee32368d9d446fea47f0532903db28e5312f5975b77c4aaf1ccb WatchSource:0}: Error finding container 7e91799977cfee32368d9d446fea47f0532903db28e5312f5975b77c4aaf1ccb: Status 404 returned error can't find the container with id 7e91799977cfee32368d9d446fea47f0532903db28e5312f5975b77c4aaf1ccb Jul 18 11:56:00.503594 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:56:00.503540 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-75cc8d5cb7-vbvt6" event={"ID":"3a1aa8e5-bd4a-4cf2-b4d4-25c0dae5c37e","Type":"ContainerStarted","Data":"7e91799977cfee32368d9d446fea47f0532903db28e5312f5975b77c4aaf1ccb"} Jul 18 11:56:02.516561 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:56:02.516511 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-75cc8d5cb7-vbvt6" event={"ID":"3a1aa8e5-bd4a-4cf2-b4d4-25c0dae5c37e","Type":"ContainerStarted","Data":"06dd46961c781480bf1b78ae94a8e5d32c8caa2cb5c42e3668eb30dd61ecf609"} Jul 18 11:56:02.516958 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:56:02.516633 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/authorino-operator-75cc8d5cb7-vbvt6" Jul 18 11:56:02.542577 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:56:02.542507 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-operator-75cc8d5cb7-vbvt6" podStartSLOduration=1.335129333 podStartE2EDuration="3.542489677s" totalImagesPullingTime="2.207360344s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-18 11:55:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-18 11:56:02.541868819 +0000 UTC m=+429.622075385" watchObservedRunningTime="2026-07-18 11:56:02.542489677 +0000 UTC m=+429.622696034" Jul 18 11:56:02.546520 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:56:02.546493 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-vq5ct"] Jul 18 11:56:02.549916 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:56:02.549897 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-vq5ct" Jul 18 11:56:02.552439 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:56:02.552422 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\"" Jul 18 11:56:02.552599 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:56:02.552545 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-gwxfl\"" Jul 18 11:56:02.561996 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:56:02.561977 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-vq5ct"] Jul 18 11:56:02.645705 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:56:02.645679 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k22rl\" (UniqueName: \"kubernetes.io/projected/1f66be24-7dfd-4669-9f68-464837b42d1b-kube-api-access-k22rl\") pod \"dns-operator-controller-manager-864dd78c4c-vq5ct\" (UID: \"1f66be24-7dfd-4669-9f68-464837b42d1b\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-vq5ct" Jul 18 11:56:02.746812 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:56:02.746778 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k22rl\" (UniqueName: \"kubernetes.io/projected/1f66be24-7dfd-4669-9f68-464837b42d1b-kube-api-access-k22rl\") pod \"dns-operator-controller-manager-864dd78c4c-vq5ct\" (UID: \"1f66be24-7dfd-4669-9f68-464837b42d1b\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-vq5ct" Jul 18 11:56:02.760317 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:56:02.760281 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k22rl\" (UniqueName: \"kubernetes.io/projected/1f66be24-7dfd-4669-9f68-464837b42d1b-kube-api-access-k22rl\") pod \"dns-operator-controller-manager-864dd78c4c-vq5ct\" (UID: \"1f66be24-7dfd-4669-9f68-464837b42d1b\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-vq5ct" Jul 18 11:56:02.859803 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:56:02.859707 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-vq5ct" Jul 18 11:56:03.021246 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:56:03.021195 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-vq5ct"] Jul 18 11:56:03.024152 ip-10-0-136-240 kubenswrapper[2585]: W0718 11:56:03.024126 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1f66be24_7dfd_4669_9f68_464837b42d1b.slice/crio-e3a402055363f45d30d0e7a60a391fcd0d4c3a6b13d98747b463460187eb46f1 WatchSource:0}: Error finding container e3a402055363f45d30d0e7a60a391fcd0d4c3a6b13d98747b463460187eb46f1: Status 404 returned error can't find the container with id e3a402055363f45d30d0e7a60a391fcd0d4c3a6b13d98747b463460187eb46f1 Jul 18 11:56:03.522203 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:56:03.522169 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-vq5ct" event={"ID":"1f66be24-7dfd-4669-9f68-464837b42d1b","Type":"ContainerStarted","Data":"e3a402055363f45d30d0e7a60a391fcd0d4c3a6b13d98747b463460187eb46f1"} Jul 18 11:56:05.537020 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:56:05.536988 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-vq5ct" event={"ID":"1f66be24-7dfd-4669-9f68-464837b42d1b","Type":"ContainerStarted","Data":"7111a0f3b9fdebf0dd41cf4af14502fc554fcf375fcfd5c39eca1f149edb8257"} Jul 18 11:56:05.537393 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:56:05.537091 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-vq5ct" Jul 18 11:56:05.555724 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:56:05.555683 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-vq5ct" podStartSLOduration=1.66594692 podStartE2EDuration="3.555671625s" totalImagesPullingTime="1.889724705s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-18 11:56:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-18 11:56:05.554650407 +0000 UTC m=+432.634856764" watchObservedRunningTime="2026-07-18 11:56:05.555671625 +0000 UTC m=+432.635877981" Jul 18 11:56:06.040800 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:56:06.040765 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-cpfns"] Jul 18 11:56:06.050219 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:56:06.050195 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-cpfns" Jul 18 11:56:06.053771 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:56:06.053745 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-wwk74\"" Jul 18 11:56:06.054237 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:56:06.054206 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-cpfns"] Jul 18 11:56:06.177140 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:56:06.177106 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bx6dz\" (UniqueName: \"kubernetes.io/projected/f82f2ba0-86db-458a-b369-b8d01a9a794f-kube-api-access-bx6dz\") pod \"limitador-operator-controller-manager-859d7b78bd-cpfns\" (UID: \"f82f2ba0-86db-458a-b369-b8d01a9a794f\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-cpfns" Jul 18 11:56:06.278582 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:56:06.278518 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bx6dz\" (UniqueName: \"kubernetes.io/projected/f82f2ba0-86db-458a-b369-b8d01a9a794f-kube-api-access-bx6dz\") pod \"limitador-operator-controller-manager-859d7b78bd-cpfns\" (UID: \"f82f2ba0-86db-458a-b369-b8d01a9a794f\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-cpfns" Jul 18 11:56:06.287614 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:56:06.287584 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bx6dz\" (UniqueName: \"kubernetes.io/projected/f82f2ba0-86db-458a-b369-b8d01a9a794f-kube-api-access-bx6dz\") pod \"limitador-operator-controller-manager-859d7b78bd-cpfns\" (UID: \"f82f2ba0-86db-458a-b369-b8d01a9a794f\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-cpfns" Jul 18 11:56:06.361680 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:56:06.361612 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-cpfns" Jul 18 11:56:06.494005 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:56:06.493930 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-cpfns"] Jul 18 11:56:06.496373 ip-10-0-136-240 kubenswrapper[2585]: W0718 11:56:06.496347 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf82f2ba0_86db_458a_b369_b8d01a9a794f.slice/crio-9b65303daa92be7c521b5b00e9a68fa08cb4f208ea84680ff1a7be982d9e4edd WatchSource:0}: Error finding container 9b65303daa92be7c521b5b00e9a68fa08cb4f208ea84680ff1a7be982d9e4edd: Status 404 returned error can't find the container with id 9b65303daa92be7c521b5b00e9a68fa08cb4f208ea84680ff1a7be982d9e4edd Jul 18 11:56:06.542612 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:56:06.542582 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-cpfns" event={"ID":"f82f2ba0-86db-458a-b369-b8d01a9a794f","Type":"ContainerStarted","Data":"9b65303daa92be7c521b5b00e9a68fa08cb4f208ea84680ff1a7be982d9e4edd"} Jul 18 11:56:08.554082 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:56:08.554043 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-cpfns" event={"ID":"f82f2ba0-86db-458a-b369-b8d01a9a794f","Type":"ContainerStarted","Data":"51135e0d105ca48594e145e3dbf3c57c6a1f4cb7e72d2a9f690a724c7d21f989"} Jul 18 11:56:08.554469 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:56:08.554149 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-cpfns" Jul 18 11:56:08.572007 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:56:08.571959 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-cpfns" podStartSLOduration=1.186938102 podStartE2EDuration="2.571944254s" totalImagesPullingTime="1.385006152s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-18 11:56:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-18 11:56:08.570848303 +0000 UTC m=+435.651054663" watchObservedRunningTime="2026-07-18 11:56:08.571944254 +0000 UTC m=+435.652150610" Jul 18 11:56:13.524904 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:56:13.524880 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/authorino-operator-75cc8d5cb7-vbvt6" Jul 18 11:56:16.544850 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:56:16.544781 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-vq5ct" Jul 18 11:56:19.561965 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:56:19.561936 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-cpfns" Jul 18 11:56:29.271413 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:56:29.271377 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-cpfns"] Jul 18 11:56:29.271843 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:56:29.271658 2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-cpfns" podUID="f82f2ba0-86db-458a-b369-b8d01a9a794f" containerName="manager" containerID="cri-o://51135e0d105ca48594e145e3dbf3c57c6a1f4cb7e72d2a9f690a724c7d21f989" gracePeriod=2 Jul 18 11:56:29.280122 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:56:29.280094 2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-cpfns"] Jul 18 11:56:29.301522 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:56:29.301498 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-5kx76"] Jul 18 11:56:29.301872 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:56:29.301859 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f82f2ba0-86db-458a-b369-b8d01a9a794f" containerName="manager" Jul 18 11:56:29.301922 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:56:29.301874 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="f82f2ba0-86db-458a-b369-b8d01a9a794f" containerName="manager" Jul 18 11:56:29.301957 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:56:29.301944 2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="f82f2ba0-86db-458a-b369-b8d01a9a794f" containerName="manager" Jul 18 11:56:29.304874 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:56:29.304852 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-5kx76" Jul 18 11:56:29.307071 ip-10-0-136-240 kubenswrapper[2585]: E0718 11:56:29.307046 2585 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-cpfns\" is forbidden: User \"system:node:ip-10-0-136-240.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-136-240.ec2.internal' and this object" podUID="f82f2ba0-86db-458a-b369-b8d01a9a794f" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-cpfns" Jul 18 11:56:29.315902 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:56:29.315876 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-5kx76"] Jul 18 11:56:29.361666 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:56:29.361639 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rdz5x\" (UniqueName: \"kubernetes.io/projected/2f6f85c7-2112-4511-b178-6d602a8ce6fe-kube-api-access-rdz5x\") pod \"limitador-operator-controller-manager-859d7b78bd-5kx76\" (UID: \"2f6f85c7-2112-4511-b178-6d602a8ce6fe\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-5kx76" Jul 18 11:56:29.462715 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:56:29.462689 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rdz5x\" (UniqueName: \"kubernetes.io/projected/2f6f85c7-2112-4511-b178-6d602a8ce6fe-kube-api-access-rdz5x\") pod \"limitador-operator-controller-manager-859d7b78bd-5kx76\" (UID: \"2f6f85c7-2112-4511-b178-6d602a8ce6fe\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-5kx76" Jul 18 11:56:29.477787 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:56:29.470638 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rdz5x\" (UniqueName: \"kubernetes.io/projected/2f6f85c7-2112-4511-b178-6d602a8ce6fe-kube-api-access-rdz5x\") pod \"limitador-operator-controller-manager-859d7b78bd-5kx76\" (UID: \"2f6f85c7-2112-4511-b178-6d602a8ce6fe\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-5kx76" Jul 18 11:56:29.510506 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:56:29.510484 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-cpfns" Jul 18 11:56:29.563776 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:56:29.563724 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bx6dz\" (UniqueName: \"kubernetes.io/projected/f82f2ba0-86db-458a-b369-b8d01a9a794f-kube-api-access-bx6dz\") pod \"f82f2ba0-86db-458a-b369-b8d01a9a794f\" (UID: \"f82f2ba0-86db-458a-b369-b8d01a9a794f\") " Jul 18 11:56:29.565783 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:56:29.565755 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f82f2ba0-86db-458a-b369-b8d01a9a794f-kube-api-access-bx6dz" (OuterVolumeSpecName: "kube-api-access-bx6dz") pod "f82f2ba0-86db-458a-b369-b8d01a9a794f" (UID: "f82f2ba0-86db-458a-b369-b8d01a9a794f"). InnerVolumeSpecName "kube-api-access-bx6dz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 18 11:56:29.651614 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:56:29.651584 2585 generic.go:356] "Generic (PLEG): container finished" podID="f82f2ba0-86db-458a-b369-b8d01a9a794f" containerID="51135e0d105ca48594e145e3dbf3c57c6a1f4cb7e72d2a9f690a724c7d21f989" exitCode=0 Jul 18 11:56:29.651752 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:56:29.651635 2585 scope.go:117] "RemoveContainer" containerID="51135e0d105ca48594e145e3dbf3c57c6a1f4cb7e72d2a9f690a724c7d21f989" Jul 18 11:56:29.651752 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:56:29.651636 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-cpfns" Jul 18 11:56:29.653085 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:56:29.653010 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-5kx76" Jul 18 11:56:29.664397 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:56:29.664380 2585 scope.go:117] "RemoveContainer" containerID="51135e0d105ca48594e145e3dbf3c57c6a1f4cb7e72d2a9f690a724c7d21f989" Jul 18 11:56:29.664635 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:56:29.664613 2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bx6dz\" (UniqueName: \"kubernetes.io/projected/f82f2ba0-86db-458a-b369-b8d01a9a794f-kube-api-access-bx6dz\") on node \"ip-10-0-136-240.ec2.internal\" DevicePath \"\"" Jul 18 11:56:29.664741 ip-10-0-136-240 kubenswrapper[2585]: E0718 11:56:29.664697 2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"51135e0d105ca48594e145e3dbf3c57c6a1f4cb7e72d2a9f690a724c7d21f989\": container with ID starting with 51135e0d105ca48594e145e3dbf3c57c6a1f4cb7e72d2a9f690a724c7d21f989 not found: ID does not exist" containerID="51135e0d105ca48594e145e3dbf3c57c6a1f4cb7e72d2a9f690a724c7d21f989" Jul 18 11:56:29.664797 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:56:29.664731 2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"51135e0d105ca48594e145e3dbf3c57c6a1f4cb7e72d2a9f690a724c7d21f989"} err="failed to get container status \"51135e0d105ca48594e145e3dbf3c57c6a1f4cb7e72d2a9f690a724c7d21f989\": rpc error: code = NotFound desc = could not find container \"51135e0d105ca48594e145e3dbf3c57c6a1f4cb7e72d2a9f690a724c7d21f989\": container with ID starting with 51135e0d105ca48594e145e3dbf3c57c6a1f4cb7e72d2a9f690a724c7d21f989 not found: ID does not exist" Jul 18 11:56:29.792906 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:56:29.792877 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-5kx76"] Jul 18 11:56:29.795314 ip-10-0-136-240 kubenswrapper[2585]: W0718 11:56:29.795289 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2f6f85c7_2112_4511_b178_6d602a8ce6fe.slice/crio-7385eb79bf6b5634c4c7c2a810a0a09572fa7468ba9d38ee3452ff3397ac0001 WatchSource:0}: Error finding container 7385eb79bf6b5634c4c7c2a810a0a09572fa7468ba9d38ee3452ff3397ac0001: Status 404 returned error can't find the container with id 7385eb79bf6b5634c4c7c2a810a0a09572fa7468ba9d38ee3452ff3397ac0001 Jul 18 11:56:30.658692 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:56:30.658664 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-5kx76" event={"ID":"2f6f85c7-2112-4511-b178-6d602a8ce6fe","Type":"ContainerStarted","Data":"828119fda431f69b2f6ab24569a0e1f1438db62f67e06727bea2798051487884"} Jul 18 11:56:30.659149 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:56:30.658712 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-5kx76" event={"ID":"2f6f85c7-2112-4511-b178-6d602a8ce6fe","Type":"ContainerStarted","Data":"7385eb79bf6b5634c4c7c2a810a0a09572fa7468ba9d38ee3452ff3397ac0001"} Jul 18 11:56:30.659149 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:56:30.658794 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-5kx76" Jul 18 11:56:30.677496 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:56:30.677443 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-5kx76" podStartSLOduration=1.677429684 podStartE2EDuration="1.677429684s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-18 11:56:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-18 11:56:30.675665164 +0000 UTC m=+457.755871523" watchObservedRunningTime="2026-07-18 11:56:30.677429684 +0000 UTC m=+457.757636040" Jul 18 11:56:31.468621 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:56:31.468582 2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f82f2ba0-86db-458a-b369-b8d01a9a794f" path="/var/lib/kubelet/pods/f82f2ba0-86db-458a-b369-b8d01a9a794f/volumes" Jul 18 11:56:41.666195 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:56:41.666166 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-5kx76" Jul 18 11:56:58.069495 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:56:58.069465 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-2gcb2"] Jul 18 11:56:58.074107 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:56:58.074088 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-2gcb2" Jul 18 11:56:58.077279 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:56:58.077262 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-5pqgx\"" Jul 18 11:56:58.085483 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:56:58.085462 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-2gcb2"] Jul 18 11:56:58.200919 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:56:58.200888 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/faf0508a-7db7-4cbe-915a-499c404d0bd0-istio-podinfo\") pod \"maas-default-gateway-openshift-default-56654dcf7-2gcb2\" (UID: \"faf0508a-7db7-4cbe-915a-499c404d0bd0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-2gcb2" Jul 18 11:56:58.201094 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:56:58.200980 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/faf0508a-7db7-4cbe-915a-499c404d0bd0-istio-token\") pod \"maas-default-gateway-openshift-default-56654dcf7-2gcb2\" (UID: \"faf0508a-7db7-4cbe-915a-499c404d0bd0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-2gcb2" Jul 18 11:56:58.201094 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:56:58.201044 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/faf0508a-7db7-4cbe-915a-499c404d0bd0-workload-certs\") pod \"maas-default-gateway-openshift-default-56654dcf7-2gcb2\" (UID: \"faf0508a-7db7-4cbe-915a-499c404d0bd0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-2gcb2" Jul 18 11:56:58.201094 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:56:58.201070 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/faf0508a-7db7-4cbe-915a-499c404d0bd0-workload-socket\") pod \"maas-default-gateway-openshift-default-56654dcf7-2gcb2\" (UID: \"faf0508a-7db7-4cbe-915a-499c404d0bd0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-2gcb2" Jul 18 11:56:58.201094 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:56:58.201091 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/faf0508a-7db7-4cbe-915a-499c404d0bd0-istio-data\") pod \"maas-default-gateway-openshift-default-56654dcf7-2gcb2\" (UID: \"faf0508a-7db7-4cbe-915a-499c404d0bd0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-2gcb2" Jul 18 11:56:58.201303 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:56:58.201118 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/faf0508a-7db7-4cbe-915a-499c404d0bd0-credential-socket\") pod \"maas-default-gateway-openshift-default-56654dcf7-2gcb2\" (UID: \"faf0508a-7db7-4cbe-915a-499c404d0bd0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-2gcb2" Jul 18 11:56:58.201303 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:56:58.201146 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/faf0508a-7db7-4cbe-915a-499c404d0bd0-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-56654dcf7-2gcb2\" (UID: \"faf0508a-7db7-4cbe-915a-499c404d0bd0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-2gcb2" Jul 18 11:56:58.201303 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:56:58.201176 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rjfjw\" (UniqueName: \"kubernetes.io/projected/faf0508a-7db7-4cbe-915a-499c404d0bd0-kube-api-access-rjfjw\") pod \"maas-default-gateway-openshift-default-56654dcf7-2gcb2\" (UID: \"faf0508a-7db7-4cbe-915a-499c404d0bd0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-2gcb2" Jul 18 11:56:58.201303 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:56:58.201211 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/faf0508a-7db7-4cbe-915a-499c404d0bd0-istio-envoy\") pod \"maas-default-gateway-openshift-default-56654dcf7-2gcb2\" (UID: \"faf0508a-7db7-4cbe-915a-499c404d0bd0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-2gcb2" Jul 18 11:56:58.301916 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:56:58.301886 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/faf0508a-7db7-4cbe-915a-499c404d0bd0-istio-envoy\") pod \"maas-default-gateway-openshift-default-56654dcf7-2gcb2\" (UID: \"faf0508a-7db7-4cbe-915a-499c404d0bd0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-2gcb2" Jul 18 11:56:58.302055 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:56:58.301924 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/faf0508a-7db7-4cbe-915a-499c404d0bd0-istio-podinfo\") pod \"maas-default-gateway-openshift-default-56654dcf7-2gcb2\" (UID: \"faf0508a-7db7-4cbe-915a-499c404d0bd0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-2gcb2" Jul 18 11:56:58.302055 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:56:58.301971 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/faf0508a-7db7-4cbe-915a-499c404d0bd0-istio-token\") pod \"maas-default-gateway-openshift-default-56654dcf7-2gcb2\" (UID: \"faf0508a-7db7-4cbe-915a-499c404d0bd0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-2gcb2" Jul 18 11:56:58.302055 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:56:58.302008 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/faf0508a-7db7-4cbe-915a-499c404d0bd0-workload-certs\") pod \"maas-default-gateway-openshift-default-56654dcf7-2gcb2\" (UID: \"faf0508a-7db7-4cbe-915a-499c404d0bd0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-2gcb2" Jul 18 11:56:58.302055 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:56:58.302027 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/faf0508a-7db7-4cbe-915a-499c404d0bd0-workload-socket\") pod \"maas-default-gateway-openshift-default-56654dcf7-2gcb2\" (UID: \"faf0508a-7db7-4cbe-915a-499c404d0bd0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-2gcb2" Jul 18 11:56:58.302055 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:56:58.302048 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/faf0508a-7db7-4cbe-915a-499c404d0bd0-istio-data\") pod \"maas-default-gateway-openshift-default-56654dcf7-2gcb2\" (UID: \"faf0508a-7db7-4cbe-915a-499c404d0bd0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-2gcb2" Jul 18 11:56:58.302309 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:56:58.302074 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/faf0508a-7db7-4cbe-915a-499c404d0bd0-credential-socket\") pod \"maas-default-gateway-openshift-default-56654dcf7-2gcb2\" (UID: \"faf0508a-7db7-4cbe-915a-499c404d0bd0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-2gcb2" Jul 18 11:56:58.302309 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:56:58.302117 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/faf0508a-7db7-4cbe-915a-499c404d0bd0-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-56654dcf7-2gcb2\" (UID: \"faf0508a-7db7-4cbe-915a-499c404d0bd0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-2gcb2" Jul 18 11:56:58.302309 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:56:58.302147 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rjfjw\" (UniqueName: \"kubernetes.io/projected/faf0508a-7db7-4cbe-915a-499c404d0bd0-kube-api-access-rjfjw\") pod \"maas-default-gateway-openshift-default-56654dcf7-2gcb2\" (UID: \"faf0508a-7db7-4cbe-915a-499c404d0bd0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-2gcb2" Jul 18 11:56:58.302924 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:56:58.302895 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/faf0508a-7db7-4cbe-915a-499c404d0bd0-istio-data\") pod \"maas-default-gateway-openshift-default-56654dcf7-2gcb2\" (UID: \"faf0508a-7db7-4cbe-915a-499c404d0bd0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-2gcb2" Jul 18 11:56:58.303026 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:56:58.302925 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/faf0508a-7db7-4cbe-915a-499c404d0bd0-workload-certs\") pod \"maas-default-gateway-openshift-default-56654dcf7-2gcb2\" (UID: \"faf0508a-7db7-4cbe-915a-499c404d0bd0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-2gcb2" Jul 18 11:56:58.303323 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:56:58.303297 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/faf0508a-7db7-4cbe-915a-499c404d0bd0-workload-socket\") pod \"maas-default-gateway-openshift-default-56654dcf7-2gcb2\" (UID: \"faf0508a-7db7-4cbe-915a-499c404d0bd0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-2gcb2" Jul 18 11:56:58.303436 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:56:58.303410 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/faf0508a-7db7-4cbe-915a-499c404d0bd0-credential-socket\") pod \"maas-default-gateway-openshift-default-56654dcf7-2gcb2\" (UID: \"faf0508a-7db7-4cbe-915a-499c404d0bd0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-2gcb2" Jul 18 11:56:58.303787 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:56:58.303763 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/faf0508a-7db7-4cbe-915a-499c404d0bd0-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-56654dcf7-2gcb2\" (UID: \"faf0508a-7db7-4cbe-915a-499c404d0bd0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-2gcb2" Jul 18 11:56:58.310840 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:56:58.310810 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/faf0508a-7db7-4cbe-915a-499c404d0bd0-istio-envoy\") pod \"maas-default-gateway-openshift-default-56654dcf7-2gcb2\" (UID: \"faf0508a-7db7-4cbe-915a-499c404d0bd0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-2gcb2" Jul 18 11:56:58.311291 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:56:58.311272 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/faf0508a-7db7-4cbe-915a-499c404d0bd0-istio-podinfo\") pod \"maas-default-gateway-openshift-default-56654dcf7-2gcb2\" (UID: \"faf0508a-7db7-4cbe-915a-499c404d0bd0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-2gcb2" Jul 18 11:56:58.321885 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:56:58.321839 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/faf0508a-7db7-4cbe-915a-499c404d0bd0-istio-token\") pod \"maas-default-gateway-openshift-default-56654dcf7-2gcb2\" (UID: \"faf0508a-7db7-4cbe-915a-499c404d0bd0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-2gcb2" Jul 18 11:56:58.321980 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:56:58.321963 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rjfjw\" (UniqueName: \"kubernetes.io/projected/faf0508a-7db7-4cbe-915a-499c404d0bd0-kube-api-access-rjfjw\") pod \"maas-default-gateway-openshift-default-56654dcf7-2gcb2\" (UID: \"faf0508a-7db7-4cbe-915a-499c404d0bd0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-2gcb2" Jul 18 11:56:58.385211 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:56:58.385188 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-2gcb2" Jul 18 11:56:58.530765 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:56:58.530741 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-2gcb2"] Jul 18 11:56:58.532539 ip-10-0-136-240 kubenswrapper[2585]: W0718 11:56:58.532517 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfaf0508a_7db7_4cbe_915a_499c404d0bd0.slice/crio-22df8a8b2e8f705bf95c32a2595e81f216dece75542ad318537a4eb86ce65afb WatchSource:0}: Error finding container 22df8a8b2e8f705bf95c32a2595e81f216dece75542ad318537a4eb86ce65afb: Status 404 returned error can't find the container with id 22df8a8b2e8f705bf95c32a2595e81f216dece75542ad318537a4eb86ce65afb Jul 18 11:56:58.534671 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:56:58.534637 2585 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 18 11:56:58.534749 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:56:58.534700 2585 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 18 11:56:58.534749 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:56:58.534729 2585 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 18 11:56:58.797396 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:56:58.797360 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-2gcb2" event={"ID":"faf0508a-7db7-4cbe-915a-499c404d0bd0","Type":"ContainerStarted","Data":"8316b5ff9c6ae17d41cf4f5f17d2558744a670f5d8460c0a288dcfc1f558cf5a"} Jul 18 11:56:58.797672 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:56:58.797440 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-2gcb2" event={"ID":"faf0508a-7db7-4cbe-915a-499c404d0bd0","Type":"ContainerStarted","Data":"22df8a8b2e8f705bf95c32a2595e81f216dece75542ad318537a4eb86ce65afb"} Jul 18 11:56:58.819030 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:56:58.818977 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-2gcb2" podStartSLOduration=0.818959535 podStartE2EDuration="818.959535ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-18 11:56:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-18 11:56:58.818043727 +0000 UTC m=+485.898250085" watchObservedRunningTime="2026-07-18 11:56:58.818959535 +0000 UTC m=+485.899165892" Jul 18 11:56:59.385325 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:56:59.385293 2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-2gcb2" Jul 18 11:56:59.390165 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:56:59.390141 2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-2gcb2" Jul 18 11:56:59.802948 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:56:59.802915 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-2gcb2" Jul 18 11:56:59.804065 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:56:59.804043 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-2gcb2" Jul 18 11:57:19.514875 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:57:19.514839 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-75d556bd77-pp8x9"] Jul 18 11:57:19.518391 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:57:19.518370 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-pp8x9" Jul 18 11:57:19.520979 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:57:19.520960 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-xlzqh\"" Jul 18 11:57:19.526974 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:57:19.526947 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-75d556bd77-pp8x9"] Jul 18 11:57:19.584208 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:57:19.584187 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-phhng\" (UniqueName: \"kubernetes.io/projected/fbd1b5c3-a4ee-4e88-80cb-c3980b165cdd-kube-api-access-phhng\") pod \"authorino-75d556bd77-pp8x9\" (UID: \"fbd1b5c3-a4ee-4e88-80cb-c3980b165cdd\") " pod="kuadrant-system/authorino-75d556bd77-pp8x9" Jul 18 11:57:19.685002 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:57:19.684963 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-phhng\" (UniqueName: \"kubernetes.io/projected/fbd1b5c3-a4ee-4e88-80cb-c3980b165cdd-kube-api-access-phhng\") pod \"authorino-75d556bd77-pp8x9\" (UID: \"fbd1b5c3-a4ee-4e88-80cb-c3980b165cdd\") " pod="kuadrant-system/authorino-75d556bd77-pp8x9" Jul 18 11:57:19.693597 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:57:19.693573 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-phhng\" (UniqueName: \"kubernetes.io/projected/fbd1b5c3-a4ee-4e88-80cb-c3980b165cdd-kube-api-access-phhng\") pod \"authorino-75d556bd77-pp8x9\" (UID: \"fbd1b5c3-a4ee-4e88-80cb-c3980b165cdd\") " pod="kuadrant-system/authorino-75d556bd77-pp8x9" Jul 18 11:57:19.829170 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:57:19.829112 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-pp8x9" Jul 18 11:57:19.963070 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:57:19.962999 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-75d556bd77-pp8x9"] Jul 18 11:57:19.965495 ip-10-0-136-240 kubenswrapper[2585]: W0718 11:57:19.965471 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podfbd1b5c3_a4ee_4e88_80cb_c3980b165cdd.slice/crio-fad060455ae64ef4a95f2d72244de50b2052e0a4b8ea038c4463c4712d81c85f WatchSource:0}: Error finding container fad060455ae64ef4a95f2d72244de50b2052e0a4b8ea038c4463c4712d81c85f: Status 404 returned error can't find the container with id fad060455ae64ef4a95f2d72244de50b2052e0a4b8ea038c4463c4712d81c85f Jul 18 11:57:20.905189 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:57:20.905128 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-pp8x9" event={"ID":"fbd1b5c3-a4ee-4e88-80cb-c3980b165cdd","Type":"ContainerStarted","Data":"fad060455ae64ef4a95f2d72244de50b2052e0a4b8ea038c4463c4712d81c85f"} Jul 18 11:57:22.923543 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:57:22.923497 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-pp8x9" event={"ID":"fbd1b5c3-a4ee-4e88-80cb-c3980b165cdd","Type":"ContainerStarted","Data":"1ccdf2bfa1c83479dba6fb900d565314bd39d020b24d77d85915016007827a86"} Jul 18 11:57:22.938995 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:57:22.938933 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-75d556bd77-pp8x9" podStartSLOduration=1.103320542 podStartE2EDuration="3.938916933s" totalImagesPullingTime="2.835596391s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-18 11:57:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-18 11:57:22.937352861 +0000 UTC m=+510.017559221" watchObservedRunningTime="2026-07-18 11:57:22.938916933 +0000 UTC m=+510.019123289" Jul 18 11:57:48.591397 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:57:48.591311 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-5d4b67c446-q2tfj"] Jul 18 11:57:48.594917 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:57:48.594894 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-q2tfj" Jul 18 11:57:48.602997 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:57:48.602973 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-q2tfj"] Jul 18 11:57:48.745073 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:57:48.745045 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2s2bv\" (UniqueName: \"kubernetes.io/projected/d7761158-d8cc-411f-92d7-e49963ea7fc7-kube-api-access-2s2bv\") pod \"authorino-5d4b67c446-q2tfj\" (UID: \"d7761158-d8cc-411f-92d7-e49963ea7fc7\") " pod="kuadrant-system/authorino-5d4b67c446-q2tfj" Jul 18 11:57:48.814217 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:57:48.814183 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-q2tfj"] Jul 18 11:57:48.814489 ip-10-0-136-240 kubenswrapper[2585]: E0718 11:57:48.814467 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-2s2bv], unattached volumes=[], failed to process volumes=[]: context canceled" pod="kuadrant-system/authorino-5d4b67c446-q2tfj" podUID="d7761158-d8cc-411f-92d7-e49963ea7fc7" Jul 18 11:57:48.838018 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:57:48.837993 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-6c4dd6cfb5-gw8tb"] Jul 18 11:57:48.841541 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:57:48.841495 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6c4dd6cfb5-gw8tb" Jul 18 11:57:48.845486 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:57:48.845456 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2s2bv\" (UniqueName: \"kubernetes.io/projected/d7761158-d8cc-411f-92d7-e49963ea7fc7-kube-api-access-2s2bv\") pod \"authorino-5d4b67c446-q2tfj\" (UID: \"d7761158-d8cc-411f-92d7-e49963ea7fc7\") " pod="kuadrant-system/authorino-5d4b67c446-q2tfj" Jul 18 11:57:48.847777 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:57:48.847751 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6c4dd6cfb5-gw8tb"] Jul 18 11:57:48.859382 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:57:48.859360 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2s2bv\" (UniqueName: \"kubernetes.io/projected/d7761158-d8cc-411f-92d7-e49963ea7fc7-kube-api-access-2s2bv\") pod \"authorino-5d4b67c446-q2tfj\" (UID: \"d7761158-d8cc-411f-92d7-e49963ea7fc7\") " pod="kuadrant-system/authorino-5d4b67c446-q2tfj" Jul 18 11:57:48.946333 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:57:48.946302 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gcj2z\" (UniqueName: \"kubernetes.io/projected/382f681a-44aa-4118-9d1a-d0f31e0df614-kube-api-access-gcj2z\") pod \"authorino-6c4dd6cfb5-gw8tb\" (UID: \"382f681a-44aa-4118-9d1a-d0f31e0df614\") " pod="kuadrant-system/authorino-6c4dd6cfb5-gw8tb" Jul 18 11:57:48.949364 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:57:48.949336 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6c4dd6cfb5-gw8tb"] Jul 18 11:57:48.949649 ip-10-0-136-240 kubenswrapper[2585]: E0718 11:57:48.949628 2585 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-gcj2z], unattached volumes=[], failed to process volumes=[]: context canceled" pod="kuadrant-system/authorino-6c4dd6cfb5-gw8tb" podUID="382f681a-44aa-4118-9d1a-d0f31e0df614" Jul 18 11:57:48.975122 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:57:48.975098 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-795bc545c7-4zh9h"] Jul 18 11:57:48.978619 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:57:48.978602 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-795bc545c7-4zh9h" Jul 18 11:57:48.981098 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:57:48.981076 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\"" Jul 18 11:57:48.986012 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:57:48.985988 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-795bc545c7-4zh9h"] Jul 18 11:57:49.047062 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:57:49.047041 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gcj2z\" (UniqueName: \"kubernetes.io/projected/382f681a-44aa-4118-9d1a-d0f31e0df614-kube-api-access-gcj2z\") pod \"authorino-6c4dd6cfb5-gw8tb\" (UID: \"382f681a-44aa-4118-9d1a-d0f31e0df614\") " pod="kuadrant-system/authorino-6c4dd6cfb5-gw8tb" Jul 18 11:57:49.049916 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:57:49.049895 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6c4dd6cfb5-gw8tb" Jul 18 11:57:49.050018 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:57:49.049903 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-q2tfj" Jul 18 11:57:49.056247 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:57:49.056224 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gcj2z\" (UniqueName: \"kubernetes.io/projected/382f681a-44aa-4118-9d1a-d0f31e0df614-kube-api-access-gcj2z\") pod \"authorino-6c4dd6cfb5-gw8tb\" (UID: \"382f681a-44aa-4118-9d1a-d0f31e0df614\") " pod="kuadrant-system/authorino-6c4dd6cfb5-gw8tb" Jul 18 11:57:49.056342 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:57:49.056285 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6c4dd6cfb5-gw8tb" Jul 18 11:57:49.068364 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:57:49.068347 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-q2tfj" Jul 18 11:57:49.147654 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:57:49.147597 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gcj2z\" (UniqueName: \"kubernetes.io/projected/382f681a-44aa-4118-9d1a-d0f31e0df614-kube-api-access-gcj2z\") pod \"382f681a-44aa-4118-9d1a-d0f31e0df614\" (UID: \"382f681a-44aa-4118-9d1a-d0f31e0df614\") " Jul 18 11:57:49.147753 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:57:49.147727 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/ff12bf45-bdbb-4252-847c-24de86a67842-tls-cert\") pod \"authorino-795bc545c7-4zh9h\" (UID: \"ff12bf45-bdbb-4252-847c-24de86a67842\") " pod="kuadrant-system/authorino-795bc545c7-4zh9h" Jul 18 11:57:49.147798 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:57:49.147778 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jlsb7\" (UniqueName: \"kubernetes.io/projected/ff12bf45-bdbb-4252-847c-24de86a67842-kube-api-access-jlsb7\") pod \"authorino-795bc545c7-4zh9h\" (UID: \"ff12bf45-bdbb-4252-847c-24de86a67842\") " pod="kuadrant-system/authorino-795bc545c7-4zh9h" Jul 18 11:57:49.149770 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:57:49.149747 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/382f681a-44aa-4118-9d1a-d0f31e0df614-kube-api-access-gcj2z" (OuterVolumeSpecName: "kube-api-access-gcj2z") pod "382f681a-44aa-4118-9d1a-d0f31e0df614" (UID: "382f681a-44aa-4118-9d1a-d0f31e0df614"). InnerVolumeSpecName "kube-api-access-gcj2z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 18 11:57:49.249080 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:57:49.249059 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2s2bv\" (UniqueName: \"kubernetes.io/projected/d7761158-d8cc-411f-92d7-e49963ea7fc7-kube-api-access-2s2bv\") pod \"d7761158-d8cc-411f-92d7-e49963ea7fc7\" (UID: \"d7761158-d8cc-411f-92d7-e49963ea7fc7\") " Jul 18 11:57:49.249261 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:57:49.249244 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/ff12bf45-bdbb-4252-847c-24de86a67842-tls-cert\") pod \"authorino-795bc545c7-4zh9h\" (UID: \"ff12bf45-bdbb-4252-847c-24de86a67842\") " pod="kuadrant-system/authorino-795bc545c7-4zh9h" Jul 18 11:57:49.249333 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:57:49.249317 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jlsb7\" (UniqueName: \"kubernetes.io/projected/ff12bf45-bdbb-4252-847c-24de86a67842-kube-api-access-jlsb7\") pod \"authorino-795bc545c7-4zh9h\" (UID: \"ff12bf45-bdbb-4252-847c-24de86a67842\") " pod="kuadrant-system/authorino-795bc545c7-4zh9h" Jul 18 11:57:49.249409 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:57:49.249395 2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gcj2z\" (UniqueName: \"kubernetes.io/projected/382f681a-44aa-4118-9d1a-d0f31e0df614-kube-api-access-gcj2z\") on node \"ip-10-0-136-240.ec2.internal\" DevicePath \"\"" Jul 18 11:57:49.251143 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:57:49.251123 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d7761158-d8cc-411f-92d7-e49963ea7fc7-kube-api-access-2s2bv" (OuterVolumeSpecName: "kube-api-access-2s2bv") pod "d7761158-d8cc-411f-92d7-e49963ea7fc7" (UID: "d7761158-d8cc-411f-92d7-e49963ea7fc7"). InnerVolumeSpecName "kube-api-access-2s2bv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 18 11:57:49.251698 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:57:49.251680 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/ff12bf45-bdbb-4252-847c-24de86a67842-tls-cert\") pod \"authorino-795bc545c7-4zh9h\" (UID: \"ff12bf45-bdbb-4252-847c-24de86a67842\") " pod="kuadrant-system/authorino-795bc545c7-4zh9h" Jul 18 11:57:49.259223 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:57:49.259201 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jlsb7\" (UniqueName: \"kubernetes.io/projected/ff12bf45-bdbb-4252-847c-24de86a67842-kube-api-access-jlsb7\") pod \"authorino-795bc545c7-4zh9h\" (UID: \"ff12bf45-bdbb-4252-847c-24de86a67842\") " pod="kuadrant-system/authorino-795bc545c7-4zh9h" Jul 18 11:57:49.289301 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:57:49.289284 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-795bc545c7-4zh9h" Jul 18 11:57:49.350851 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:57:49.350819 2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2s2bv\" (UniqueName: \"kubernetes.io/projected/d7761158-d8cc-411f-92d7-e49963ea7fc7-kube-api-access-2s2bv\") on node \"ip-10-0-136-240.ec2.internal\" DevicePath \"\"" Jul 18 11:57:49.420833 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:57:49.420755 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-795bc545c7-4zh9h"] Jul 18 11:57:49.423084 ip-10-0-136-240 kubenswrapper[2585]: W0718 11:57:49.423063 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podff12bf45_bdbb_4252_847c_24de86a67842.slice/crio-5cc980a14b37a54d102e9ab6333f750796573fa3068340a86985b86cdfe4e29f WatchSource:0}: Error finding container 5cc980a14b37a54d102e9ab6333f750796573fa3068340a86985b86cdfe4e29f: Status 404 returned error can't find the container with id 5cc980a14b37a54d102e9ab6333f750796573fa3068340a86985b86cdfe4e29f Jul 18 11:57:50.059923 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:57:50.059836 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-795bc545c7-4zh9h" event={"ID":"ff12bf45-bdbb-4252-847c-24de86a67842","Type":"ContainerStarted","Data":"7b1de34e01909f0b11e0cff264ec04dc820eebfb78ef2b5be46847b035c5f7fc"} Jul 18 11:57:50.059923 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:57:50.059876 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6c4dd6cfb5-gw8tb" Jul 18 11:57:50.060333 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:57:50.059844 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-q2tfj" Jul 18 11:57:50.060333 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:57:50.059874 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-795bc545c7-4zh9h" event={"ID":"ff12bf45-bdbb-4252-847c-24de86a67842","Type":"ContainerStarted","Data":"5cc980a14b37a54d102e9ab6333f750796573fa3068340a86985b86cdfe4e29f"} Jul 18 11:57:50.079450 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:57:50.079396 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-795bc545c7-4zh9h" podStartSLOduration=1.783837454 podStartE2EDuration="2.079381776s" totalImagesPullingTime="295.544322ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-18 11:57:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-18 11:57:50.077676703 +0000 UTC m=+537.157883060" watchObservedRunningTime="2026-07-18 11:57:50.079381776 +0000 UTC m=+537.159588133" Jul 18 11:57:50.106092 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:57:50.106064 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-q2tfj"] Jul 18 11:57:50.110729 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:57:50.110705 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-75d556bd77-pp8x9"] Jul 18 11:57:50.111132 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:57:50.111101 2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-75d556bd77-pp8x9" podUID="fbd1b5c3-a4ee-4e88-80cb-c3980b165cdd" containerName="authorino" containerID="cri-o://1ccdf2bfa1c83479dba6fb900d565314bd39d020b24d77d85915016007827a86" gracePeriod=30 Jul 18 11:57:50.113111 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:57:50.113091 2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-q2tfj"] Jul 18 11:57:50.138957 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:57:50.138932 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6c4dd6cfb5-gw8tb"] Jul 18 11:57:50.141815 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:57:50.141790 2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-6c4dd6cfb5-gw8tb"] Jul 18 11:57:50.362388 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:57:50.362364 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-pp8x9" Jul 18 11:57:50.461055 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:57:50.461021 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-phhng\" (UniqueName: \"kubernetes.io/projected/fbd1b5c3-a4ee-4e88-80cb-c3980b165cdd-kube-api-access-phhng\") pod \"fbd1b5c3-a4ee-4e88-80cb-c3980b165cdd\" (UID: \"fbd1b5c3-a4ee-4e88-80cb-c3980b165cdd\") " Jul 18 11:57:50.463237 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:57:50.463210 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fbd1b5c3-a4ee-4e88-80cb-c3980b165cdd-kube-api-access-phhng" (OuterVolumeSpecName: "kube-api-access-phhng") pod "fbd1b5c3-a4ee-4e88-80cb-c3980b165cdd" (UID: "fbd1b5c3-a4ee-4e88-80cb-c3980b165cdd"). InnerVolumeSpecName "kube-api-access-phhng". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 18 11:57:50.561889 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:57:50.561850 2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-phhng\" (UniqueName: \"kubernetes.io/projected/fbd1b5c3-a4ee-4e88-80cb-c3980b165cdd-kube-api-access-phhng\") on node \"ip-10-0-136-240.ec2.internal\" DevicePath \"\"" Jul 18 11:57:51.065438 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:57:51.065400 2585 generic.go:356] "Generic (PLEG): container finished" podID="fbd1b5c3-a4ee-4e88-80cb-c3980b165cdd" containerID="1ccdf2bfa1c83479dba6fb900d565314bd39d020b24d77d85915016007827a86" exitCode=0 Jul 18 11:57:51.065931 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:57:51.065460 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-pp8x9" Jul 18 11:57:51.065931 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:57:51.065482 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-pp8x9" event={"ID":"fbd1b5c3-a4ee-4e88-80cb-c3980b165cdd","Type":"ContainerDied","Data":"1ccdf2bfa1c83479dba6fb900d565314bd39d020b24d77d85915016007827a86"} Jul 18 11:57:51.065931 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:57:51.065528 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-pp8x9" event={"ID":"fbd1b5c3-a4ee-4e88-80cb-c3980b165cdd","Type":"ContainerDied","Data":"fad060455ae64ef4a95f2d72244de50b2052e0a4b8ea038c4463c4712d81c85f"} Jul 18 11:57:51.065931 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:57:51.065545 2585 scope.go:117] "RemoveContainer" containerID="1ccdf2bfa1c83479dba6fb900d565314bd39d020b24d77d85915016007827a86" Jul 18 11:57:51.074686 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:57:51.074671 2585 scope.go:117] "RemoveContainer" containerID="1ccdf2bfa1c83479dba6fb900d565314bd39d020b24d77d85915016007827a86" Jul 18 11:57:51.074924 ip-10-0-136-240 kubenswrapper[2585]: E0718 11:57:51.074909 2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1ccdf2bfa1c83479dba6fb900d565314bd39d020b24d77d85915016007827a86\": container with ID starting with 1ccdf2bfa1c83479dba6fb900d565314bd39d020b24d77d85915016007827a86 not found: ID does not exist" containerID="1ccdf2bfa1c83479dba6fb900d565314bd39d020b24d77d85915016007827a86" Jul 18 11:57:51.074972 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:57:51.074932 2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1ccdf2bfa1c83479dba6fb900d565314bd39d020b24d77d85915016007827a86"} err="failed to get container status \"1ccdf2bfa1c83479dba6fb900d565314bd39d020b24d77d85915016007827a86\": rpc error: code = NotFound desc = could not find container \"1ccdf2bfa1c83479dba6fb900d565314bd39d020b24d77d85915016007827a86\": container with ID starting with 1ccdf2bfa1c83479dba6fb900d565314bd39d020b24d77d85915016007827a86 not found: ID does not exist" Jul 18 11:57:51.086044 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:57:51.086022 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-75d556bd77-pp8x9"] Jul 18 11:57:51.089720 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:57:51.089702 2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-75d556bd77-pp8x9"] Jul 18 11:57:51.469135 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:57:51.469104 2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="382f681a-44aa-4118-9d1a-d0f31e0df614" path="/var/lib/kubelet/pods/382f681a-44aa-4118-9d1a-d0f31e0df614/volumes" Jul 18 11:57:51.469332 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:57:51.469321 2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d7761158-d8cc-411f-92d7-e49963ea7fc7" path="/var/lib/kubelet/pods/d7761158-d8cc-411f-92d7-e49963ea7fc7/volumes" Jul 18 11:57:51.469515 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:57:51.469504 2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fbd1b5c3-a4ee-4e88-80cb-c3980b165cdd" path="/var/lib/kubelet/pods/fbd1b5c3-a4ee-4e88-80cb-c3980b165cdd/volumes" Jul 18 11:58:40.307920 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:58:40.307889 2585 generic.go:356] "Generic (PLEG): container finished" podID="faf0508a-7db7-4cbe-915a-499c404d0bd0" containerID="8316b5ff9c6ae17d41cf4f5f17d2558744a670f5d8460c0a288dcfc1f558cf5a" exitCode=0 Jul 18 11:58:40.308318 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:58:40.307961 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-2gcb2" event={"ID":"faf0508a-7db7-4cbe-915a-499c404d0bd0","Type":"ContainerDied","Data":"8316b5ff9c6ae17d41cf4f5f17d2558744a670f5d8460c0a288dcfc1f558cf5a"} Jul 18 11:58:40.308369 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:58:40.308353 2585 scope.go:117] "RemoveContainer" containerID="8316b5ff9c6ae17d41cf4f5f17d2558744a670f5d8460c0a288dcfc1f558cf5a" Jul 18 11:58:40.309357 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:58:40.309333 2585 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 18 11:58:40.309412 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:58:40.309396 2585 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 18 11:58:40.309445 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:58:40.309426 2585 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 18 11:58:41.320774 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:58:41.320740 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-2gcb2" event={"ID":"faf0508a-7db7-4cbe-915a-499c404d0bd0","Type":"ContainerStarted","Data":"8b38fd947ad400c3681fccb8621aab3dc3aefb75b37f34b058bdb862fcdfbd97"} Jul 18 11:58:41.386066 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:58:41.386035 2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-2gcb2" Jul 18 11:58:41.386834 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:58:41.386809 2585 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-56654dcf7-2gcb2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused" start-of-body= Jul 18 11:58:41.386892 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:58:41.386872 2585 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-2gcb2" podUID="faf0508a-7db7-4cbe-915a-499c404d0bd0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused" Jul 18 11:58:42.386222 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:58:42.386178 2585 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-56654dcf7-2gcb2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused" start-of-body= Jul 18 11:58:42.386743 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:58:42.386261 2585 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-2gcb2" podUID="faf0508a-7db7-4cbe-915a-499c404d0bd0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused" Jul 18 11:58:43.386602 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:58:43.386569 2585 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-56654dcf7-2gcb2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused" start-of-body= Jul 18 11:58:43.387169 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:58:43.387140 2585 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-2gcb2" podUID="faf0508a-7db7-4cbe-915a-499c404d0bd0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused" Jul 18 11:58:44.386480 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:58:44.386441 2585 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-56654dcf7-2gcb2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused" start-of-body= Jul 18 11:58:44.386703 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:58:44.386523 2585 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-2gcb2" podUID="faf0508a-7db7-4cbe-915a-499c404d0bd0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused" Jul 18 11:58:44.803152 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:58:44.803114 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-2gcb2" Jul 18 11:58:45.385652 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:58:45.385613 2585 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-56654dcf7-2gcb2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused" start-of-body= Jul 18 11:58:45.385958 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:58:45.385928 2585 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-2gcb2" podUID="faf0508a-7db7-4cbe-915a-499c404d0bd0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused" Jul 18 11:58:46.389840 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:58:46.389810 2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-2gcb2" Jul 18 11:58:46.390725 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:58:46.390709 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-2gcb2" Jul 18 11:58:53.385662 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:58:53.385633 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-j9l7p_8ec47eff-5fbc-4130-baef-d8544aa4e786/console-operator/1.log" Jul 18 11:58:53.388387 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:58:53.388368 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-j9l7p_8ec47eff-5fbc-4130-baef-d8544aa4e786/console-operator/1.log" Jul 18 11:58:53.395839 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:58:53.395817 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-wq4lg_247c97c8-982b-4176-ae10-afc396745360/ovn-acl-logging/0.log" Jul 18 11:58:53.398365 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:58:53.398345 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-wq4lg_247c97c8-982b-4176-ae10-afc396745360/ovn-acl-logging/0.log" Jul 18 11:59:09.936678 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:59:09.936637 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-tjmmg"] Jul 18 11:59:09.937488 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:59:09.937204 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fbd1b5c3-a4ee-4e88-80cb-c3980b165cdd" containerName="authorino" Jul 18 11:59:09.937488 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:59:09.937224 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="fbd1b5c3-a4ee-4e88-80cb-c3980b165cdd" containerName="authorino" Jul 18 11:59:09.937488 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:59:09.937333 2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="fbd1b5c3-a4ee-4e88-80cb-c3980b165cdd" containerName="authorino" Jul 18 11:59:09.940608 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:59:09.940588 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-tjmmg" Jul 18 11:59:09.942947 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:59:09.942929 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Jul 18 11:59:09.943828 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:59:09.943810 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Jul 18 11:59:09.943893 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:59:09.943842 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-k9r9z\"" Jul 18 11:59:09.943893 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:59:09.943861 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-simulated-kserve-self-signed-certs\"" Jul 18 11:59:09.955283 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:59:09.955257 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-tjmmg"] Jul 18 11:59:10.028908 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:59:10.028877 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/67ebd32b-5627-48c6-ae45-2db77bd3f9ed-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-tjmmg\" (UID: \"67ebd32b-5627-48c6-ae45-2db77bd3f9ed\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-tjmmg" Jul 18 11:59:10.029043 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:59:10.028940 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/67ebd32b-5627-48c6-ae45-2db77bd3f9ed-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-tjmmg\" (UID: \"67ebd32b-5627-48c6-ae45-2db77bd3f9ed\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-tjmmg" Jul 18 11:59:10.029043 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:59:10.028986 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s8xdk\" (UniqueName: \"kubernetes.io/projected/67ebd32b-5627-48c6-ae45-2db77bd3f9ed-kube-api-access-s8xdk\") pod \"e2e-distinct-simulated-kserve-59975d69f9-tjmmg\" (UID: \"67ebd32b-5627-48c6-ae45-2db77bd3f9ed\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-tjmmg" Jul 18 11:59:10.029043 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:59:10.029016 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/67ebd32b-5627-48c6-ae45-2db77bd3f9ed-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-tjmmg\" (UID: \"67ebd32b-5627-48c6-ae45-2db77bd3f9ed\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-tjmmg" Jul 18 11:59:10.029145 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:59:10.029050 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/67ebd32b-5627-48c6-ae45-2db77bd3f9ed-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-tjmmg\" (UID: \"67ebd32b-5627-48c6-ae45-2db77bd3f9ed\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-tjmmg" Jul 18 11:59:10.029145 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:59:10.029074 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/67ebd32b-5627-48c6-ae45-2db77bd3f9ed-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-tjmmg\" (UID: \"67ebd32b-5627-48c6-ae45-2db77bd3f9ed\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-tjmmg" Jul 18 11:59:10.130173 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:59:10.130127 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/67ebd32b-5627-48c6-ae45-2db77bd3f9ed-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-tjmmg\" (UID: \"67ebd32b-5627-48c6-ae45-2db77bd3f9ed\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-tjmmg" Jul 18 11:59:10.130173 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:59:10.130179 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s8xdk\" (UniqueName: \"kubernetes.io/projected/67ebd32b-5627-48c6-ae45-2db77bd3f9ed-kube-api-access-s8xdk\") pod \"e2e-distinct-simulated-kserve-59975d69f9-tjmmg\" (UID: \"67ebd32b-5627-48c6-ae45-2db77bd3f9ed\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-tjmmg" Jul 18 11:59:10.130377 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:59:10.130206 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/67ebd32b-5627-48c6-ae45-2db77bd3f9ed-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-tjmmg\" (UID: \"67ebd32b-5627-48c6-ae45-2db77bd3f9ed\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-tjmmg" Jul 18 11:59:10.130377 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:59:10.130246 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/67ebd32b-5627-48c6-ae45-2db77bd3f9ed-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-tjmmg\" (UID: \"67ebd32b-5627-48c6-ae45-2db77bd3f9ed\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-tjmmg" Jul 18 11:59:10.130377 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:59:10.130281 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/67ebd32b-5627-48c6-ae45-2db77bd3f9ed-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-tjmmg\" (UID: \"67ebd32b-5627-48c6-ae45-2db77bd3f9ed\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-tjmmg" Jul 18 11:59:10.130377 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:59:10.130341 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/67ebd32b-5627-48c6-ae45-2db77bd3f9ed-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-tjmmg\" (UID: \"67ebd32b-5627-48c6-ae45-2db77bd3f9ed\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-tjmmg" Jul 18 11:59:10.130713 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:59:10.130687 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/67ebd32b-5627-48c6-ae45-2db77bd3f9ed-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-tjmmg\" (UID: \"67ebd32b-5627-48c6-ae45-2db77bd3f9ed\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-tjmmg" Jul 18 11:59:10.130796 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:59:10.130698 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/67ebd32b-5627-48c6-ae45-2db77bd3f9ed-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-tjmmg\" (UID: \"67ebd32b-5627-48c6-ae45-2db77bd3f9ed\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-tjmmg" Jul 18 11:59:10.130796 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:59:10.130768 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/67ebd32b-5627-48c6-ae45-2db77bd3f9ed-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-tjmmg\" (UID: \"67ebd32b-5627-48c6-ae45-2db77bd3f9ed\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-tjmmg" Jul 18 11:59:10.132622 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:59:10.132601 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/67ebd32b-5627-48c6-ae45-2db77bd3f9ed-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-tjmmg\" (UID: \"67ebd32b-5627-48c6-ae45-2db77bd3f9ed\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-tjmmg" Jul 18 11:59:10.132908 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:59:10.132886 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/67ebd32b-5627-48c6-ae45-2db77bd3f9ed-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-tjmmg\" (UID: \"67ebd32b-5627-48c6-ae45-2db77bd3f9ed\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-tjmmg" Jul 18 11:59:10.137904 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:59:10.137882 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s8xdk\" (UniqueName: \"kubernetes.io/projected/67ebd32b-5627-48c6-ae45-2db77bd3f9ed-kube-api-access-s8xdk\") pod \"e2e-distinct-simulated-kserve-59975d69f9-tjmmg\" (UID: \"67ebd32b-5627-48c6-ae45-2db77bd3f9ed\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-tjmmg" Jul 18 11:59:10.256149 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:59:10.256116 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-tjmmg" Jul 18 11:59:10.396691 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:59:10.396653 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-tjmmg"] Jul 18 11:59:10.399109 ip-10-0-136-240 kubenswrapper[2585]: W0718 11:59:10.399079 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod67ebd32b_5627_48c6_ae45_2db77bd3f9ed.slice/crio-eabd84515e9e5afbcbb3bc90a02a149b70ebbd68a615707e19059f8b6e0c0ef2 WatchSource:0}: Error finding container eabd84515e9e5afbcbb3bc90a02a149b70ebbd68a615707e19059f8b6e0c0ef2: Status 404 returned error can't find the container with id eabd84515e9e5afbcbb3bc90a02a149b70ebbd68a615707e19059f8b6e0c0ef2 Jul 18 11:59:10.400416 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:59:10.400394 2585 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 18 11:59:10.461135 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:59:10.461092 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-tjmmg" event={"ID":"67ebd32b-5627-48c6-ae45-2db77bd3f9ed","Type":"ContainerStarted","Data":"eabd84515e9e5afbcbb3bc90a02a149b70ebbd68a615707e19059f8b6e0c0ef2"} Jul 18 11:59:12.473174 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:59:12.473138 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-tjmmg" event={"ID":"67ebd32b-5627-48c6-ae45-2db77bd3f9ed","Type":"ContainerStarted","Data":"7bb2ad67ba4f02d2e724e8f4d977712973db1ab57fa26f010558e275738bbb50"} Jul 18 11:59:12.491235 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:59:12.491184 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-tjmmg" podStartSLOduration=2.385666801 podStartE2EDuration="3.491169208s" totalImagesPullingTime="1.105502407s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-18 11:59:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-18 11:59:12.489256024 +0000 UTC m=+619.569462380" watchObservedRunningTime="2026-07-18 11:59:12.491169208 +0000 UTC m=+619.571375565" Jul 18 11:59:20.257367 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:59:20.257290 2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-tjmmg" Jul 18 11:59:20.257367 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:59:20.257350 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-tjmmg" Jul 18 11:59:20.269896 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:59:20.269872 2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-tjmmg" Jul 18 11:59:20.519847 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:59:20.519763 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-tjmmg" Jul 18 11:59:36.846165 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:59:36.846126 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-6lbk7"] Jul 18 11:59:36.849639 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:59:36.849618 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-6lbk7" Jul 18 11:59:36.851980 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:59:36.851960 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"facebook-opt-125m-simulated-kserve-self-signed-certs\"" Jul 18 11:59:36.859378 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:59:36.859353 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-6lbk7"] Jul 18 11:59:36.966095 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:59:36.966056 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3daa0fca-c6f0-4020-9eb2-cb5db4393ac0-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-6lbk7\" (UID: \"3daa0fca-c6f0-4020-9eb2-cb5db4393ac0\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-6lbk7" Jul 18 11:59:36.966095 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:59:36.966101 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3daa0fca-c6f0-4020-9eb2-cb5db4393ac0-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-6lbk7\" (UID: \"3daa0fca-c6f0-4020-9eb2-cb5db4393ac0\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-6lbk7" Jul 18 11:59:36.966307 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:59:36.966121 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3daa0fca-c6f0-4020-9eb2-cb5db4393ac0-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-6lbk7\" (UID: \"3daa0fca-c6f0-4020-9eb2-cb5db4393ac0\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-6lbk7" Jul 18 11:59:36.966307 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:59:36.966166 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3daa0fca-c6f0-4020-9eb2-cb5db4393ac0-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-6lbk7\" (UID: \"3daa0fca-c6f0-4020-9eb2-cb5db4393ac0\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-6lbk7" Jul 18 11:59:36.966307 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:59:36.966183 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3daa0fca-c6f0-4020-9eb2-cb5db4393ac0-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-6lbk7\" (UID: \"3daa0fca-c6f0-4020-9eb2-cb5db4393ac0\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-6lbk7" Jul 18 11:59:36.966307 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:59:36.966216 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5778j\" (UniqueName: \"kubernetes.io/projected/3daa0fca-c6f0-4020-9eb2-cb5db4393ac0-kube-api-access-5778j\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-6lbk7\" (UID: \"3daa0fca-c6f0-4020-9eb2-cb5db4393ac0\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-6lbk7" Jul 18 11:59:37.067225 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:59:37.067175 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3daa0fca-c6f0-4020-9eb2-cb5db4393ac0-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-6lbk7\" (UID: \"3daa0fca-c6f0-4020-9eb2-cb5db4393ac0\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-6lbk7" Jul 18 11:59:37.067225 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:59:37.067225 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3daa0fca-c6f0-4020-9eb2-cb5db4393ac0-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-6lbk7\" (UID: \"3daa0fca-c6f0-4020-9eb2-cb5db4393ac0\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-6lbk7" Jul 18 11:59:37.067453 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:59:37.067262 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5778j\" (UniqueName: \"kubernetes.io/projected/3daa0fca-c6f0-4020-9eb2-cb5db4393ac0-kube-api-access-5778j\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-6lbk7\" (UID: \"3daa0fca-c6f0-4020-9eb2-cb5db4393ac0\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-6lbk7" Jul 18 11:59:37.067453 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:59:37.067333 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3daa0fca-c6f0-4020-9eb2-cb5db4393ac0-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-6lbk7\" (UID: \"3daa0fca-c6f0-4020-9eb2-cb5db4393ac0\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-6lbk7" Jul 18 11:59:37.067453 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:59:37.067348 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3daa0fca-c6f0-4020-9eb2-cb5db4393ac0-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-6lbk7\" (UID: \"3daa0fca-c6f0-4020-9eb2-cb5db4393ac0\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-6lbk7" Jul 18 11:59:37.067453 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:59:37.067366 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3daa0fca-c6f0-4020-9eb2-cb5db4393ac0-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-6lbk7\" (UID: \"3daa0fca-c6f0-4020-9eb2-cb5db4393ac0\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-6lbk7" Jul 18 11:59:37.067678 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:59:37.067625 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3daa0fca-c6f0-4020-9eb2-cb5db4393ac0-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-6lbk7\" (UID: \"3daa0fca-c6f0-4020-9eb2-cb5db4393ac0\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-6lbk7" Jul 18 11:59:37.067737 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:59:37.067692 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3daa0fca-c6f0-4020-9eb2-cb5db4393ac0-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-6lbk7\" (UID: \"3daa0fca-c6f0-4020-9eb2-cb5db4393ac0\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-6lbk7" Jul 18 11:59:37.067737 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:59:37.067707 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3daa0fca-c6f0-4020-9eb2-cb5db4393ac0-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-6lbk7\" (UID: \"3daa0fca-c6f0-4020-9eb2-cb5db4393ac0\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-6lbk7" Jul 18 11:59:37.069769 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:59:37.069744 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3daa0fca-c6f0-4020-9eb2-cb5db4393ac0-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-6lbk7\" (UID: \"3daa0fca-c6f0-4020-9eb2-cb5db4393ac0\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-6lbk7" Jul 18 11:59:37.070053 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:59:37.070036 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3daa0fca-c6f0-4020-9eb2-cb5db4393ac0-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-6lbk7\" (UID: \"3daa0fca-c6f0-4020-9eb2-cb5db4393ac0\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-6lbk7" Jul 18 11:59:37.075333 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:59:37.075305 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5778j\" (UniqueName: \"kubernetes.io/projected/3daa0fca-c6f0-4020-9eb2-cb5db4393ac0-kube-api-access-5778j\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-6lbk7\" (UID: \"3daa0fca-c6f0-4020-9eb2-cb5db4393ac0\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-6lbk7" Jul 18 11:59:37.160327 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:59:37.160268 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-6lbk7" Jul 18 11:59:37.292883 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:59:37.292854 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-6lbk7"] Jul 18 11:59:37.296307 ip-10-0-136-240 kubenswrapper[2585]: W0718 11:59:37.296283 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3daa0fca_c6f0_4020_9eb2_cb5db4393ac0.slice/crio-3e45e6bbb84c780d54ea4571ca0ce570a99f624fff2af5bf27e8c7edb46c8f55 WatchSource:0}: Error finding container 3e45e6bbb84c780d54ea4571ca0ce570a99f624fff2af5bf27e8c7edb46c8f55: Status 404 returned error can't find the container with id 3e45e6bbb84c780d54ea4571ca0ce570a99f624fff2af5bf27e8c7edb46c8f55 Jul 18 11:59:37.589157 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:59:37.589111 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-6lbk7" event={"ID":"3daa0fca-c6f0-4020-9eb2-cb5db4393ac0","Type":"ContainerStarted","Data":"a83e1c55912c83d7cc7ab4c4a59b28e228e937b6c80a7a1f71c4c6e5e380b371"} Jul 18 11:59:37.589338 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:59:37.589166 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-6lbk7" event={"ID":"3daa0fca-c6f0-4020-9eb2-cb5db4393ac0","Type":"ContainerStarted","Data":"3e45e6bbb84c780d54ea4571ca0ce570a99f624fff2af5bf27e8c7edb46c8f55"} Jul 18 11:59:37.607156 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:59:37.607111 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-6lbk7" podStartSLOduration=1.441244797 podStartE2EDuration="1.607095518s" totalImagesPullingTime="165.850721ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-18 11:59:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-18 11:59:37.604975358 +0000 UTC m=+644.685181712" watchObservedRunningTime="2026-07-18 11:59:37.607095518 +0000 UTC m=+644.687301874" Jul 18 11:59:45.733206 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:59:45.733169 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-mjjj4"] Jul 18 11:59:45.737004 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:59:45.736985 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-mjjj4" Jul 18 11:59:45.739378 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:59:45.739355 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\"" Jul 18 11:59:45.747235 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:59:45.747213 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-mjjj4"] Jul 18 11:59:45.846389 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:59:45.846361 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/136cde5b-95d7-480c-b712-3a6233f43e5d-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-mjjj4\" (UID: \"136cde5b-95d7-480c-b712-3a6233f43e5d\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-mjjj4" Jul 18 11:59:45.846530 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:59:45.846399 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/136cde5b-95d7-480c-b712-3a6233f43e5d-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-mjjj4\" (UID: \"136cde5b-95d7-480c-b712-3a6233f43e5d\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-mjjj4" Jul 18 11:59:45.846530 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:59:45.846420 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/136cde5b-95d7-480c-b712-3a6233f43e5d-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-mjjj4\" (UID: \"136cde5b-95d7-480c-b712-3a6233f43e5d\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-mjjj4" Jul 18 11:59:45.846641 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:59:45.846576 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/136cde5b-95d7-480c-b712-3a6233f43e5d-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-mjjj4\" (UID: \"136cde5b-95d7-480c-b712-3a6233f43e5d\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-mjjj4" Jul 18 11:59:45.846641 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:59:45.846622 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-trcnk\" (UniqueName: \"kubernetes.io/projected/136cde5b-95d7-480c-b712-3a6233f43e5d-kube-api-access-trcnk\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-mjjj4\" (UID: \"136cde5b-95d7-480c-b712-3a6233f43e5d\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-mjjj4" Jul 18 11:59:45.846726 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:59:45.846668 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/136cde5b-95d7-480c-b712-3a6233f43e5d-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-mjjj4\" (UID: \"136cde5b-95d7-480c-b712-3a6233f43e5d\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-mjjj4" Jul 18 11:59:45.947468 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:59:45.947440 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/136cde5b-95d7-480c-b712-3a6233f43e5d-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-mjjj4\" (UID: \"136cde5b-95d7-480c-b712-3a6233f43e5d\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-mjjj4" Jul 18 11:59:45.947654 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:59:45.947507 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/136cde5b-95d7-480c-b712-3a6233f43e5d-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-mjjj4\" (UID: \"136cde5b-95d7-480c-b712-3a6233f43e5d\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-mjjj4" Jul 18 11:59:45.947654 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:59:45.947561 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/136cde5b-95d7-480c-b712-3a6233f43e5d-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-mjjj4\" (UID: \"136cde5b-95d7-480c-b712-3a6233f43e5d\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-mjjj4" Jul 18 11:59:45.947654 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:59:45.947617 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/136cde5b-95d7-480c-b712-3a6233f43e5d-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-mjjj4\" (UID: \"136cde5b-95d7-480c-b712-3a6233f43e5d\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-mjjj4" Jul 18 11:59:45.947827 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:59:45.947685 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/136cde5b-95d7-480c-b712-3a6233f43e5d-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-mjjj4\" (UID: \"136cde5b-95d7-480c-b712-3a6233f43e5d\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-mjjj4" Jul 18 11:59:45.947827 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:59:45.947721 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-trcnk\" (UniqueName: \"kubernetes.io/projected/136cde5b-95d7-480c-b712-3a6233f43e5d-kube-api-access-trcnk\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-mjjj4\" (UID: \"136cde5b-95d7-480c-b712-3a6233f43e5d\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-mjjj4" Jul 18 11:59:45.947923 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:59:45.947901 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/136cde5b-95d7-480c-b712-3a6233f43e5d-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-mjjj4\" (UID: \"136cde5b-95d7-480c-b712-3a6233f43e5d\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-mjjj4" Jul 18 11:59:45.947977 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:59:45.947947 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/136cde5b-95d7-480c-b712-3a6233f43e5d-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-mjjj4\" (UID: \"136cde5b-95d7-480c-b712-3a6233f43e5d\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-mjjj4" Jul 18 11:59:45.948124 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:59:45.948105 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/136cde5b-95d7-480c-b712-3a6233f43e5d-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-mjjj4\" (UID: \"136cde5b-95d7-480c-b712-3a6233f43e5d\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-mjjj4" Jul 18 11:59:45.949954 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:59:45.949931 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/136cde5b-95d7-480c-b712-3a6233f43e5d-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-mjjj4\" (UID: \"136cde5b-95d7-480c-b712-3a6233f43e5d\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-mjjj4" Jul 18 11:59:45.950339 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:59:45.950320 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/136cde5b-95d7-480c-b712-3a6233f43e5d-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-mjjj4\" (UID: \"136cde5b-95d7-480c-b712-3a6233f43e5d\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-mjjj4" Jul 18 11:59:45.955643 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:59:45.955622 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-trcnk\" (UniqueName: \"kubernetes.io/projected/136cde5b-95d7-480c-b712-3a6233f43e5d-kube-api-access-trcnk\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-mjjj4\" (UID: \"136cde5b-95d7-480c-b712-3a6233f43e5d\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-mjjj4" Jul 18 11:59:46.048206 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:59:46.048140 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-mjjj4" Jul 18 11:59:46.180235 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:59:46.180210 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-mjjj4"] Jul 18 11:59:46.182819 ip-10-0-136-240 kubenswrapper[2585]: W0718 11:59:46.182788 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod136cde5b_95d7_480c_b712_3a6233f43e5d.slice/crio-d8e4b4da478985b89dd92e8a8446b341f4334b626c676939d25c7dc39fa4ff5e WatchSource:0}: Error finding container d8e4b4da478985b89dd92e8a8446b341f4334b626c676939d25c7dc39fa4ff5e: Status 404 returned error can't find the container with id d8e4b4da478985b89dd92e8a8446b341f4334b626c676939d25c7dc39fa4ff5e Jul 18 11:59:46.638337 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:59:46.638247 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-mjjj4" event={"ID":"136cde5b-95d7-480c-b712-3a6233f43e5d","Type":"ContainerStarted","Data":"a64ebb41aadb46314c26ecec361afe9722ba4b8d728162b94705038da283fc72"} Jul 18 11:59:46.638337 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:59:46.638292 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-mjjj4" event={"ID":"136cde5b-95d7-480c-b712-3a6233f43e5d","Type":"ContainerStarted","Data":"d8e4b4da478985b89dd92e8a8446b341f4334b626c676939d25c7dc39fa4ff5e"} Jul 18 11:59:46.655932 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:59:46.655883 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-mjjj4" podStartSLOduration=1.492866117 podStartE2EDuration="1.655870231s" totalImagesPullingTime="163.004114ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-18 11:59:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-18 11:59:46.653757944 +0000 UTC m=+653.733964302" watchObservedRunningTime="2026-07-18 11:59:46.655870231 +0000 UTC m=+653.736076588" Jul 18 11:59:47.160836 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:59:47.160805 2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-6lbk7" Jul 18 11:59:47.161188 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:59:47.160854 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-6lbk7" Jul 18 11:59:47.173148 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:59:47.173125 2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-6lbk7" Jul 18 11:59:47.654236 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:59:47.654208 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-6lbk7" Jul 18 11:59:56.049160 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:59:56.049124 2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-mjjj4" Jul 18 11:59:56.049160 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:59:56.049164 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-mjjj4" Jul 18 11:59:56.061544 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:59:56.061519 2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-mjjj4" Jul 18 11:59:56.704281 ip-10-0-136-240 kubenswrapper[2585]: I0718 11:59:56.704249 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-mjjj4" Jul 18 12:01:19.190769 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:01:19.190680 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-795bc545c7-4zh9h"] Jul 18 12:01:19.191600 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:01:19.191524 2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-795bc545c7-4zh9h" podUID="ff12bf45-bdbb-4252-847c-24de86a67842" containerName="authorino" containerID="cri-o://7b1de34e01909f0b11e0cff264ec04dc820eebfb78ef2b5be46847b035c5f7fc" gracePeriod=30 Jul 18 12:01:19.445916 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:01:19.445859 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-795bc545c7-4zh9h" Jul 18 12:01:19.598649 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:01:19.598625 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/ff12bf45-bdbb-4252-847c-24de86a67842-tls-cert\") pod \"ff12bf45-bdbb-4252-847c-24de86a67842\" (UID: \"ff12bf45-bdbb-4252-847c-24de86a67842\") " Jul 18 12:01:19.598649 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:01:19.598653 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jlsb7\" (UniqueName: \"kubernetes.io/projected/ff12bf45-bdbb-4252-847c-24de86a67842-kube-api-access-jlsb7\") pod \"ff12bf45-bdbb-4252-847c-24de86a67842\" (UID: \"ff12bf45-bdbb-4252-847c-24de86a67842\") " Jul 18 12:01:19.600876 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:01:19.600842 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ff12bf45-bdbb-4252-847c-24de86a67842-kube-api-access-jlsb7" (OuterVolumeSpecName: "kube-api-access-jlsb7") pod "ff12bf45-bdbb-4252-847c-24de86a67842" (UID: "ff12bf45-bdbb-4252-847c-24de86a67842"). InnerVolumeSpecName "kube-api-access-jlsb7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 18 12:01:19.609620 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:01:19.609522 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ff12bf45-bdbb-4252-847c-24de86a67842-tls-cert" (OuterVolumeSpecName: "tls-cert") pod "ff12bf45-bdbb-4252-847c-24de86a67842" (UID: "ff12bf45-bdbb-4252-847c-24de86a67842"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 18 12:01:19.699418 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:01:19.699357 2585 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/ff12bf45-bdbb-4252-847c-24de86a67842-tls-cert\") on node \"ip-10-0-136-240.ec2.internal\" DevicePath \"\"" Jul 18 12:01:19.699418 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:01:19.699384 2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jlsb7\" (UniqueName: \"kubernetes.io/projected/ff12bf45-bdbb-4252-847c-24de86a67842-kube-api-access-jlsb7\") on node \"ip-10-0-136-240.ec2.internal\" DevicePath \"\"" Jul 18 12:01:20.097080 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:01:20.097044 2585 generic.go:356] "Generic (PLEG): container finished" podID="ff12bf45-bdbb-4252-847c-24de86a67842" containerID="7b1de34e01909f0b11e0cff264ec04dc820eebfb78ef2b5be46847b035c5f7fc" exitCode=0 Jul 18 12:01:20.097258 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:01:20.097102 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-795bc545c7-4zh9h" Jul 18 12:01:20.097258 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:01:20.097133 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-795bc545c7-4zh9h" event={"ID":"ff12bf45-bdbb-4252-847c-24de86a67842","Type":"ContainerDied","Data":"7b1de34e01909f0b11e0cff264ec04dc820eebfb78ef2b5be46847b035c5f7fc"} Jul 18 12:01:20.097258 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:01:20.097180 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-795bc545c7-4zh9h" event={"ID":"ff12bf45-bdbb-4252-847c-24de86a67842","Type":"ContainerDied","Data":"5cc980a14b37a54d102e9ab6333f750796573fa3068340a86985b86cdfe4e29f"} Jul 18 12:01:20.097258 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:01:20.097197 2585 scope.go:117] "RemoveContainer" containerID="7b1de34e01909f0b11e0cff264ec04dc820eebfb78ef2b5be46847b035c5f7fc" Jul 18 12:01:20.106722 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:01:20.106706 2585 scope.go:117] "RemoveContainer" containerID="7b1de34e01909f0b11e0cff264ec04dc820eebfb78ef2b5be46847b035c5f7fc" Jul 18 12:01:20.106976 ip-10-0-136-240 kubenswrapper[2585]: E0718 12:01:20.106959 2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7b1de34e01909f0b11e0cff264ec04dc820eebfb78ef2b5be46847b035c5f7fc\": container with ID starting with 7b1de34e01909f0b11e0cff264ec04dc820eebfb78ef2b5be46847b035c5f7fc not found: ID does not exist" containerID="7b1de34e01909f0b11e0cff264ec04dc820eebfb78ef2b5be46847b035c5f7fc" Jul 18 12:01:20.107024 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:01:20.106986 2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7b1de34e01909f0b11e0cff264ec04dc820eebfb78ef2b5be46847b035c5f7fc"} err="failed to get container status \"7b1de34e01909f0b11e0cff264ec04dc820eebfb78ef2b5be46847b035c5f7fc\": rpc error: code = NotFound desc = could not find container \"7b1de34e01909f0b11e0cff264ec04dc820eebfb78ef2b5be46847b035c5f7fc\": container with ID starting with 7b1de34e01909f0b11e0cff264ec04dc820eebfb78ef2b5be46847b035c5f7fc not found: ID does not exist" Jul 18 12:01:20.118390 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:01:20.118370 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-795bc545c7-4zh9h"] Jul 18 12:01:20.121990 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:01:20.121968 2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-795bc545c7-4zh9h"] Jul 18 12:01:21.469036 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:01:21.469002 2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ff12bf45-bdbb-4252-847c-24de86a67842" path="/var/lib/kubelet/pods/ff12bf45-bdbb-4252-847c-24de86a67842/volumes" Jul 18 12:03:53.420818 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:03:53.420747 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-j9l7p_8ec47eff-5fbc-4130-baef-d8544aa4e786/console-operator/1.log" Jul 18 12:03:53.423287 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:03:53.420829 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-j9l7p_8ec47eff-5fbc-4130-baef-d8544aa4e786/console-operator/1.log" Jul 18 12:03:53.428293 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:03:53.428269 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-wq4lg_247c97c8-982b-4176-ae10-afc396745360/ovn-acl-logging/0.log" Jul 18 12:03:53.428431 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:03:53.428420 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-wq4lg_247c97c8-982b-4176-ae10-afc396745360/ovn-acl-logging/0.log" Jul 18 12:08:53.453749 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:08:53.453638 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-j9l7p_8ec47eff-5fbc-4130-baef-d8544aa4e786/console-operator/1.log" Jul 18 12:08:53.457611 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:08:53.454706 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-j9l7p_8ec47eff-5fbc-4130-baef-d8544aa4e786/console-operator/1.log" Jul 18 12:08:53.460337 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:08:53.460318 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-wq4lg_247c97c8-982b-4176-ae10-afc396745360/ovn-acl-logging/0.log" Jul 18 12:08:53.461406 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:08:53.461387 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-wq4lg_247c97c8-982b-4176-ae10-afc396745360/ovn-acl-logging/0.log" Jul 18 12:11:51.952134 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:11:51.952098 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-a-83a1304a-kserve-85dbcbc57f-lfmkk"] Jul 18 12:11:51.952638 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:11:51.952562 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ff12bf45-bdbb-4252-847c-24de86a67842" containerName="authorino" Jul 18 12:11:51.952638 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:11:51.952580 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="ff12bf45-bdbb-4252-847c-24de86a67842" containerName="authorino" Jul 18 12:11:51.952729 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:11:51.952656 2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="ff12bf45-bdbb-4252-847c-24de86a67842" containerName="authorino" Jul 18 12:11:51.955525 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:11:51.955507 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-83a1304a-kserve-85dbcbc57f-lfmkk" Jul 18 12:11:51.957968 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:11:51.957947 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-a-83a1304a-kserve-self-signed-certs\"" Jul 18 12:11:51.972919 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:11:51.972897 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-83a1304a-kserve-85dbcbc57f-lfmkk"] Jul 18 12:11:52.059851 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:11:52.059820 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d628c373-734b-4576-b11c-68c664ac7aed-home\") pod \"e2e-same-id-a-83a1304a-kserve-85dbcbc57f-lfmkk\" (UID: \"d628c373-734b-4576-b11c-68c664ac7aed\") " pod="llm/e2e-same-id-a-83a1304a-kserve-85dbcbc57f-lfmkk" Jul 18 12:11:52.060008 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:11:52.059857 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d628c373-734b-4576-b11c-68c664ac7aed-dshm\") pod \"e2e-same-id-a-83a1304a-kserve-85dbcbc57f-lfmkk\" (UID: \"d628c373-734b-4576-b11c-68c664ac7aed\") " pod="llm/e2e-same-id-a-83a1304a-kserve-85dbcbc57f-lfmkk" Jul 18 12:11:52.060008 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:11:52.059880 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d628c373-734b-4576-b11c-68c664ac7aed-tls-certs\") pod \"e2e-same-id-a-83a1304a-kserve-85dbcbc57f-lfmkk\" (UID: \"d628c373-734b-4576-b11c-68c664ac7aed\") " pod="llm/e2e-same-id-a-83a1304a-kserve-85dbcbc57f-lfmkk" Jul 18 12:11:52.060008 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:11:52.059987 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d628c373-734b-4576-b11c-68c664ac7aed-tmp-dir\") pod \"e2e-same-id-a-83a1304a-kserve-85dbcbc57f-lfmkk\" (UID: \"d628c373-734b-4576-b11c-68c664ac7aed\") " pod="llm/e2e-same-id-a-83a1304a-kserve-85dbcbc57f-lfmkk" Jul 18 12:11:52.060008 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:11:52.060008 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d628c373-734b-4576-b11c-68c664ac7aed-model-cache\") pod \"e2e-same-id-a-83a1304a-kserve-85dbcbc57f-lfmkk\" (UID: \"d628c373-734b-4576-b11c-68c664ac7aed\") " pod="llm/e2e-same-id-a-83a1304a-kserve-85dbcbc57f-lfmkk" Jul 18 12:11:52.060179 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:11:52.060054 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wt8nk\" (UniqueName: \"kubernetes.io/projected/d628c373-734b-4576-b11c-68c664ac7aed-kube-api-access-wt8nk\") pod \"e2e-same-id-a-83a1304a-kserve-85dbcbc57f-lfmkk\" (UID: \"d628c373-734b-4576-b11c-68c664ac7aed\") " pod="llm/e2e-same-id-a-83a1304a-kserve-85dbcbc57f-lfmkk" Jul 18 12:11:52.161577 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:11:52.161518 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d628c373-734b-4576-b11c-68c664ac7aed-tmp-dir\") pod \"e2e-same-id-a-83a1304a-kserve-85dbcbc57f-lfmkk\" (UID: \"d628c373-734b-4576-b11c-68c664ac7aed\") " pod="llm/e2e-same-id-a-83a1304a-kserve-85dbcbc57f-lfmkk" Jul 18 12:11:52.161746 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:11:52.161585 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d628c373-734b-4576-b11c-68c664ac7aed-model-cache\") pod \"e2e-same-id-a-83a1304a-kserve-85dbcbc57f-lfmkk\" (UID: \"d628c373-734b-4576-b11c-68c664ac7aed\") " pod="llm/e2e-same-id-a-83a1304a-kserve-85dbcbc57f-lfmkk" Jul 18 12:11:52.161746 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:11:52.161627 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wt8nk\" (UniqueName: \"kubernetes.io/projected/d628c373-734b-4576-b11c-68c664ac7aed-kube-api-access-wt8nk\") pod \"e2e-same-id-a-83a1304a-kserve-85dbcbc57f-lfmkk\" (UID: \"d628c373-734b-4576-b11c-68c664ac7aed\") " pod="llm/e2e-same-id-a-83a1304a-kserve-85dbcbc57f-lfmkk" Jul 18 12:11:52.161746 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:11:52.161705 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d628c373-734b-4576-b11c-68c664ac7aed-home\") pod \"e2e-same-id-a-83a1304a-kserve-85dbcbc57f-lfmkk\" (UID: \"d628c373-734b-4576-b11c-68c664ac7aed\") " pod="llm/e2e-same-id-a-83a1304a-kserve-85dbcbc57f-lfmkk" Jul 18 12:11:52.161746 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:11:52.161742 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d628c373-734b-4576-b11c-68c664ac7aed-dshm\") pod \"e2e-same-id-a-83a1304a-kserve-85dbcbc57f-lfmkk\" (UID: \"d628c373-734b-4576-b11c-68c664ac7aed\") " pod="llm/e2e-same-id-a-83a1304a-kserve-85dbcbc57f-lfmkk" Jul 18 12:11:52.161968 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:11:52.161773 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d628c373-734b-4576-b11c-68c664ac7aed-tls-certs\") pod \"e2e-same-id-a-83a1304a-kserve-85dbcbc57f-lfmkk\" (UID: \"d628c373-734b-4576-b11c-68c664ac7aed\") " pod="llm/e2e-same-id-a-83a1304a-kserve-85dbcbc57f-lfmkk" Jul 18 12:11:52.161968 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:11:52.161958 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d628c373-734b-4576-b11c-68c664ac7aed-tmp-dir\") pod \"e2e-same-id-a-83a1304a-kserve-85dbcbc57f-lfmkk\" (UID: \"d628c373-734b-4576-b11c-68c664ac7aed\") " pod="llm/e2e-same-id-a-83a1304a-kserve-85dbcbc57f-lfmkk" Jul 18 12:11:52.162077 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:11:52.161989 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d628c373-734b-4576-b11c-68c664ac7aed-model-cache\") pod \"e2e-same-id-a-83a1304a-kserve-85dbcbc57f-lfmkk\" (UID: \"d628c373-734b-4576-b11c-68c664ac7aed\") " pod="llm/e2e-same-id-a-83a1304a-kserve-85dbcbc57f-lfmkk" Jul 18 12:11:52.162077 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:11:52.162067 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d628c373-734b-4576-b11c-68c664ac7aed-home\") pod \"e2e-same-id-a-83a1304a-kserve-85dbcbc57f-lfmkk\" (UID: \"d628c373-734b-4576-b11c-68c664ac7aed\") " pod="llm/e2e-same-id-a-83a1304a-kserve-85dbcbc57f-lfmkk" Jul 18 12:11:52.163976 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:11:52.163950 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d628c373-734b-4576-b11c-68c664ac7aed-dshm\") pod \"e2e-same-id-a-83a1304a-kserve-85dbcbc57f-lfmkk\" (UID: \"d628c373-734b-4576-b11c-68c664ac7aed\") " pod="llm/e2e-same-id-a-83a1304a-kserve-85dbcbc57f-lfmkk" Jul 18 12:11:52.164294 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:11:52.164278 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d628c373-734b-4576-b11c-68c664ac7aed-tls-certs\") pod \"e2e-same-id-a-83a1304a-kserve-85dbcbc57f-lfmkk\" (UID: \"d628c373-734b-4576-b11c-68c664ac7aed\") " pod="llm/e2e-same-id-a-83a1304a-kserve-85dbcbc57f-lfmkk" Jul 18 12:11:52.169335 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:11:52.169316 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wt8nk\" (UniqueName: \"kubernetes.io/projected/d628c373-734b-4576-b11c-68c664ac7aed-kube-api-access-wt8nk\") pod \"e2e-same-id-a-83a1304a-kserve-85dbcbc57f-lfmkk\" (UID: \"d628c373-734b-4576-b11c-68c664ac7aed\") " pod="llm/e2e-same-id-a-83a1304a-kserve-85dbcbc57f-lfmkk" Jul 18 12:11:52.264563 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:11:52.264525 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-83a1304a-kserve-85dbcbc57f-lfmkk" Jul 18 12:11:52.398333 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:11:52.398301 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-83a1304a-kserve-85dbcbc57f-lfmkk"] Jul 18 12:11:52.400077 ip-10-0-136-240 kubenswrapper[2585]: W0718 12:11:52.400039 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd628c373_734b_4576_b11c_68c664ac7aed.slice/crio-ffaa352e6b9306e793aff986550ac9a8992fdc15e68cd36a4a8951f2630afa90 WatchSource:0}: Error finding container ffaa352e6b9306e793aff986550ac9a8992fdc15e68cd36a4a8951f2630afa90: Status 404 returned error can't find the container with id ffaa352e6b9306e793aff986550ac9a8992fdc15e68cd36a4a8951f2630afa90 Jul 18 12:11:53.193214 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:11:53.193182 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-83a1304a-kserve-85dbcbc57f-lfmkk" event={"ID":"d628c373-734b-4576-b11c-68c664ac7aed","Type":"ContainerStarted","Data":"c09d6be4a3f8f4ddbd202d2bf5f5433f6292afb7ed7caf9a0bb0e66789ee71e5"} Jul 18 12:11:53.193585 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:11:53.193221 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-83a1304a-kserve-85dbcbc57f-lfmkk" event={"ID":"d628c373-734b-4576-b11c-68c664ac7aed","Type":"ContainerStarted","Data":"ffaa352e6b9306e793aff986550ac9a8992fdc15e68cd36a4a8951f2630afa90"} Jul 18 12:11:53.211406 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:11:53.211362 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-same-id-a-83a1304a-kserve-85dbcbc57f-lfmkk" podStartSLOduration=2.21134905 podStartE2EDuration="2.21134905s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-18 12:11:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-18 12:11:53.209635377 +0000 UTC m=+1380.289841733" watchObservedRunningTime="2026-07-18 12:11:53.21134905 +0000 UTC m=+1380.291555406" Jul 18 12:12:02.265667 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:12:02.265581 2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-a-83a1304a-kserve-85dbcbc57f-lfmkk" Jul 18 12:12:02.265667 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:12:02.265633 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-a-83a1304a-kserve-85dbcbc57f-lfmkk" Jul 18 12:12:02.277867 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:12:02.277843 2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-a-83a1304a-kserve-85dbcbc57f-lfmkk" Jul 18 12:12:02.335400 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:12:02.335361 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-b-83a1304a-kserve-5f5c8f6dc5-d294t"] Jul 18 12:12:02.339386 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:12:02.339359 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-83a1304a-kserve-5f5c8f6dc5-d294t" Jul 18 12:12:02.341820 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:12:02.341801 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-b-83a1304a-kserve-self-signed-certs\"" Jul 18 12:12:02.351405 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:12:02.351380 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-83a1304a-kserve-5f5c8f6dc5-d294t"] Jul 18 12:12:02.450638 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:12:02.450604 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3a14f229-f9d6-4060-b3a9-380f80633a9b-tmp-dir\") pod \"e2e-same-id-b-83a1304a-kserve-5f5c8f6dc5-d294t\" (UID: \"3a14f229-f9d6-4060-b3a9-380f80633a9b\") " pod="llm/e2e-same-id-b-83a1304a-kserve-5f5c8f6dc5-d294t" Jul 18 12:12:02.450799 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:12:02.450668 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qgcrh\" (UniqueName: \"kubernetes.io/projected/3a14f229-f9d6-4060-b3a9-380f80633a9b-kube-api-access-qgcrh\") pod \"e2e-same-id-b-83a1304a-kserve-5f5c8f6dc5-d294t\" (UID: \"3a14f229-f9d6-4060-b3a9-380f80633a9b\") " pod="llm/e2e-same-id-b-83a1304a-kserve-5f5c8f6dc5-d294t" Jul 18 12:12:02.450799 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:12:02.450732 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3a14f229-f9d6-4060-b3a9-380f80633a9b-tls-certs\") pod \"e2e-same-id-b-83a1304a-kserve-5f5c8f6dc5-d294t\" (UID: \"3a14f229-f9d6-4060-b3a9-380f80633a9b\") " pod="llm/e2e-same-id-b-83a1304a-kserve-5f5c8f6dc5-d294t" Jul 18 12:12:02.450799 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:12:02.450785 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3a14f229-f9d6-4060-b3a9-380f80633a9b-dshm\") pod \"e2e-same-id-b-83a1304a-kserve-5f5c8f6dc5-d294t\" (UID: \"3a14f229-f9d6-4060-b3a9-380f80633a9b\") " pod="llm/e2e-same-id-b-83a1304a-kserve-5f5c8f6dc5-d294t" Jul 18 12:12:02.450904 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:12:02.450810 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3a14f229-f9d6-4060-b3a9-380f80633a9b-home\") pod \"e2e-same-id-b-83a1304a-kserve-5f5c8f6dc5-d294t\" (UID: \"3a14f229-f9d6-4060-b3a9-380f80633a9b\") " pod="llm/e2e-same-id-b-83a1304a-kserve-5f5c8f6dc5-d294t" Jul 18 12:12:02.450904 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:12:02.450843 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3a14f229-f9d6-4060-b3a9-380f80633a9b-model-cache\") pod \"e2e-same-id-b-83a1304a-kserve-5f5c8f6dc5-d294t\" (UID: \"3a14f229-f9d6-4060-b3a9-380f80633a9b\") " pod="llm/e2e-same-id-b-83a1304a-kserve-5f5c8f6dc5-d294t" Jul 18 12:12:02.552254 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:12:02.552174 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3a14f229-f9d6-4060-b3a9-380f80633a9b-tls-certs\") pod \"e2e-same-id-b-83a1304a-kserve-5f5c8f6dc5-d294t\" (UID: \"3a14f229-f9d6-4060-b3a9-380f80633a9b\") " pod="llm/e2e-same-id-b-83a1304a-kserve-5f5c8f6dc5-d294t" Jul 18 12:12:02.552254 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:12:02.552212 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3a14f229-f9d6-4060-b3a9-380f80633a9b-dshm\") pod \"e2e-same-id-b-83a1304a-kserve-5f5c8f6dc5-d294t\" (UID: \"3a14f229-f9d6-4060-b3a9-380f80633a9b\") " pod="llm/e2e-same-id-b-83a1304a-kserve-5f5c8f6dc5-d294t" Jul 18 12:12:02.552254 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:12:02.552235 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3a14f229-f9d6-4060-b3a9-380f80633a9b-home\") pod \"e2e-same-id-b-83a1304a-kserve-5f5c8f6dc5-d294t\" (UID: \"3a14f229-f9d6-4060-b3a9-380f80633a9b\") " pod="llm/e2e-same-id-b-83a1304a-kserve-5f5c8f6dc5-d294t" Jul 18 12:12:02.552472 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:12:02.552347 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3a14f229-f9d6-4060-b3a9-380f80633a9b-model-cache\") pod \"e2e-same-id-b-83a1304a-kserve-5f5c8f6dc5-d294t\" (UID: \"3a14f229-f9d6-4060-b3a9-380f80633a9b\") " pod="llm/e2e-same-id-b-83a1304a-kserve-5f5c8f6dc5-d294t" Jul 18 12:12:02.552472 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:12:02.552413 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3a14f229-f9d6-4060-b3a9-380f80633a9b-tmp-dir\") pod \"e2e-same-id-b-83a1304a-kserve-5f5c8f6dc5-d294t\" (UID: \"3a14f229-f9d6-4060-b3a9-380f80633a9b\") " pod="llm/e2e-same-id-b-83a1304a-kserve-5f5c8f6dc5-d294t" Jul 18 12:12:02.552595 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:12:02.552479 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qgcrh\" (UniqueName: \"kubernetes.io/projected/3a14f229-f9d6-4060-b3a9-380f80633a9b-kube-api-access-qgcrh\") pod \"e2e-same-id-b-83a1304a-kserve-5f5c8f6dc5-d294t\" (UID: \"3a14f229-f9d6-4060-b3a9-380f80633a9b\") " pod="llm/e2e-same-id-b-83a1304a-kserve-5f5c8f6dc5-d294t" Jul 18 12:12:02.552720 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:12:02.552699 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3a14f229-f9d6-4060-b3a9-380f80633a9b-home\") pod \"e2e-same-id-b-83a1304a-kserve-5f5c8f6dc5-d294t\" (UID: \"3a14f229-f9d6-4060-b3a9-380f80633a9b\") " pod="llm/e2e-same-id-b-83a1304a-kserve-5f5c8f6dc5-d294t" Jul 18 12:12:02.552778 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:12:02.552745 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3a14f229-f9d6-4060-b3a9-380f80633a9b-model-cache\") pod \"e2e-same-id-b-83a1304a-kserve-5f5c8f6dc5-d294t\" (UID: \"3a14f229-f9d6-4060-b3a9-380f80633a9b\") " pod="llm/e2e-same-id-b-83a1304a-kserve-5f5c8f6dc5-d294t" Jul 18 12:12:02.552853 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:12:02.552832 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3a14f229-f9d6-4060-b3a9-380f80633a9b-tmp-dir\") pod \"e2e-same-id-b-83a1304a-kserve-5f5c8f6dc5-d294t\" (UID: \"3a14f229-f9d6-4060-b3a9-380f80633a9b\") " pod="llm/e2e-same-id-b-83a1304a-kserve-5f5c8f6dc5-d294t" Jul 18 12:12:02.554661 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:12:02.554632 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3a14f229-f9d6-4060-b3a9-380f80633a9b-dshm\") pod \"e2e-same-id-b-83a1304a-kserve-5f5c8f6dc5-d294t\" (UID: \"3a14f229-f9d6-4060-b3a9-380f80633a9b\") " pod="llm/e2e-same-id-b-83a1304a-kserve-5f5c8f6dc5-d294t" Jul 18 12:12:02.554898 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:12:02.554881 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3a14f229-f9d6-4060-b3a9-380f80633a9b-tls-certs\") pod \"e2e-same-id-b-83a1304a-kserve-5f5c8f6dc5-d294t\" (UID: \"3a14f229-f9d6-4060-b3a9-380f80633a9b\") " pod="llm/e2e-same-id-b-83a1304a-kserve-5f5c8f6dc5-d294t" Jul 18 12:12:02.560223 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:12:02.560200 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qgcrh\" (UniqueName: \"kubernetes.io/projected/3a14f229-f9d6-4060-b3a9-380f80633a9b-kube-api-access-qgcrh\") pod \"e2e-same-id-b-83a1304a-kserve-5f5c8f6dc5-d294t\" (UID: \"3a14f229-f9d6-4060-b3a9-380f80633a9b\") " pod="llm/e2e-same-id-b-83a1304a-kserve-5f5c8f6dc5-d294t" Jul 18 12:12:02.651799 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:12:02.651770 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-83a1304a-kserve-5f5c8f6dc5-d294t" Jul 18 12:12:02.788388 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:12:02.788348 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-83a1304a-kserve-5f5c8f6dc5-d294t"] Jul 18 12:12:02.789880 ip-10-0-136-240 kubenswrapper[2585]: W0718 12:12:02.789850 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3a14f229_f9d6_4060_b3a9_380f80633a9b.slice/crio-2ea15750dcb4869d8cc067e2a5d46e02bc3e430da09ff32e96c45b9ed44e0ba3 WatchSource:0}: Error finding container 2ea15750dcb4869d8cc067e2a5d46e02bc3e430da09ff32e96c45b9ed44e0ba3: Status 404 returned error can't find the container with id 2ea15750dcb4869d8cc067e2a5d46e02bc3e430da09ff32e96c45b9ed44e0ba3 Jul 18 12:12:03.251996 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:12:03.251963 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-83a1304a-kserve-5f5c8f6dc5-d294t" event={"ID":"3a14f229-f9d6-4060-b3a9-380f80633a9b","Type":"ContainerStarted","Data":"7014e67b5927bb4d05171570d4341abe9a0f61d04ed398a9f08d963fe1db688d"} Jul 18 12:12:03.252164 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:12:03.252008 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-83a1304a-kserve-5f5c8f6dc5-d294t" event={"ID":"3a14f229-f9d6-4060-b3a9-380f80633a9b","Type":"ContainerStarted","Data":"2ea15750dcb4869d8cc067e2a5d46e02bc3e430da09ff32e96c45b9ed44e0ba3"} Jul 18 12:12:03.263660 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:12:03.263635 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-a-83a1304a-kserve-85dbcbc57f-lfmkk" Jul 18 12:12:03.268512 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:12:03.268466 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-same-id-b-83a1304a-kserve-5f5c8f6dc5-d294t" podStartSLOduration=1.268450875 podStartE2EDuration="1.268450875s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-18 12:12:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-18 12:12:03.267014128 +0000 UTC m=+1390.347220486" watchObservedRunningTime="2026-07-18 12:12:03.268450875 +0000 UTC m=+1390.348657233" Jul 18 12:12:12.652055 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:12:12.652020 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-b-83a1304a-kserve-5f5c8f6dc5-d294t" Jul 18 12:12:12.652497 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:12:12.652320 2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-b-83a1304a-kserve-5f5c8f6dc5-d294t" Jul 18 12:12:12.664494 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:12:12.664471 2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-b-83a1304a-kserve-5f5c8f6dc5-d294t" Jul 18 12:12:13.313516 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:12:13.313488 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-b-83a1304a-kserve-5f5c8f6dc5-d294t" Jul 18 12:13:53.489995 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:13:53.489970 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-j9l7p_8ec47eff-5fbc-4130-baef-d8544aa4e786/console-operator/1.log" Jul 18 12:13:53.492440 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:13:53.492242 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-j9l7p_8ec47eff-5fbc-4130-baef-d8544aa4e786/console-operator/1.log" Jul 18 12:13:53.497206 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:13:53.497187 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-wq4lg_247c97c8-982b-4176-ae10-afc396745360/ovn-acl-logging/0.log" Jul 18 12:13:53.499515 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:13:53.499501 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-wq4lg_247c97c8-982b-4176-ae10-afc396745360/ovn-acl-logging/0.log" Jul 18 12:15:00.742509 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:15:00.742472 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-83a1304a-kserve-85dbcbc57f-lfmkk"] Jul 18 12:15:02.049533 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:15:02.049503 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-83a1304a-kserve-5f5c8f6dc5-d294t"] Jul 18 12:15:15.763498 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:15:15.763435 2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-same-id-a-83a1304a-kserve-85dbcbc57f-lfmkk" podUID="d628c373-734b-4576-b11c-68c664ac7aed" containerName="main" containerID="cri-o://c09d6be4a3f8f4ddbd202d2bf5f5433f6292afb7ed7caf9a0bb0e66789ee71e5" gracePeriod=45 Jul 18 12:15:16.014219 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:15:16.014165 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-83a1304a-kserve-85dbcbc57f-lfmkk" Jul 18 12:15:16.201143 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:15:16.201114 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d628c373-734b-4576-b11c-68c664ac7aed-tls-certs\") pod \"d628c373-734b-4576-b11c-68c664ac7aed\" (UID: \"d628c373-734b-4576-b11c-68c664ac7aed\") " Jul 18 12:15:16.201318 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:15:16.201155 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d628c373-734b-4576-b11c-68c664ac7aed-home\") pod \"d628c373-734b-4576-b11c-68c664ac7aed\" (UID: \"d628c373-734b-4576-b11c-68c664ac7aed\") " Jul 18 12:15:16.201318 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:15:16.201175 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d628c373-734b-4576-b11c-68c664ac7aed-dshm\") pod \"d628c373-734b-4576-b11c-68c664ac7aed\" (UID: \"d628c373-734b-4576-b11c-68c664ac7aed\") " Jul 18 12:15:16.201318 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:15:16.201190 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d628c373-734b-4576-b11c-68c664ac7aed-model-cache\") pod \"d628c373-734b-4576-b11c-68c664ac7aed\" (UID: \"d628c373-734b-4576-b11c-68c664ac7aed\") " Jul 18 12:15:16.201318 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:15:16.201247 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wt8nk\" (UniqueName: \"kubernetes.io/projected/d628c373-734b-4576-b11c-68c664ac7aed-kube-api-access-wt8nk\") pod \"d628c373-734b-4576-b11c-68c664ac7aed\" (UID: \"d628c373-734b-4576-b11c-68c664ac7aed\") " Jul 18 12:15:16.201318 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:15:16.201314 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d628c373-734b-4576-b11c-68c664ac7aed-tmp-dir\") pod \"d628c373-734b-4576-b11c-68c664ac7aed\" (UID: \"d628c373-734b-4576-b11c-68c664ac7aed\") " Jul 18 12:15:16.201597 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:15:16.201384 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d628c373-734b-4576-b11c-68c664ac7aed-home" (OuterVolumeSpecName: "home") pod "d628c373-734b-4576-b11c-68c664ac7aed" (UID: "d628c373-734b-4576-b11c-68c664ac7aed"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 18 12:15:16.201597 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:15:16.201476 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d628c373-734b-4576-b11c-68c664ac7aed-model-cache" (OuterVolumeSpecName: "model-cache") pod "d628c373-734b-4576-b11c-68c664ac7aed" (UID: "d628c373-734b-4576-b11c-68c664ac7aed"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 18 12:15:16.201728 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:15:16.201637 2585 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d628c373-734b-4576-b11c-68c664ac7aed-home\") on node \"ip-10-0-136-240.ec2.internal\" DevicePath \"\"" Jul 18 12:15:16.201728 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:15:16.201654 2585 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d628c373-734b-4576-b11c-68c664ac7aed-model-cache\") on node \"ip-10-0-136-240.ec2.internal\" DevicePath \"\"" Jul 18 12:15:16.201728 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:15:16.201704 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d628c373-734b-4576-b11c-68c664ac7aed-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "d628c373-734b-4576-b11c-68c664ac7aed" (UID: "d628c373-734b-4576-b11c-68c664ac7aed"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 18 12:15:16.203579 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:15:16.203537 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d628c373-734b-4576-b11c-68c664ac7aed-dshm" (OuterVolumeSpecName: "dshm") pod "d628c373-734b-4576-b11c-68c664ac7aed" (UID: "d628c373-734b-4576-b11c-68c664ac7aed"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 18 12:15:16.203685 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:15:16.203640 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d628c373-734b-4576-b11c-68c664ac7aed-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "d628c373-734b-4576-b11c-68c664ac7aed" (UID: "d628c373-734b-4576-b11c-68c664ac7aed"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 18 12:15:16.203685 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:15:16.203637 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d628c373-734b-4576-b11c-68c664ac7aed-kube-api-access-wt8nk" (OuterVolumeSpecName: "kube-api-access-wt8nk") pod "d628c373-734b-4576-b11c-68c664ac7aed" (UID: "d628c373-734b-4576-b11c-68c664ac7aed"). InnerVolumeSpecName "kube-api-access-wt8nk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 18 12:15:16.228520 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:15:16.228495 2585 generic.go:356] "Generic (PLEG): container finished" podID="d628c373-734b-4576-b11c-68c664ac7aed" containerID="c09d6be4a3f8f4ddbd202d2bf5f5433f6292afb7ed7caf9a0bb0e66789ee71e5" exitCode=0 Jul 18 12:15:16.228638 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:15:16.228544 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-83a1304a-kserve-85dbcbc57f-lfmkk" Jul 18 12:15:16.228638 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:15:16.228534 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-83a1304a-kserve-85dbcbc57f-lfmkk" event={"ID":"d628c373-734b-4576-b11c-68c664ac7aed","Type":"ContainerDied","Data":"c09d6be4a3f8f4ddbd202d2bf5f5433f6292afb7ed7caf9a0bb0e66789ee71e5"} Jul 18 12:15:16.228638 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:15:16.228615 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-83a1304a-kserve-85dbcbc57f-lfmkk" event={"ID":"d628c373-734b-4576-b11c-68c664ac7aed","Type":"ContainerDied","Data":"ffaa352e6b9306e793aff986550ac9a8992fdc15e68cd36a4a8951f2630afa90"} Jul 18 12:15:16.228638 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:15:16.228636 2585 scope.go:117] "RemoveContainer" containerID="c09d6be4a3f8f4ddbd202d2bf5f5433f6292afb7ed7caf9a0bb0e66789ee71e5" Jul 18 12:15:16.238024 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:15:16.238007 2585 scope.go:117] "RemoveContainer" containerID="c09d6be4a3f8f4ddbd202d2bf5f5433f6292afb7ed7caf9a0bb0e66789ee71e5" Jul 18 12:15:16.238294 ip-10-0-136-240 kubenswrapper[2585]: E0718 12:15:16.238277 2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c09d6be4a3f8f4ddbd202d2bf5f5433f6292afb7ed7caf9a0bb0e66789ee71e5\": container with ID starting with c09d6be4a3f8f4ddbd202d2bf5f5433f6292afb7ed7caf9a0bb0e66789ee71e5 not found: ID does not exist" containerID="c09d6be4a3f8f4ddbd202d2bf5f5433f6292afb7ed7caf9a0bb0e66789ee71e5" Jul 18 12:15:16.238354 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:15:16.238301 2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c09d6be4a3f8f4ddbd202d2bf5f5433f6292afb7ed7caf9a0bb0e66789ee71e5"} err="failed to get container status \"c09d6be4a3f8f4ddbd202d2bf5f5433f6292afb7ed7caf9a0bb0e66789ee71e5\": rpc error: code = NotFound desc = could not find container \"c09d6be4a3f8f4ddbd202d2bf5f5433f6292afb7ed7caf9a0bb0e66789ee71e5\": container with ID starting with c09d6be4a3f8f4ddbd202d2bf5f5433f6292afb7ed7caf9a0bb0e66789ee71e5 not found: ID does not exist" Jul 18 12:15:16.248757 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:15:16.248734 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-83a1304a-kserve-85dbcbc57f-lfmkk"] Jul 18 12:15:16.252147 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:15:16.252124 2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-a-83a1304a-kserve-85dbcbc57f-lfmkk"] Jul 18 12:15:16.302426 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:15:16.302367 2585 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d628c373-734b-4576-b11c-68c664ac7aed-tmp-dir\") on node \"ip-10-0-136-240.ec2.internal\" DevicePath \"\"" Jul 18 12:15:16.302426 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:15:16.302389 2585 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d628c373-734b-4576-b11c-68c664ac7aed-tls-certs\") on node \"ip-10-0-136-240.ec2.internal\" DevicePath \"\"" Jul 18 12:15:16.302426 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:15:16.302400 2585 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d628c373-734b-4576-b11c-68c664ac7aed-dshm\") on node \"ip-10-0-136-240.ec2.internal\" DevicePath \"\"" Jul 18 12:15:16.302426 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:15:16.302413 2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wt8nk\" (UniqueName: \"kubernetes.io/projected/d628c373-734b-4576-b11c-68c664ac7aed-kube-api-access-wt8nk\") on node \"ip-10-0-136-240.ec2.internal\" DevicePath \"\"" Jul 18 12:15:17.070394 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:15:17.070354 2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-same-id-b-83a1304a-kserve-5f5c8f6dc5-d294t" podUID="3a14f229-f9d6-4060-b3a9-380f80633a9b" containerName="main" containerID="cri-o://7014e67b5927bb4d05171570d4341abe9a0f61d04ed398a9f08d963fe1db688d" gracePeriod=45 Jul 18 12:15:17.235356 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:15:17.235322 2585 generic.go:356] "Generic (PLEG): container finished" podID="3a14f229-f9d6-4060-b3a9-380f80633a9b" containerID="7014e67b5927bb4d05171570d4341abe9a0f61d04ed398a9f08d963fe1db688d" exitCode=0 Jul 18 12:15:17.235512 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:15:17.235400 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-83a1304a-kserve-5f5c8f6dc5-d294t" event={"ID":"3a14f229-f9d6-4060-b3a9-380f80633a9b","Type":"ContainerDied","Data":"7014e67b5927bb4d05171570d4341abe9a0f61d04ed398a9f08d963fe1db688d"} Jul 18 12:15:17.327870 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:15:17.327817 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-83a1304a-kserve-5f5c8f6dc5-d294t" Jul 18 12:15:17.469190 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:15:17.469156 2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d628c373-734b-4576-b11c-68c664ac7aed" path="/var/lib/kubelet/pods/d628c373-734b-4576-b11c-68c664ac7aed/volumes" Jul 18 12:15:17.512229 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:15:17.512199 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3a14f229-f9d6-4060-b3a9-380f80633a9b-tls-certs\") pod \"3a14f229-f9d6-4060-b3a9-380f80633a9b\" (UID: \"3a14f229-f9d6-4060-b3a9-380f80633a9b\") " Jul 18 12:15:17.512384 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:15:17.512250 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3a14f229-f9d6-4060-b3a9-380f80633a9b-dshm\") pod \"3a14f229-f9d6-4060-b3a9-380f80633a9b\" (UID: \"3a14f229-f9d6-4060-b3a9-380f80633a9b\") " Jul 18 12:15:17.512384 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:15:17.512273 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qgcrh\" (UniqueName: \"kubernetes.io/projected/3a14f229-f9d6-4060-b3a9-380f80633a9b-kube-api-access-qgcrh\") pod \"3a14f229-f9d6-4060-b3a9-380f80633a9b\" (UID: \"3a14f229-f9d6-4060-b3a9-380f80633a9b\") " Jul 18 12:15:17.512384 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:15:17.512344 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3a14f229-f9d6-4060-b3a9-380f80633a9b-model-cache\") pod \"3a14f229-f9d6-4060-b3a9-380f80633a9b\" (UID: \"3a14f229-f9d6-4060-b3a9-380f80633a9b\") " Jul 18 12:15:17.512384 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:15:17.512371 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3a14f229-f9d6-4060-b3a9-380f80633a9b-home\") pod \"3a14f229-f9d6-4060-b3a9-380f80633a9b\" (UID: \"3a14f229-f9d6-4060-b3a9-380f80633a9b\") " Jul 18 12:15:17.512637 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:15:17.512420 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3a14f229-f9d6-4060-b3a9-380f80633a9b-tmp-dir\") pod \"3a14f229-f9d6-4060-b3a9-380f80633a9b\" (UID: \"3a14f229-f9d6-4060-b3a9-380f80633a9b\") " Jul 18 12:15:17.512750 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:15:17.512722 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3a14f229-f9d6-4060-b3a9-380f80633a9b-home" (OuterVolumeSpecName: "home") pod "3a14f229-f9d6-4060-b3a9-380f80633a9b" (UID: "3a14f229-f9d6-4060-b3a9-380f80633a9b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 18 12:15:17.512807 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:15:17.512728 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3a14f229-f9d6-4060-b3a9-380f80633a9b-model-cache" (OuterVolumeSpecName: "model-cache") pod "3a14f229-f9d6-4060-b3a9-380f80633a9b" (UID: "3a14f229-f9d6-4060-b3a9-380f80633a9b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 18 12:15:17.512807 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:15:17.512757 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3a14f229-f9d6-4060-b3a9-380f80633a9b-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "3a14f229-f9d6-4060-b3a9-380f80633a9b" (UID: "3a14f229-f9d6-4060-b3a9-380f80633a9b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 18 12:15:17.514578 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:15:17.514537 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3a14f229-f9d6-4060-b3a9-380f80633a9b-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "3a14f229-f9d6-4060-b3a9-380f80633a9b" (UID: "3a14f229-f9d6-4060-b3a9-380f80633a9b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 18 12:15:17.515006 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:15:17.514990 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3a14f229-f9d6-4060-b3a9-380f80633a9b-dshm" (OuterVolumeSpecName: "dshm") pod "3a14f229-f9d6-4060-b3a9-380f80633a9b" (UID: "3a14f229-f9d6-4060-b3a9-380f80633a9b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 18 12:15:17.515093 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:15:17.515020 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3a14f229-f9d6-4060-b3a9-380f80633a9b-kube-api-access-qgcrh" (OuterVolumeSpecName: "kube-api-access-qgcrh") pod "3a14f229-f9d6-4060-b3a9-380f80633a9b" (UID: "3a14f229-f9d6-4060-b3a9-380f80633a9b"). InnerVolumeSpecName "kube-api-access-qgcrh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 18 12:15:17.613365 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:15:17.613293 2585 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3a14f229-f9d6-4060-b3a9-380f80633a9b-tmp-dir\") on node \"ip-10-0-136-240.ec2.internal\" DevicePath \"\"" Jul 18 12:15:17.613365 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:15:17.613313 2585 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3a14f229-f9d6-4060-b3a9-380f80633a9b-tls-certs\") on node \"ip-10-0-136-240.ec2.internal\" DevicePath \"\"" Jul 18 12:15:17.613365 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:15:17.613323 2585 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3a14f229-f9d6-4060-b3a9-380f80633a9b-dshm\") on node \"ip-10-0-136-240.ec2.internal\" DevicePath \"\"" Jul 18 12:15:17.613365 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:15:17.613331 2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qgcrh\" (UniqueName: \"kubernetes.io/projected/3a14f229-f9d6-4060-b3a9-380f80633a9b-kube-api-access-qgcrh\") on node \"ip-10-0-136-240.ec2.internal\" DevicePath \"\"" Jul 18 12:15:17.613365 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:15:17.613340 2585 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3a14f229-f9d6-4060-b3a9-380f80633a9b-model-cache\") on node \"ip-10-0-136-240.ec2.internal\" DevicePath \"\"" Jul 18 12:15:17.613365 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:15:17.613351 2585 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3a14f229-f9d6-4060-b3a9-380f80633a9b-home\") on node \"ip-10-0-136-240.ec2.internal\" DevicePath \"\"" Jul 18 12:15:18.242707 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:15:18.242671 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-83a1304a-kserve-5f5c8f6dc5-d294t" Jul 18 12:15:18.243121 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:15:18.242676 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-83a1304a-kserve-5f5c8f6dc5-d294t" event={"ID":"3a14f229-f9d6-4060-b3a9-380f80633a9b","Type":"ContainerDied","Data":"2ea15750dcb4869d8cc067e2a5d46e02bc3e430da09ff32e96c45b9ed44e0ba3"} Jul 18 12:15:18.243121 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:15:18.242781 2585 scope.go:117] "RemoveContainer" containerID="7014e67b5927bb4d05171570d4341abe9a0f61d04ed398a9f08d963fe1db688d" Jul 18 12:15:18.265570 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:15:18.265528 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-83a1304a-kserve-5f5c8f6dc5-d294t"] Jul 18 12:15:18.267847 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:15:18.267826 2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-b-83a1304a-kserve-5f5c8f6dc5-d294t"] Jul 18 12:15:19.469168 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:15:19.469135 2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3a14f229-f9d6-4060-b3a9-380f80633a9b" path="/var/lib/kubelet/pods/3a14f229-f9d6-4060-b3a9-380f80633a9b/volumes" Jul 18 12:18:25.635805 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:18:25.635774 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-736936a8-openshift-default-9bb5769b5-9s8wl"] Jul 18 12:18:25.636251 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:18:25.636145 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3a14f229-f9d6-4060-b3a9-380f80633a9b" containerName="main" Jul 18 12:18:25.636251 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:18:25.636157 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="3a14f229-f9d6-4060-b3a9-380f80633a9b" containerName="main" Jul 18 12:18:25.636251 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:18:25.636185 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d628c373-734b-4576-b11c-68c664ac7aed" containerName="main" Jul 18 12:18:25.636251 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:18:25.636191 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="d628c373-734b-4576-b11c-68c664ac7aed" containerName="main" Jul 18 12:18:25.636251 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:18:25.636244 2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="d628c373-734b-4576-b11c-68c664ac7aed" containerName="main" Jul 18 12:18:25.636251 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:18:25.636253 2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="3a14f229-f9d6-4060-b3a9-380f80633a9b" containerName="main" Jul 18 12:18:25.639297 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:18:25.639279 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-736936a8-openshift-default-9bb5769b5-9s8wl" Jul 18 12:18:25.641581 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:18:25.641561 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-736936a8-openshift-default-dockercfg-dn4p9\"" Jul 18 12:18:25.661877 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:18:25.661853 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-736936a8-openshift-default-9bb5769b5-9s8wl"] Jul 18 12:18:25.761506 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:18:25.761471 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c19f06a1-ccff-4eb8-80a1-490d3ebef8e0-istio-envoy\") pod \"e2e-ait-736936a8-openshift-default-9bb5769b5-9s8wl\" (UID: \"c19f06a1-ccff-4eb8-80a1-490d3ebef8e0\") " pod="openshift-ingress/e2e-ait-736936a8-openshift-default-9bb5769b5-9s8wl" Jul 18 12:18:25.761506 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:18:25.761509 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c19f06a1-ccff-4eb8-80a1-490d3ebef8e0-istio-podinfo\") pod \"e2e-ait-736936a8-openshift-default-9bb5769b5-9s8wl\" (UID: \"c19f06a1-ccff-4eb8-80a1-490d3ebef8e0\") " pod="openshift-ingress/e2e-ait-736936a8-openshift-default-9bb5769b5-9s8wl" Jul 18 12:18:25.761728 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:18:25.761531 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c19f06a1-ccff-4eb8-80a1-490d3ebef8e0-workload-socket\") pod \"e2e-ait-736936a8-openshift-default-9bb5769b5-9s8wl\" (UID: \"c19f06a1-ccff-4eb8-80a1-490d3ebef8e0\") " pod="openshift-ingress/e2e-ait-736936a8-openshift-default-9bb5769b5-9s8wl" Jul 18 12:18:25.761728 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:18:25.761644 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c19f06a1-ccff-4eb8-80a1-490d3ebef8e0-credential-socket\") pod \"e2e-ait-736936a8-openshift-default-9bb5769b5-9s8wl\" (UID: \"c19f06a1-ccff-4eb8-80a1-490d3ebef8e0\") " pod="openshift-ingress/e2e-ait-736936a8-openshift-default-9bb5769b5-9s8wl" Jul 18 12:18:25.761728 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:18:25.761680 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c19f06a1-ccff-4eb8-80a1-490d3ebef8e0-istio-token\") pod \"e2e-ait-736936a8-openshift-default-9bb5769b5-9s8wl\" (UID: \"c19f06a1-ccff-4eb8-80a1-490d3ebef8e0\") " pod="openshift-ingress/e2e-ait-736936a8-openshift-default-9bb5769b5-9s8wl" Jul 18 12:18:25.761728 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:18:25.761698 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c19f06a1-ccff-4eb8-80a1-490d3ebef8e0-workload-certs\") pod \"e2e-ait-736936a8-openshift-default-9bb5769b5-9s8wl\" (UID: \"c19f06a1-ccff-4eb8-80a1-490d3ebef8e0\") " pod="openshift-ingress/e2e-ait-736936a8-openshift-default-9bb5769b5-9s8wl" Jul 18 12:18:25.761856 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:18:25.761741 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c19f06a1-ccff-4eb8-80a1-490d3ebef8e0-istiod-ca-cert\") pod \"e2e-ait-736936a8-openshift-default-9bb5769b5-9s8wl\" (UID: \"c19f06a1-ccff-4eb8-80a1-490d3ebef8e0\") " pod="openshift-ingress/e2e-ait-736936a8-openshift-default-9bb5769b5-9s8wl" Jul 18 12:18:25.761856 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:18:25.761767 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t8mvh\" (UniqueName: \"kubernetes.io/projected/c19f06a1-ccff-4eb8-80a1-490d3ebef8e0-kube-api-access-t8mvh\") pod \"e2e-ait-736936a8-openshift-default-9bb5769b5-9s8wl\" (UID: \"c19f06a1-ccff-4eb8-80a1-490d3ebef8e0\") " pod="openshift-ingress/e2e-ait-736936a8-openshift-default-9bb5769b5-9s8wl" Jul 18 12:18:25.761856 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:18:25.761840 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c19f06a1-ccff-4eb8-80a1-490d3ebef8e0-istio-data\") pod \"e2e-ait-736936a8-openshift-default-9bb5769b5-9s8wl\" (UID: \"c19f06a1-ccff-4eb8-80a1-490d3ebef8e0\") " pod="openshift-ingress/e2e-ait-736936a8-openshift-default-9bb5769b5-9s8wl" Jul 18 12:18:25.862763 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:18:25.862731 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c19f06a1-ccff-4eb8-80a1-490d3ebef8e0-istio-data\") pod \"e2e-ait-736936a8-openshift-default-9bb5769b5-9s8wl\" (UID: \"c19f06a1-ccff-4eb8-80a1-490d3ebef8e0\") " pod="openshift-ingress/e2e-ait-736936a8-openshift-default-9bb5769b5-9s8wl" Jul 18 12:18:25.862909 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:18:25.862779 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c19f06a1-ccff-4eb8-80a1-490d3ebef8e0-istio-envoy\") pod \"e2e-ait-736936a8-openshift-default-9bb5769b5-9s8wl\" (UID: \"c19f06a1-ccff-4eb8-80a1-490d3ebef8e0\") " pod="openshift-ingress/e2e-ait-736936a8-openshift-default-9bb5769b5-9s8wl" Jul 18 12:18:25.862909 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:18:25.862805 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c19f06a1-ccff-4eb8-80a1-490d3ebef8e0-istio-podinfo\") pod \"e2e-ait-736936a8-openshift-default-9bb5769b5-9s8wl\" (UID: \"c19f06a1-ccff-4eb8-80a1-490d3ebef8e0\") " pod="openshift-ingress/e2e-ait-736936a8-openshift-default-9bb5769b5-9s8wl" Jul 18 12:18:25.862909 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:18:25.862829 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c19f06a1-ccff-4eb8-80a1-490d3ebef8e0-workload-socket\") pod \"e2e-ait-736936a8-openshift-default-9bb5769b5-9s8wl\" (UID: \"c19f06a1-ccff-4eb8-80a1-490d3ebef8e0\") " pod="openshift-ingress/e2e-ait-736936a8-openshift-default-9bb5769b5-9s8wl" Jul 18 12:18:25.862909 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:18:25.862878 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c19f06a1-ccff-4eb8-80a1-490d3ebef8e0-credential-socket\") pod \"e2e-ait-736936a8-openshift-default-9bb5769b5-9s8wl\" (UID: \"c19f06a1-ccff-4eb8-80a1-490d3ebef8e0\") " pod="openshift-ingress/e2e-ait-736936a8-openshift-default-9bb5769b5-9s8wl" Jul 18 12:18:25.863080 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:18:25.862911 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c19f06a1-ccff-4eb8-80a1-490d3ebef8e0-istio-token\") pod \"e2e-ait-736936a8-openshift-default-9bb5769b5-9s8wl\" (UID: \"c19f06a1-ccff-4eb8-80a1-490d3ebef8e0\") " pod="openshift-ingress/e2e-ait-736936a8-openshift-default-9bb5769b5-9s8wl" Jul 18 12:18:25.863080 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:18:25.862932 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c19f06a1-ccff-4eb8-80a1-490d3ebef8e0-workload-certs\") pod \"e2e-ait-736936a8-openshift-default-9bb5769b5-9s8wl\" (UID: \"c19f06a1-ccff-4eb8-80a1-490d3ebef8e0\") " pod="openshift-ingress/e2e-ait-736936a8-openshift-default-9bb5769b5-9s8wl" Jul 18 12:18:25.863080 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:18:25.862962 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c19f06a1-ccff-4eb8-80a1-490d3ebef8e0-istiod-ca-cert\") pod \"e2e-ait-736936a8-openshift-default-9bb5769b5-9s8wl\" (UID: \"c19f06a1-ccff-4eb8-80a1-490d3ebef8e0\") " pod="openshift-ingress/e2e-ait-736936a8-openshift-default-9bb5769b5-9s8wl" Jul 18 12:18:25.863080 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:18:25.862995 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t8mvh\" (UniqueName: \"kubernetes.io/projected/c19f06a1-ccff-4eb8-80a1-490d3ebef8e0-kube-api-access-t8mvh\") pod \"e2e-ait-736936a8-openshift-default-9bb5769b5-9s8wl\" (UID: \"c19f06a1-ccff-4eb8-80a1-490d3ebef8e0\") " pod="openshift-ingress/e2e-ait-736936a8-openshift-default-9bb5769b5-9s8wl" Jul 18 12:18:25.863255 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:18:25.863157 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c19f06a1-ccff-4eb8-80a1-490d3ebef8e0-istio-data\") pod \"e2e-ait-736936a8-openshift-default-9bb5769b5-9s8wl\" (UID: \"c19f06a1-ccff-4eb8-80a1-490d3ebef8e0\") " pod="openshift-ingress/e2e-ait-736936a8-openshift-default-9bb5769b5-9s8wl" Jul 18 12:18:25.863255 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:18:25.863168 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c19f06a1-ccff-4eb8-80a1-490d3ebef8e0-workload-socket\") pod \"e2e-ait-736936a8-openshift-default-9bb5769b5-9s8wl\" (UID: \"c19f06a1-ccff-4eb8-80a1-490d3ebef8e0\") " pod="openshift-ingress/e2e-ait-736936a8-openshift-default-9bb5769b5-9s8wl" Jul 18 12:18:25.863448 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:18:25.863415 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c19f06a1-ccff-4eb8-80a1-490d3ebef8e0-credential-socket\") pod \"e2e-ait-736936a8-openshift-default-9bb5769b5-9s8wl\" (UID: \"c19f06a1-ccff-4eb8-80a1-490d3ebef8e0\") " pod="openshift-ingress/e2e-ait-736936a8-openshift-default-9bb5769b5-9s8wl" Jul 18 12:18:25.863621 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:18:25.863454 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c19f06a1-ccff-4eb8-80a1-490d3ebef8e0-workload-certs\") pod \"e2e-ait-736936a8-openshift-default-9bb5769b5-9s8wl\" (UID: \"c19f06a1-ccff-4eb8-80a1-490d3ebef8e0\") " pod="openshift-ingress/e2e-ait-736936a8-openshift-default-9bb5769b5-9s8wl" Jul 18 12:18:25.863933 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:18:25.863907 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c19f06a1-ccff-4eb8-80a1-490d3ebef8e0-istiod-ca-cert\") pod \"e2e-ait-736936a8-openshift-default-9bb5769b5-9s8wl\" (UID: \"c19f06a1-ccff-4eb8-80a1-490d3ebef8e0\") " pod="openshift-ingress/e2e-ait-736936a8-openshift-default-9bb5769b5-9s8wl" Jul 18 12:18:25.865386 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:18:25.865363 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c19f06a1-ccff-4eb8-80a1-490d3ebef8e0-istio-envoy\") pod \"e2e-ait-736936a8-openshift-default-9bb5769b5-9s8wl\" (UID: \"c19f06a1-ccff-4eb8-80a1-490d3ebef8e0\") " pod="openshift-ingress/e2e-ait-736936a8-openshift-default-9bb5769b5-9s8wl" Jul 18 12:18:25.865508 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:18:25.865490 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c19f06a1-ccff-4eb8-80a1-490d3ebef8e0-istio-podinfo\") pod \"e2e-ait-736936a8-openshift-default-9bb5769b5-9s8wl\" (UID: \"c19f06a1-ccff-4eb8-80a1-490d3ebef8e0\") " pod="openshift-ingress/e2e-ait-736936a8-openshift-default-9bb5769b5-9s8wl" Jul 18 12:18:25.871052 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:18:25.871023 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c19f06a1-ccff-4eb8-80a1-490d3ebef8e0-istio-token\") pod \"e2e-ait-736936a8-openshift-default-9bb5769b5-9s8wl\" (UID: \"c19f06a1-ccff-4eb8-80a1-490d3ebef8e0\") " pod="openshift-ingress/e2e-ait-736936a8-openshift-default-9bb5769b5-9s8wl" Jul 18 12:18:25.871203 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:18:25.871188 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t8mvh\" (UniqueName: \"kubernetes.io/projected/c19f06a1-ccff-4eb8-80a1-490d3ebef8e0-kube-api-access-t8mvh\") pod \"e2e-ait-736936a8-openshift-default-9bb5769b5-9s8wl\" (UID: \"c19f06a1-ccff-4eb8-80a1-490d3ebef8e0\") " pod="openshift-ingress/e2e-ait-736936a8-openshift-default-9bb5769b5-9s8wl" Jul 18 12:18:25.949033 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:18:25.948980 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-736936a8-openshift-default-9bb5769b5-9s8wl" Jul 18 12:18:26.088243 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:18:26.088220 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-736936a8-openshift-default-9bb5769b5-9s8wl"] Jul 18 12:18:26.089926 ip-10-0-136-240 kubenswrapper[2585]: W0718 12:18:26.089900 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc19f06a1_ccff_4eb8_80a1_490d3ebef8e0.slice/crio-6ae634be34e00260373c1314804bef9f4452c298e161dfe7da5594d6472eb838 WatchSource:0}: Error finding container 6ae634be34e00260373c1314804bef9f4452c298e161dfe7da5594d6472eb838: Status 404 returned error can't find the container with id 6ae634be34e00260373c1314804bef9f4452c298e161dfe7da5594d6472eb838 Jul 18 12:18:26.092252 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:18:26.092215 2585 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 18 12:18:26.092318 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:18:26.092301 2585 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 18 12:18:26.092360 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:18:26.092346 2585 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 18 12:18:26.164886 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:18:26.164847 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-736936a8-openshift-default-9bb5769b5-9s8wl" event={"ID":"c19f06a1-ccff-4eb8-80a1-490d3ebef8e0","Type":"ContainerStarted","Data":"c651510a3b92f30364c885b435d98c9b082d0d5447f8093e50230aadfd2321d1"} Jul 18 12:18:26.164980 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:18:26.164911 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-736936a8-openshift-default-9bb5769b5-9s8wl" event={"ID":"c19f06a1-ccff-4eb8-80a1-490d3ebef8e0","Type":"ContainerStarted","Data":"6ae634be34e00260373c1314804bef9f4452c298e161dfe7da5594d6472eb838"} Jul 18 12:18:26.182605 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:18:26.182519 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-736936a8-openshift-default-9bb5769b5-9s8wl" podStartSLOduration=1.182500581 podStartE2EDuration="1.182500581s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-18 12:18:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-18 12:18:26.181238647 +0000 UTC m=+1773.261445017" watchObservedRunningTime="2026-07-18 12:18:26.182500581 +0000 UTC m=+1773.262706938" Jul 18 12:18:26.950040 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:18:26.950008 2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-736936a8-openshift-default-9bb5769b5-9s8wl" Jul 18 12:18:27.954439 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:18:27.954412 2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-736936a8-openshift-default-9bb5769b5-9s8wl" Jul 18 12:18:28.178703 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:18:28.178677 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-736936a8-openshift-default-9bb5769b5-9s8wl" Jul 18 12:18:28.179598 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:18:28.179581 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-736936a8-openshift-default-9bb5769b5-9s8wl" Jul 18 12:18:40.240623 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:18:40.240577 2585 generic.go:356] "Generic (PLEG): container finished" podID="c19f06a1-ccff-4eb8-80a1-490d3ebef8e0" containerID="c651510a3b92f30364c885b435d98c9b082d0d5447f8093e50230aadfd2321d1" exitCode=0 Jul 18 12:18:40.241065 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:18:40.240588 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-736936a8-openshift-default-9bb5769b5-9s8wl" event={"ID":"c19f06a1-ccff-4eb8-80a1-490d3ebef8e0","Type":"ContainerDied","Data":"c651510a3b92f30364c885b435d98c9b082d0d5447f8093e50230aadfd2321d1"} Jul 18 12:18:40.241165 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:18:40.241147 2585 scope.go:117] "RemoveContainer" containerID="c651510a3b92f30364c885b435d98c9b082d0d5447f8093e50230aadfd2321d1" Jul 18 12:18:40.242341 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:18:40.242313 2585 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 18 12:18:40.242413 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:18:40.242397 2585 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 18 12:18:40.242463 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:18:40.242448 2585 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 18 12:18:41.248222 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:18:41.248187 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-736936a8-openshift-default-9bb5769b5-9s8wl" event={"ID":"c19f06a1-ccff-4eb8-80a1-490d3ebef8e0","Type":"ContainerStarted","Data":"bd71606e5804257d27b54ee4855b362e91386d87380091419d442fd2078b2773"} Jul 18 12:18:41.950124 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:18:41.950087 2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-736936a8-openshift-default-9bb5769b5-9s8wl" Jul 18 12:18:41.951241 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:18:41.951217 2585 patch_prober.go:28] interesting pod/e2e-ait-736936a8-openshift-default-9bb5769b5-9s8wl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" start-of-body= Jul 18 12:18:41.951357 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:18:41.951266 2585 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-736936a8-openshift-default-9bb5769b5-9s8wl" podUID="c19f06a1-ccff-4eb8-80a1-490d3ebef8e0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" Jul 18 12:18:42.950254 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:18:42.950220 2585 patch_prober.go:28] interesting pod/e2e-ait-736936a8-openshift-default-9bb5769b5-9s8wl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" start-of-body= Jul 18 12:18:42.950663 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:18:42.950277 2585 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-736936a8-openshift-default-9bb5769b5-9s8wl" podUID="c19f06a1-ccff-4eb8-80a1-490d3ebef8e0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" Jul 18 12:18:43.179087 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:18:43.179050 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-736936a8-openshift-default-9bb5769b5-9s8wl" Jul 18 12:18:43.949808 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:18:43.949778 2585 patch_prober.go:28] interesting pod/e2e-ait-736936a8-openshift-default-9bb5769b5-9s8wl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" start-of-body= Jul 18 12:18:43.949977 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:18:43.949830 2585 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-736936a8-openshift-default-9bb5769b5-9s8wl" podUID="c19f06a1-ccff-4eb8-80a1-490d3ebef8e0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" Jul 18 12:18:44.949503 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:18:44.949417 2585 patch_prober.go:28] interesting pod/e2e-ait-736936a8-openshift-default-9bb5769b5-9s8wl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" start-of-body= Jul 18 12:18:44.949503 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:18:44.949487 2585 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-736936a8-openshift-default-9bb5769b5-9s8wl" podUID="c19f06a1-ccff-4eb8-80a1-490d3ebef8e0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" Jul 18 12:18:45.953505 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:18:45.953478 2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-736936a8-openshift-default-9bb5769b5-9s8wl" Jul 18 12:18:45.954385 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:18:45.954361 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-736936a8-openshift-default-9bb5769b5-9s8wl" Jul 18 12:18:53.519917 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:18:53.519886 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-j9l7p_8ec47eff-5fbc-4130-baef-d8544aa4e786/console-operator/1.log" Jul 18 12:18:53.523878 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:18:53.523845 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-j9l7p_8ec47eff-5fbc-4130-baef-d8544aa4e786/console-operator/1.log" Jul 18 12:18:53.527017 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:18:53.526999 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-wq4lg_247c97c8-982b-4176-ae10-afc396745360/ovn-acl-logging/0.log" Jul 18 12:18:53.530871 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:18:53.530853 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-wq4lg_247c97c8-982b-4176-ae10-afc396745360/ovn-acl-logging/0.log" Jul 18 12:19:08.321257 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:19:08.321217 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-736936a8-openshift-default-9bb5769b5-9s8wl"] Jul 18 12:19:08.321859 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:19:08.321431 2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-736936a8-openshift-default-9bb5769b5-9s8wl" podUID="c19f06a1-ccff-4eb8-80a1-490d3ebef8e0" containerName="istio-proxy" containerID="cri-o://bd71606e5804257d27b54ee4855b362e91386d87380091419d442fd2078b2773" gracePeriod=30 Jul 18 12:19:09.343178 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:19:09.343135 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-migrate-f1cd424d-gw-openshift-default-d98684879-5kkmq"] Jul 18 12:19:09.347192 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:19:09.347171 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-f1cd424d-gw-openshift-default-d98684879-5kkmq" Jul 18 12:19:09.349573 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:19:09.349533 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-migrate-f1cd424d-gw-openshift-default-dockercfg-f9sqn\"" Jul 18 12:19:09.359678 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:19:09.359653 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-f1cd424d-gw-openshift-default-d98684879-5kkmq"] Jul 18 12:19:09.467003 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:19:09.466969 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9eb24b37-ec67-41b4-a2f1-b69bcced3c4e-istio-data\") pod \"e2e-migrate-f1cd424d-gw-openshift-default-d98684879-5kkmq\" (UID: \"9eb24b37-ec67-41b4-a2f1-b69bcced3c4e\") " pod="openshift-ingress/e2e-migrate-f1cd424d-gw-openshift-default-d98684879-5kkmq" Jul 18 12:19:09.467003 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:19:09.467010 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9eb24b37-ec67-41b4-a2f1-b69bcced3c4e-istio-token\") pod \"e2e-migrate-f1cd424d-gw-openshift-default-d98684879-5kkmq\" (UID: \"9eb24b37-ec67-41b4-a2f1-b69bcced3c4e\") " pod="openshift-ingress/e2e-migrate-f1cd424d-gw-openshift-default-d98684879-5kkmq" Jul 18 12:19:09.467256 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:19:09.467093 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9eb24b37-ec67-41b4-a2f1-b69bcced3c4e-istio-podinfo\") pod \"e2e-migrate-f1cd424d-gw-openshift-default-d98684879-5kkmq\" (UID: \"9eb24b37-ec67-41b4-a2f1-b69bcced3c4e\") " pod="openshift-ingress/e2e-migrate-f1cd424d-gw-openshift-default-d98684879-5kkmq" Jul 18 12:19:09.467256 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:19:09.467128 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9eb24b37-ec67-41b4-a2f1-b69bcced3c4e-istio-envoy\") pod \"e2e-migrate-f1cd424d-gw-openshift-default-d98684879-5kkmq\" (UID: \"9eb24b37-ec67-41b4-a2f1-b69bcced3c4e\") " pod="openshift-ingress/e2e-migrate-f1cd424d-gw-openshift-default-d98684879-5kkmq" Jul 18 12:19:09.467256 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:19:09.467156 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9eb24b37-ec67-41b4-a2f1-b69bcced3c4e-credential-socket\") pod \"e2e-migrate-f1cd424d-gw-openshift-default-d98684879-5kkmq\" (UID: \"9eb24b37-ec67-41b4-a2f1-b69bcced3c4e\") " pod="openshift-ingress/e2e-migrate-f1cd424d-gw-openshift-default-d98684879-5kkmq" Jul 18 12:19:09.467256 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:19:09.467191 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9eb24b37-ec67-41b4-a2f1-b69bcced3c4e-workload-certs\") pod \"e2e-migrate-f1cd424d-gw-openshift-default-d98684879-5kkmq\" (UID: \"9eb24b37-ec67-41b4-a2f1-b69bcced3c4e\") " pod="openshift-ingress/e2e-migrate-f1cd424d-gw-openshift-default-d98684879-5kkmq" Jul 18 12:19:09.467256 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:19:09.467219 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9eb24b37-ec67-41b4-a2f1-b69bcced3c4e-workload-socket\") pod \"e2e-migrate-f1cd424d-gw-openshift-default-d98684879-5kkmq\" (UID: \"9eb24b37-ec67-41b4-a2f1-b69bcced3c4e\") " pod="openshift-ingress/e2e-migrate-f1cd424d-gw-openshift-default-d98684879-5kkmq" Jul 18 12:19:09.467256 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:19:09.467246 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5tmkb\" (UniqueName: \"kubernetes.io/projected/9eb24b37-ec67-41b4-a2f1-b69bcced3c4e-kube-api-access-5tmkb\") pod \"e2e-migrate-f1cd424d-gw-openshift-default-d98684879-5kkmq\" (UID: \"9eb24b37-ec67-41b4-a2f1-b69bcced3c4e\") " pod="openshift-ingress/e2e-migrate-f1cd424d-gw-openshift-default-d98684879-5kkmq" Jul 18 12:19:09.467475 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:19:09.467266 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9eb24b37-ec67-41b4-a2f1-b69bcced3c4e-istiod-ca-cert\") pod \"e2e-migrate-f1cd424d-gw-openshift-default-d98684879-5kkmq\" (UID: \"9eb24b37-ec67-41b4-a2f1-b69bcced3c4e\") " pod="openshift-ingress/e2e-migrate-f1cd424d-gw-openshift-default-d98684879-5kkmq" Jul 18 12:19:09.568389 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:19:09.568358 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9eb24b37-ec67-41b4-a2f1-b69bcced3c4e-istio-data\") pod \"e2e-migrate-f1cd424d-gw-openshift-default-d98684879-5kkmq\" (UID: \"9eb24b37-ec67-41b4-a2f1-b69bcced3c4e\") " pod="openshift-ingress/e2e-migrate-f1cd424d-gw-openshift-default-d98684879-5kkmq" Jul 18 12:19:09.568601 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:19:09.568408 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9eb24b37-ec67-41b4-a2f1-b69bcced3c4e-istio-token\") pod \"e2e-migrate-f1cd424d-gw-openshift-default-d98684879-5kkmq\" (UID: \"9eb24b37-ec67-41b4-a2f1-b69bcced3c4e\") " pod="openshift-ingress/e2e-migrate-f1cd424d-gw-openshift-default-d98684879-5kkmq" Jul 18 12:19:09.568601 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:19:09.568447 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9eb24b37-ec67-41b4-a2f1-b69bcced3c4e-istio-podinfo\") pod \"e2e-migrate-f1cd424d-gw-openshift-default-d98684879-5kkmq\" (UID: \"9eb24b37-ec67-41b4-a2f1-b69bcced3c4e\") " pod="openshift-ingress/e2e-migrate-f1cd424d-gw-openshift-default-d98684879-5kkmq" Jul 18 12:19:09.568601 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:19:09.568499 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9eb24b37-ec67-41b4-a2f1-b69bcced3c4e-istio-envoy\") pod \"e2e-migrate-f1cd424d-gw-openshift-default-d98684879-5kkmq\" (UID: \"9eb24b37-ec67-41b4-a2f1-b69bcced3c4e\") " pod="openshift-ingress/e2e-migrate-f1cd424d-gw-openshift-default-d98684879-5kkmq" Jul 18 12:19:09.568800 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:19:09.568749 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9eb24b37-ec67-41b4-a2f1-b69bcced3c4e-credential-socket\") pod \"e2e-migrate-f1cd424d-gw-openshift-default-d98684879-5kkmq\" (UID: \"9eb24b37-ec67-41b4-a2f1-b69bcced3c4e\") " pod="openshift-ingress/e2e-migrate-f1cd424d-gw-openshift-default-d98684879-5kkmq" Jul 18 12:19:09.568860 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:19:09.568808 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9eb24b37-ec67-41b4-a2f1-b69bcced3c4e-workload-certs\") pod \"e2e-migrate-f1cd424d-gw-openshift-default-d98684879-5kkmq\" (UID: \"9eb24b37-ec67-41b4-a2f1-b69bcced3c4e\") " pod="openshift-ingress/e2e-migrate-f1cd424d-gw-openshift-default-d98684879-5kkmq" Jul 18 12:19:09.568860 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:19:09.568849 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9eb24b37-ec67-41b4-a2f1-b69bcced3c4e-workload-socket\") pod \"e2e-migrate-f1cd424d-gw-openshift-default-d98684879-5kkmq\" (UID: \"9eb24b37-ec67-41b4-a2f1-b69bcced3c4e\") " pod="openshift-ingress/e2e-migrate-f1cd424d-gw-openshift-default-d98684879-5kkmq" Jul 18 12:19:09.568964 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:19:09.568875 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9eb24b37-ec67-41b4-a2f1-b69bcced3c4e-istio-data\") pod \"e2e-migrate-f1cd424d-gw-openshift-default-d98684879-5kkmq\" (UID: \"9eb24b37-ec67-41b4-a2f1-b69bcced3c4e\") " pod="openshift-ingress/e2e-migrate-f1cd424d-gw-openshift-default-d98684879-5kkmq" Jul 18 12:19:09.568964 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:19:09.568881 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5tmkb\" (UniqueName: \"kubernetes.io/projected/9eb24b37-ec67-41b4-a2f1-b69bcced3c4e-kube-api-access-5tmkb\") pod \"e2e-migrate-f1cd424d-gw-openshift-default-d98684879-5kkmq\" (UID: \"9eb24b37-ec67-41b4-a2f1-b69bcced3c4e\") " pod="openshift-ingress/e2e-migrate-f1cd424d-gw-openshift-default-d98684879-5kkmq" Jul 18 12:19:09.568964 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:19:09.568933 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9eb24b37-ec67-41b4-a2f1-b69bcced3c4e-istiod-ca-cert\") pod \"e2e-migrate-f1cd424d-gw-openshift-default-d98684879-5kkmq\" (UID: \"9eb24b37-ec67-41b4-a2f1-b69bcced3c4e\") " pod="openshift-ingress/e2e-migrate-f1cd424d-gw-openshift-default-d98684879-5kkmq" Jul 18 12:19:09.569130 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:19:09.569100 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9eb24b37-ec67-41b4-a2f1-b69bcced3c4e-workload-certs\") pod \"e2e-migrate-f1cd424d-gw-openshift-default-d98684879-5kkmq\" (UID: \"9eb24b37-ec67-41b4-a2f1-b69bcced3c4e\") " pod="openshift-ingress/e2e-migrate-f1cd424d-gw-openshift-default-d98684879-5kkmq" Jul 18 12:19:09.569189 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:19:09.569160 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9eb24b37-ec67-41b4-a2f1-b69bcced3c4e-credential-socket\") pod \"e2e-migrate-f1cd424d-gw-openshift-default-d98684879-5kkmq\" (UID: \"9eb24b37-ec67-41b4-a2f1-b69bcced3c4e\") " pod="openshift-ingress/e2e-migrate-f1cd424d-gw-openshift-default-d98684879-5kkmq" Jul 18 12:19:09.569582 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:19:09.569540 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9eb24b37-ec67-41b4-a2f1-b69bcced3c4e-istiod-ca-cert\") pod \"e2e-migrate-f1cd424d-gw-openshift-default-d98684879-5kkmq\" (UID: \"9eb24b37-ec67-41b4-a2f1-b69bcced3c4e\") " pod="openshift-ingress/e2e-migrate-f1cd424d-gw-openshift-default-d98684879-5kkmq" Jul 18 12:19:09.569699 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:19:09.569675 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9eb24b37-ec67-41b4-a2f1-b69bcced3c4e-workload-socket\") pod \"e2e-migrate-f1cd424d-gw-openshift-default-d98684879-5kkmq\" (UID: \"9eb24b37-ec67-41b4-a2f1-b69bcced3c4e\") " pod="openshift-ingress/e2e-migrate-f1cd424d-gw-openshift-default-d98684879-5kkmq" Jul 18 12:19:09.570987 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:19:09.570969 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9eb24b37-ec67-41b4-a2f1-b69bcced3c4e-istio-envoy\") pod \"e2e-migrate-f1cd424d-gw-openshift-default-d98684879-5kkmq\" (UID: \"9eb24b37-ec67-41b4-a2f1-b69bcced3c4e\") " pod="openshift-ingress/e2e-migrate-f1cd424d-gw-openshift-default-d98684879-5kkmq" Jul 18 12:19:09.571291 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:19:09.571269 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9eb24b37-ec67-41b4-a2f1-b69bcced3c4e-istio-podinfo\") pod \"e2e-migrate-f1cd424d-gw-openshift-default-d98684879-5kkmq\" (UID: \"9eb24b37-ec67-41b4-a2f1-b69bcced3c4e\") " pod="openshift-ingress/e2e-migrate-f1cd424d-gw-openshift-default-d98684879-5kkmq" Jul 18 12:19:09.576401 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:19:09.576376 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9eb24b37-ec67-41b4-a2f1-b69bcced3c4e-istio-token\") pod \"e2e-migrate-f1cd424d-gw-openshift-default-d98684879-5kkmq\" (UID: \"9eb24b37-ec67-41b4-a2f1-b69bcced3c4e\") " pod="openshift-ingress/e2e-migrate-f1cd424d-gw-openshift-default-d98684879-5kkmq" Jul 18 12:19:09.576530 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:19:09.576515 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5tmkb\" (UniqueName: \"kubernetes.io/projected/9eb24b37-ec67-41b4-a2f1-b69bcced3c4e-kube-api-access-5tmkb\") pod \"e2e-migrate-f1cd424d-gw-openshift-default-d98684879-5kkmq\" (UID: \"9eb24b37-ec67-41b4-a2f1-b69bcced3c4e\") " pod="openshift-ingress/e2e-migrate-f1cd424d-gw-openshift-default-d98684879-5kkmq" Jul 18 12:19:09.658326 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:19:09.658260 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-f1cd424d-gw-openshift-default-d98684879-5kkmq" Jul 18 12:19:09.798051 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:19:09.798028 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-f1cd424d-gw-openshift-default-d98684879-5kkmq"] Jul 18 12:19:09.799311 ip-10-0-136-240 kubenswrapper[2585]: W0718 12:19:09.799286 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9eb24b37_ec67_41b4_a2f1_b69bcced3c4e.slice/crio-5279a4b01f6e4090afa49ba761e967e77c5f4b92b66a6f682861ce3b948d785b WatchSource:0}: Error finding container 5279a4b01f6e4090afa49ba761e967e77c5f4b92b66a6f682861ce3b948d785b: Status 404 returned error can't find the container with id 5279a4b01f6e4090afa49ba761e967e77c5f4b92b66a6f682861ce3b948d785b Jul 18 12:19:09.801490 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:19:09.801463 2585 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 18 12:19:09.801587 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:19:09.801524 2585 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 18 12:19:09.801587 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:19:09.801574 2585 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 18 12:19:10.402992 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:19:10.402960 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-f1cd424d-gw-openshift-default-d98684879-5kkmq" event={"ID":"9eb24b37-ec67-41b4-a2f1-b69bcced3c4e","Type":"ContainerStarted","Data":"bbe6bc506612b272116811ab0f5d8e0fa3b80b3ab773c33f0ce51b4983127de9"} Jul 18 12:19:10.403399 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:19:10.403016 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-f1cd424d-gw-openshift-default-d98684879-5kkmq" event={"ID":"9eb24b37-ec67-41b4-a2f1-b69bcced3c4e","Type":"ContainerStarted","Data":"5279a4b01f6e4090afa49ba761e967e77c5f4b92b66a6f682861ce3b948d785b"} Jul 18 12:19:10.424429 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:19:10.424369 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-migrate-f1cd424d-gw-openshift-default-d98684879-5kkmq" podStartSLOduration=1.42435175 podStartE2EDuration="1.42435175s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-18 12:19:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-18 12:19:10.422683242 +0000 UTC m=+1817.502889622" watchObservedRunningTime="2026-07-18 12:19:10.42435175 +0000 UTC m=+1817.504558106" Jul 18 12:19:10.658855 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:19:10.658773 2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-f1cd424d-gw-openshift-default-d98684879-5kkmq" Jul 18 12:19:10.663761 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:19:10.663738 2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-f1cd424d-gw-openshift-default-d98684879-5kkmq" Jul 18 12:19:11.409422 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:19:11.409382 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-f1cd424d-gw-openshift-default-d98684879-5kkmq" Jul 18 12:19:11.410320 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:19:11.410301 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-f1cd424d-gw-openshift-default-d98684879-5kkmq" Jul 18 12:19:13.582435 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:19:13.582367 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-736936a8-openshift-default-9bb5769b5-9s8wl" Jul 18 12:19:13.715702 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:19:13.715668 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c19f06a1-ccff-4eb8-80a1-490d3ebef8e0-istio-envoy\") pod \"c19f06a1-ccff-4eb8-80a1-490d3ebef8e0\" (UID: \"c19f06a1-ccff-4eb8-80a1-490d3ebef8e0\") " Jul 18 12:19:13.715885 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:19:13.715732 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c19f06a1-ccff-4eb8-80a1-490d3ebef8e0-istio-data\") pod \"c19f06a1-ccff-4eb8-80a1-490d3ebef8e0\" (UID: \"c19f06a1-ccff-4eb8-80a1-490d3ebef8e0\") " Jul 18 12:19:13.715885 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:19:13.715769 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c19f06a1-ccff-4eb8-80a1-490d3ebef8e0-istio-token\") pod \"c19f06a1-ccff-4eb8-80a1-490d3ebef8e0\" (UID: \"c19f06a1-ccff-4eb8-80a1-490d3ebef8e0\") " Jul 18 12:19:13.715885 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:19:13.715836 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c19f06a1-ccff-4eb8-80a1-490d3ebef8e0-workload-socket\") pod \"c19f06a1-ccff-4eb8-80a1-490d3ebef8e0\" (UID: \"c19f06a1-ccff-4eb8-80a1-490d3ebef8e0\") " Jul 18 12:19:13.715885 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:19:13.715860 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c19f06a1-ccff-4eb8-80a1-490d3ebef8e0-istio-podinfo\") pod \"c19f06a1-ccff-4eb8-80a1-490d3ebef8e0\" (UID: \"c19f06a1-ccff-4eb8-80a1-490d3ebef8e0\") " Jul 18 12:19:13.716108 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:19:13.715898 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-t8mvh\" (UniqueName: \"kubernetes.io/projected/c19f06a1-ccff-4eb8-80a1-490d3ebef8e0-kube-api-access-t8mvh\") pod \"c19f06a1-ccff-4eb8-80a1-490d3ebef8e0\" (UID: \"c19f06a1-ccff-4eb8-80a1-490d3ebef8e0\") " Jul 18 12:19:13.716108 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:19:13.715976 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c19f06a1-ccff-4eb8-80a1-490d3ebef8e0-workload-certs\") pod \"c19f06a1-ccff-4eb8-80a1-490d3ebef8e0\" (UID: \"c19f06a1-ccff-4eb8-80a1-490d3ebef8e0\") " Jul 18 12:19:13.716108 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:19:13.716052 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c19f06a1-ccff-4eb8-80a1-490d3ebef8e0-credential-socket\") pod \"c19f06a1-ccff-4eb8-80a1-490d3ebef8e0\" (UID: \"c19f06a1-ccff-4eb8-80a1-490d3ebef8e0\") " Jul 18 12:19:13.716108 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:19:13.716097 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c19f06a1-ccff-4eb8-80a1-490d3ebef8e0-istiod-ca-cert\") pod \"c19f06a1-ccff-4eb8-80a1-490d3ebef8e0\" (UID: \"c19f06a1-ccff-4eb8-80a1-490d3ebef8e0\") " Jul 18 12:19:13.716309 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:19:13.716251 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c19f06a1-ccff-4eb8-80a1-490d3ebef8e0-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "c19f06a1-ccff-4eb8-80a1-490d3ebef8e0" (UID: "c19f06a1-ccff-4eb8-80a1-490d3ebef8e0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 18 12:19:13.716309 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:19:13.716285 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c19f06a1-ccff-4eb8-80a1-490d3ebef8e0-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "c19f06a1-ccff-4eb8-80a1-490d3ebef8e0" (UID: "c19f06a1-ccff-4eb8-80a1-490d3ebef8e0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 18 12:19:13.716537 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:19:13.716512 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c19f06a1-ccff-4eb8-80a1-490d3ebef8e0-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "c19f06a1-ccff-4eb8-80a1-490d3ebef8e0" (UID: "c19f06a1-ccff-4eb8-80a1-490d3ebef8e0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 18 12:19:13.716658 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:19:13.716515 2585 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c19f06a1-ccff-4eb8-80a1-490d3ebef8e0-workload-socket\") on node \"ip-10-0-136-240.ec2.internal\" DevicePath \"\"" Jul 18 12:19:13.716658 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:19:13.716582 2585 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c19f06a1-ccff-4eb8-80a1-490d3ebef8e0-workload-certs\") on node \"ip-10-0-136-240.ec2.internal\" DevicePath \"\"" Jul 18 12:19:13.716658 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:19:13.716649 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c19f06a1-ccff-4eb8-80a1-490d3ebef8e0-istio-data" (OuterVolumeSpecName: "istio-data") pod "c19f06a1-ccff-4eb8-80a1-490d3ebef8e0" (UID: "c19f06a1-ccff-4eb8-80a1-490d3ebef8e0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 18 12:19:13.716881 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:19:13.716856 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c19f06a1-ccff-4eb8-80a1-490d3ebef8e0-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "c19f06a1-ccff-4eb8-80a1-490d3ebef8e0" (UID: "c19f06a1-ccff-4eb8-80a1-490d3ebef8e0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 18 12:19:13.718391 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:19:13.718366 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c19f06a1-ccff-4eb8-80a1-490d3ebef8e0-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "c19f06a1-ccff-4eb8-80a1-490d3ebef8e0" (UID: "c19f06a1-ccff-4eb8-80a1-490d3ebef8e0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 18 12:19:13.718534 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:19:13.718516 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c19f06a1-ccff-4eb8-80a1-490d3ebef8e0-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "c19f06a1-ccff-4eb8-80a1-490d3ebef8e0" (UID: "c19f06a1-ccff-4eb8-80a1-490d3ebef8e0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 18 12:19:13.718702 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:19:13.718596 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c19f06a1-ccff-4eb8-80a1-490d3ebef8e0-istio-token" (OuterVolumeSpecName: "istio-token") pod "c19f06a1-ccff-4eb8-80a1-490d3ebef8e0" (UID: "c19f06a1-ccff-4eb8-80a1-490d3ebef8e0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 18 12:19:13.718812 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:19:13.718795 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c19f06a1-ccff-4eb8-80a1-490d3ebef8e0-kube-api-access-t8mvh" (OuterVolumeSpecName: "kube-api-access-t8mvh") pod "c19f06a1-ccff-4eb8-80a1-490d3ebef8e0" (UID: "c19f06a1-ccff-4eb8-80a1-490d3ebef8e0"). InnerVolumeSpecName "kube-api-access-t8mvh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 18 12:19:13.817889 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:19:13.817867 2585 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c19f06a1-ccff-4eb8-80a1-490d3ebef8e0-istio-data\") on node \"ip-10-0-136-240.ec2.internal\" DevicePath \"\"" Jul 18 12:19:13.817889 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:19:13.817886 2585 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c19f06a1-ccff-4eb8-80a1-490d3ebef8e0-istio-token\") on node \"ip-10-0-136-240.ec2.internal\" DevicePath \"\"" Jul 18 12:19:13.817889 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:19:13.817894 2585 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c19f06a1-ccff-4eb8-80a1-490d3ebef8e0-istio-podinfo\") on node \"ip-10-0-136-240.ec2.internal\" DevicePath \"\"" Jul 18 12:19:13.818050 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:19:13.817903 2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t8mvh\" (UniqueName: \"kubernetes.io/projected/c19f06a1-ccff-4eb8-80a1-490d3ebef8e0-kube-api-access-t8mvh\") on node \"ip-10-0-136-240.ec2.internal\" DevicePath \"\"" Jul 18 12:19:13.818050 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:19:13.817911 2585 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c19f06a1-ccff-4eb8-80a1-490d3ebef8e0-credential-socket\") on node \"ip-10-0-136-240.ec2.internal\" DevicePath \"\"" Jul 18 12:19:13.818050 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:19:13.817920 2585 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c19f06a1-ccff-4eb8-80a1-490d3ebef8e0-istiod-ca-cert\") on node \"ip-10-0-136-240.ec2.internal\" DevicePath \"\"" Jul 18 12:19:13.818050 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:19:13.817929 2585 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c19f06a1-ccff-4eb8-80a1-490d3ebef8e0-istio-envoy\") on node \"ip-10-0-136-240.ec2.internal\" DevicePath \"\"" Jul 18 12:19:14.426115 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:19:14.426081 2585 generic.go:356] "Generic (PLEG): container finished" podID="c19f06a1-ccff-4eb8-80a1-490d3ebef8e0" containerID="bd71606e5804257d27b54ee4855b362e91386d87380091419d442fd2078b2773" exitCode=0 Jul 18 12:19:14.426288 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:19:14.426122 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-736936a8-openshift-default-9bb5769b5-9s8wl" event={"ID":"c19f06a1-ccff-4eb8-80a1-490d3ebef8e0","Type":"ContainerDied","Data":"bd71606e5804257d27b54ee4855b362e91386d87380091419d442fd2078b2773"} Jul 18 12:19:14.426288 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:19:14.426125 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-736936a8-openshift-default-9bb5769b5-9s8wl" Jul 18 12:19:14.426288 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:19:14.426169 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-736936a8-openshift-default-9bb5769b5-9s8wl" event={"ID":"c19f06a1-ccff-4eb8-80a1-490d3ebef8e0","Type":"ContainerDied","Data":"6ae634be34e00260373c1314804bef9f4452c298e161dfe7da5594d6472eb838"} Jul 18 12:19:14.426288 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:19:14.426195 2585 scope.go:117] "RemoveContainer" containerID="bd71606e5804257d27b54ee4855b362e91386d87380091419d442fd2078b2773" Jul 18 12:19:14.435777 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:19:14.435759 2585 scope.go:117] "RemoveContainer" containerID="c651510a3b92f30364c885b435d98c9b082d0d5447f8093e50230aadfd2321d1" Jul 18 12:19:14.443979 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:19:14.443874 2585 scope.go:117] "RemoveContainer" containerID="bd71606e5804257d27b54ee4855b362e91386d87380091419d442fd2078b2773" Jul 18 12:19:14.444134 ip-10-0-136-240 kubenswrapper[2585]: E0718 12:19:14.444115 2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bd71606e5804257d27b54ee4855b362e91386d87380091419d442fd2078b2773\": container with ID starting with bd71606e5804257d27b54ee4855b362e91386d87380091419d442fd2078b2773 not found: ID does not exist" containerID="bd71606e5804257d27b54ee4855b362e91386d87380091419d442fd2078b2773" Jul 18 12:19:14.444171 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:19:14.444144 2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bd71606e5804257d27b54ee4855b362e91386d87380091419d442fd2078b2773"} err="failed to get container status \"bd71606e5804257d27b54ee4855b362e91386d87380091419d442fd2078b2773\": rpc error: code = NotFound desc = could not find container \"bd71606e5804257d27b54ee4855b362e91386d87380091419d442fd2078b2773\": container with ID starting with bd71606e5804257d27b54ee4855b362e91386d87380091419d442fd2078b2773 not found: ID does not exist" Jul 18 12:19:14.444171 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:19:14.444160 2585 scope.go:117] "RemoveContainer" containerID="c651510a3b92f30364c885b435d98c9b082d0d5447f8093e50230aadfd2321d1" Jul 18 12:19:14.444377 ip-10-0-136-240 kubenswrapper[2585]: E0718 12:19:14.444362 2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c651510a3b92f30364c885b435d98c9b082d0d5447f8093e50230aadfd2321d1\": container with ID starting with c651510a3b92f30364c885b435d98c9b082d0d5447f8093e50230aadfd2321d1 not found: ID does not exist" containerID="c651510a3b92f30364c885b435d98c9b082d0d5447f8093e50230aadfd2321d1" Jul 18 12:19:14.444411 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:19:14.444384 2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c651510a3b92f30364c885b435d98c9b082d0d5447f8093e50230aadfd2321d1"} err="failed to get container status \"c651510a3b92f30364c885b435d98c9b082d0d5447f8093e50230aadfd2321d1\": rpc error: code = NotFound desc = could not find container \"c651510a3b92f30364c885b435d98c9b082d0d5447f8093e50230aadfd2321d1\": container with ID starting with c651510a3b92f30364c885b435d98c9b082d0d5447f8093e50230aadfd2321d1 not found: ID does not exist" Jul 18 12:19:14.447196 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:19:14.447175 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-736936a8-openshift-default-9bb5769b5-9s8wl"] Jul 18 12:19:14.451822 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:19:14.451793 2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-736936a8-openshift-default-9bb5769b5-9s8wl"] Jul 18 12:19:15.469284 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:19:15.469249 2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c19f06a1-ccff-4eb8-80a1-490d3ebef8e0" path="/var/lib/kubelet/pods/c19f06a1-ccff-4eb8-80a1-490d3ebef8e0/volumes" Jul 18 12:19:25.483038 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:19:25.483006 2585 generic.go:356] "Generic (PLEG): container finished" podID="9eb24b37-ec67-41b4-a2f1-b69bcced3c4e" containerID="bbe6bc506612b272116811ab0f5d8e0fa3b80b3ab773c33f0ce51b4983127de9" exitCode=0 Jul 18 12:19:25.483447 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:19:25.483078 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-f1cd424d-gw-openshift-default-d98684879-5kkmq" event={"ID":"9eb24b37-ec67-41b4-a2f1-b69bcced3c4e","Type":"ContainerDied","Data":"bbe6bc506612b272116811ab0f5d8e0fa3b80b3ab773c33f0ce51b4983127de9"} Jul 18 12:19:25.483447 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:19:25.483441 2585 scope.go:117] "RemoveContainer" containerID="bbe6bc506612b272116811ab0f5d8e0fa3b80b3ab773c33f0ce51b4983127de9" Jul 18 12:19:25.484476 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:19:25.484451 2585 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 18 12:19:25.484562 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:19:25.484531 2585 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 18 12:19:25.484624 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:19:25.484608 2585 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 18 12:19:26.410240 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:19:26.410208 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-f1cd424d-gw-openshift-default-d98684879-5kkmq" Jul 18 12:19:26.494212 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:19:26.494173 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-f1cd424d-gw-openshift-default-d98684879-5kkmq" event={"ID":"9eb24b37-ec67-41b4-a2f1-b69bcced3c4e","Type":"ContainerStarted","Data":"ab7dbd75dd93080aaa4d542256e9a0529a87d1b409ca700c79906671b1a4d8eb"} Jul 18 12:19:26.658676 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:19:26.658637 2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-f1cd424d-gw-openshift-default-d98684879-5kkmq" Jul 18 12:19:26.660004 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:19:26.659975 2585 patch_prober.go:28] interesting pod/e2e-migrate-f1cd424d-gw-openshift-default-d98684879-5kkmq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" start-of-body= Jul 18 12:19:26.660145 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:19:26.660034 2585 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-f1cd424d-gw-openshift-default-d98684879-5kkmq" podUID="9eb24b37-ec67-41b4-a2f1-b69bcced3c4e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" Jul 18 12:19:27.658896 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:19:27.658856 2585 patch_prober.go:28] interesting pod/e2e-migrate-f1cd424d-gw-openshift-default-d98684879-5kkmq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" start-of-body= Jul 18 12:19:27.659273 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:19:27.658915 2585 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-f1cd424d-gw-openshift-default-d98684879-5kkmq" podUID="9eb24b37-ec67-41b4-a2f1-b69bcced3c4e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" Jul 18 12:19:28.659131 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:19:28.659100 2585 patch_prober.go:28] interesting pod/e2e-migrate-f1cd424d-gw-openshift-default-d98684879-5kkmq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" start-of-body= Jul 18 12:19:28.659530 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:19:28.659157 2585 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-f1cd424d-gw-openshift-default-d98684879-5kkmq" podUID="9eb24b37-ec67-41b4-a2f1-b69bcced3c4e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" Jul 18 12:19:29.658855 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:19:29.658821 2585 patch_prober.go:28] interesting pod/e2e-migrate-f1cd424d-gw-openshift-default-d98684879-5kkmq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" start-of-body= Jul 18 12:19:29.659032 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:19:29.658873 2585 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-f1cd424d-gw-openshift-default-d98684879-5kkmq" podUID="9eb24b37-ec67-41b4-a2f1-b69bcced3c4e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" Jul 18 12:19:30.659310 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:19:30.659268 2585 patch_prober.go:28] interesting pod/e2e-migrate-f1cd424d-gw-openshift-default-d98684879-5kkmq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" start-of-body= Jul 18 12:19:30.659850 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:19:30.659342 2585 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-f1cd424d-gw-openshift-default-d98684879-5kkmq" podUID="9eb24b37-ec67-41b4-a2f1-b69bcced3c4e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" Jul 18 12:19:31.663350 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:19:31.663322 2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-f1cd424d-gw-openshift-default-d98684879-5kkmq" Jul 18 12:19:31.663862 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:19:31.663718 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-f1cd424d-gw-openshift-default-d98684879-5kkmq" Jul 18 12:19:31.664321 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:19:31.664296 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-f1cd424d-gw-openshift-default-d98684879-5kkmq" Jul 18 12:19:52.088440 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:19:52.088405 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-f1cd424d-gw-openshift-default-d98684879-5kkmq"] Jul 18 12:19:52.088976 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:19:52.088649 2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-migrate-f1cd424d-gw-openshift-default-d98684879-5kkmq" podUID="9eb24b37-ec67-41b4-a2f1-b69bcced3c4e" containerName="istio-proxy" containerID="cri-o://ab7dbd75dd93080aaa4d542256e9a0529a87d1b409ca700c79906671b1a4d8eb" gracePeriod=30 Jul 18 12:19:52.838866 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:19:52.838829 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-3dff9e6d-openshift-default-74d94fd969-4nnkx"] Jul 18 12:19:52.839219 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:19:52.839206 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c19f06a1-ccff-4eb8-80a1-490d3ebef8e0" containerName="istio-proxy" Jul 18 12:19:52.839265 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:19:52.839221 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="c19f06a1-ccff-4eb8-80a1-490d3ebef8e0" containerName="istio-proxy" Jul 18 12:19:52.839265 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:19:52.839247 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c19f06a1-ccff-4eb8-80a1-490d3ebef8e0" containerName="istio-proxy" Jul 18 12:19:52.839265 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:19:52.839253 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="c19f06a1-ccff-4eb8-80a1-490d3ebef8e0" containerName="istio-proxy" Jul 18 12:19:52.839359 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:19:52.839310 2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="c19f06a1-ccff-4eb8-80a1-490d3ebef8e0" containerName="istio-proxy" Jul 18 12:19:52.839359 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:19:52.839320 2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="c19f06a1-ccff-4eb8-80a1-490d3ebef8e0" containerName="istio-proxy" Jul 18 12:19:52.842604 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:19:52.842585 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-3dff9e6d-openshift-default-74d94fd969-4nnkx" Jul 18 12:19:52.844971 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:19:52.844944 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-3dff9e6d-openshift-default-dockercfg-cbtgf\"" Jul 18 12:19:52.854678 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:19:52.854655 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-3dff9e6d-openshift-default-74d94fd969-4nnkx"] Jul 18 12:19:52.986077 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:19:52.986044 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/49cac5e7-2a6f-493d-b280-31637879b57e-workload-socket\") pod \"e2e-ait-3dff9e6d-openshift-default-74d94fd969-4nnkx\" (UID: \"49cac5e7-2a6f-493d-b280-31637879b57e\") " pod="openshift-ingress/e2e-ait-3dff9e6d-openshift-default-74d94fd969-4nnkx" Jul 18 12:19:52.986077 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:19:52.986077 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/49cac5e7-2a6f-493d-b280-31637879b57e-istio-data\") pod \"e2e-ait-3dff9e6d-openshift-default-74d94fd969-4nnkx\" (UID: \"49cac5e7-2a6f-493d-b280-31637879b57e\") " pod="openshift-ingress/e2e-ait-3dff9e6d-openshift-default-74d94fd969-4nnkx" Jul 18 12:19:52.986326 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:19:52.986099 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/49cac5e7-2a6f-493d-b280-31637879b57e-istio-envoy\") pod \"e2e-ait-3dff9e6d-openshift-default-74d94fd969-4nnkx\" (UID: \"49cac5e7-2a6f-493d-b280-31637879b57e\") " pod="openshift-ingress/e2e-ait-3dff9e6d-openshift-default-74d94fd969-4nnkx" Jul 18 12:19:52.986326 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:19:52.986184 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/49cac5e7-2a6f-493d-b280-31637879b57e-istio-token\") pod \"e2e-ait-3dff9e6d-openshift-default-74d94fd969-4nnkx\" (UID: \"49cac5e7-2a6f-493d-b280-31637879b57e\") " pod="openshift-ingress/e2e-ait-3dff9e6d-openshift-default-74d94fd969-4nnkx" Jul 18 12:19:52.986326 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:19:52.986230 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/49cac5e7-2a6f-493d-b280-31637879b57e-credential-socket\") pod \"e2e-ait-3dff9e6d-openshift-default-74d94fd969-4nnkx\" (UID: \"49cac5e7-2a6f-493d-b280-31637879b57e\") " pod="openshift-ingress/e2e-ait-3dff9e6d-openshift-default-74d94fd969-4nnkx" Jul 18 12:19:52.986326 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:19:52.986262 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/49cac5e7-2a6f-493d-b280-31637879b57e-istiod-ca-cert\") pod \"e2e-ait-3dff9e6d-openshift-default-74d94fd969-4nnkx\" (UID: \"49cac5e7-2a6f-493d-b280-31637879b57e\") " pod="openshift-ingress/e2e-ait-3dff9e6d-openshift-default-74d94fd969-4nnkx" Jul 18 12:19:52.986545 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:19:52.986372 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/49cac5e7-2a6f-493d-b280-31637879b57e-istio-podinfo\") pod \"e2e-ait-3dff9e6d-openshift-default-74d94fd969-4nnkx\" (UID: \"49cac5e7-2a6f-493d-b280-31637879b57e\") " pod="openshift-ingress/e2e-ait-3dff9e6d-openshift-default-74d94fd969-4nnkx" Jul 18 12:19:52.986545 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:19:52.986421 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/49cac5e7-2a6f-493d-b280-31637879b57e-workload-certs\") pod \"e2e-ait-3dff9e6d-openshift-default-74d94fd969-4nnkx\" (UID: \"49cac5e7-2a6f-493d-b280-31637879b57e\") " pod="openshift-ingress/e2e-ait-3dff9e6d-openshift-default-74d94fd969-4nnkx" Jul 18 12:19:52.986545 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:19:52.986475 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w9mdk\" (UniqueName: \"kubernetes.io/projected/49cac5e7-2a6f-493d-b280-31637879b57e-kube-api-access-w9mdk\") pod \"e2e-ait-3dff9e6d-openshift-default-74d94fd969-4nnkx\" (UID: \"49cac5e7-2a6f-493d-b280-31637879b57e\") " pod="openshift-ingress/e2e-ait-3dff9e6d-openshift-default-74d94fd969-4nnkx" Jul 18 12:19:53.087378 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:19:53.087342 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/49cac5e7-2a6f-493d-b280-31637879b57e-istio-podinfo\") pod \"e2e-ait-3dff9e6d-openshift-default-74d94fd969-4nnkx\" (UID: \"49cac5e7-2a6f-493d-b280-31637879b57e\") " pod="openshift-ingress/e2e-ait-3dff9e6d-openshift-default-74d94fd969-4nnkx" Jul 18 12:19:53.087606 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:19:53.087413 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/49cac5e7-2a6f-493d-b280-31637879b57e-workload-certs\") pod \"e2e-ait-3dff9e6d-openshift-default-74d94fd969-4nnkx\" (UID: \"49cac5e7-2a6f-493d-b280-31637879b57e\") " pod="openshift-ingress/e2e-ait-3dff9e6d-openshift-default-74d94fd969-4nnkx" Jul 18 12:19:53.087606 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:19:53.087456 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w9mdk\" (UniqueName: \"kubernetes.io/projected/49cac5e7-2a6f-493d-b280-31637879b57e-kube-api-access-w9mdk\") pod \"e2e-ait-3dff9e6d-openshift-default-74d94fd969-4nnkx\" (UID: \"49cac5e7-2a6f-493d-b280-31637879b57e\") " pod="openshift-ingress/e2e-ait-3dff9e6d-openshift-default-74d94fd969-4nnkx" Jul 18 12:19:53.087606 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:19:53.087511 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/49cac5e7-2a6f-493d-b280-31637879b57e-workload-socket\") pod \"e2e-ait-3dff9e6d-openshift-default-74d94fd969-4nnkx\" (UID: \"49cac5e7-2a6f-493d-b280-31637879b57e\") " pod="openshift-ingress/e2e-ait-3dff9e6d-openshift-default-74d94fd969-4nnkx" Jul 18 12:19:53.087606 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:19:53.087539 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/49cac5e7-2a6f-493d-b280-31637879b57e-istio-data\") pod \"e2e-ait-3dff9e6d-openshift-default-74d94fd969-4nnkx\" (UID: \"49cac5e7-2a6f-493d-b280-31637879b57e\") " pod="openshift-ingress/e2e-ait-3dff9e6d-openshift-default-74d94fd969-4nnkx" Jul 18 12:19:53.087606 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:19:53.087603 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/49cac5e7-2a6f-493d-b280-31637879b57e-istio-envoy\") pod \"e2e-ait-3dff9e6d-openshift-default-74d94fd969-4nnkx\" (UID: \"49cac5e7-2a6f-493d-b280-31637879b57e\") " pod="openshift-ingress/e2e-ait-3dff9e6d-openshift-default-74d94fd969-4nnkx" Jul 18 12:19:53.087887 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:19:53.087638 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/49cac5e7-2a6f-493d-b280-31637879b57e-istio-token\") pod \"e2e-ait-3dff9e6d-openshift-default-74d94fd969-4nnkx\" (UID: \"49cac5e7-2a6f-493d-b280-31637879b57e\") " pod="openshift-ingress/e2e-ait-3dff9e6d-openshift-default-74d94fd969-4nnkx" Jul 18 12:19:53.087887 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:19:53.087677 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/49cac5e7-2a6f-493d-b280-31637879b57e-credential-socket\") pod \"e2e-ait-3dff9e6d-openshift-default-74d94fd969-4nnkx\" (UID: \"49cac5e7-2a6f-493d-b280-31637879b57e\") " pod="openshift-ingress/e2e-ait-3dff9e6d-openshift-default-74d94fd969-4nnkx" Jul 18 12:19:53.087887 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:19:53.087707 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/49cac5e7-2a6f-493d-b280-31637879b57e-istiod-ca-cert\") pod \"e2e-ait-3dff9e6d-openshift-default-74d94fd969-4nnkx\" (UID: \"49cac5e7-2a6f-493d-b280-31637879b57e\") " pod="openshift-ingress/e2e-ait-3dff9e6d-openshift-default-74d94fd969-4nnkx" Jul 18 12:19:53.088059 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:19:53.087883 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/49cac5e7-2a6f-493d-b280-31637879b57e-workload-certs\") pod \"e2e-ait-3dff9e6d-openshift-default-74d94fd969-4nnkx\" (UID: \"49cac5e7-2a6f-493d-b280-31637879b57e\") " pod="openshift-ingress/e2e-ait-3dff9e6d-openshift-default-74d94fd969-4nnkx" Jul 18 12:19:53.088213 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:19:53.088187 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/49cac5e7-2a6f-493d-b280-31637879b57e-istio-data\") pod \"e2e-ait-3dff9e6d-openshift-default-74d94fd969-4nnkx\" (UID: \"49cac5e7-2a6f-493d-b280-31637879b57e\") " pod="openshift-ingress/e2e-ait-3dff9e6d-openshift-default-74d94fd969-4nnkx" Jul 18 12:19:53.088422 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:19:53.088365 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/49cac5e7-2a6f-493d-b280-31637879b57e-istiod-ca-cert\") pod \"e2e-ait-3dff9e6d-openshift-default-74d94fd969-4nnkx\" (UID: \"49cac5e7-2a6f-493d-b280-31637879b57e\") " pod="openshift-ingress/e2e-ait-3dff9e6d-openshift-default-74d94fd969-4nnkx" Jul 18 12:19:53.088658 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:19:53.088634 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/49cac5e7-2a6f-493d-b280-31637879b57e-credential-socket\") pod \"e2e-ait-3dff9e6d-openshift-default-74d94fd969-4nnkx\" (UID: \"49cac5e7-2a6f-493d-b280-31637879b57e\") " pod="openshift-ingress/e2e-ait-3dff9e6d-openshift-default-74d94fd969-4nnkx" Jul 18 12:19:53.089112 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:19:53.088879 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/49cac5e7-2a6f-493d-b280-31637879b57e-workload-socket\") pod \"e2e-ait-3dff9e6d-openshift-default-74d94fd969-4nnkx\" (UID: \"49cac5e7-2a6f-493d-b280-31637879b57e\") " pod="openshift-ingress/e2e-ait-3dff9e6d-openshift-default-74d94fd969-4nnkx" Jul 18 12:19:53.091096 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:19:53.091068 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/49cac5e7-2a6f-493d-b280-31637879b57e-istio-podinfo\") pod \"e2e-ait-3dff9e6d-openshift-default-74d94fd969-4nnkx\" (UID: \"49cac5e7-2a6f-493d-b280-31637879b57e\") " pod="openshift-ingress/e2e-ait-3dff9e6d-openshift-default-74d94fd969-4nnkx" Jul 18 12:19:53.091400 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:19:53.091381 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/49cac5e7-2a6f-493d-b280-31637879b57e-istio-envoy\") pod \"e2e-ait-3dff9e6d-openshift-default-74d94fd969-4nnkx\" (UID: \"49cac5e7-2a6f-493d-b280-31637879b57e\") " pod="openshift-ingress/e2e-ait-3dff9e6d-openshift-default-74d94fd969-4nnkx" Jul 18 12:19:53.095895 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:19:53.095877 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/49cac5e7-2a6f-493d-b280-31637879b57e-istio-token\") pod \"e2e-ait-3dff9e6d-openshift-default-74d94fd969-4nnkx\" (UID: \"49cac5e7-2a6f-493d-b280-31637879b57e\") " pod="openshift-ingress/e2e-ait-3dff9e6d-openshift-default-74d94fd969-4nnkx" Jul 18 12:19:53.095989 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:19:53.095957 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w9mdk\" (UniqueName: \"kubernetes.io/projected/49cac5e7-2a6f-493d-b280-31637879b57e-kube-api-access-w9mdk\") pod \"e2e-ait-3dff9e6d-openshift-default-74d94fd969-4nnkx\" (UID: \"49cac5e7-2a6f-493d-b280-31637879b57e\") " pod="openshift-ingress/e2e-ait-3dff9e6d-openshift-default-74d94fd969-4nnkx" Jul 18 12:19:53.153453 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:19:53.153429 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-3dff9e6d-openshift-default-74d94fd969-4nnkx" Jul 18 12:19:53.299370 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:19:53.299342 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-3dff9e6d-openshift-default-74d94fd969-4nnkx"] Jul 18 12:19:53.300818 ip-10-0-136-240 kubenswrapper[2585]: W0718 12:19:53.300783 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod49cac5e7_2a6f_493d_b280_31637879b57e.slice/crio-ea716dbd1b4134c92136be4966444c64b9f6163d73b1fa350266c94dfdf5e7de WatchSource:0}: Error finding container ea716dbd1b4134c92136be4966444c64b9f6163d73b1fa350266c94dfdf5e7de: Status 404 returned error can't find the container with id ea716dbd1b4134c92136be4966444c64b9f6163d73b1fa350266c94dfdf5e7de Jul 18 12:19:53.302980 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:19:53.302949 2585 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 18 12:19:53.303080 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:19:53.303019 2585 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 18 12:19:53.303080 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:19:53.303048 2585 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 18 12:19:53.635089 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:19:53.635007 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-3dff9e6d-openshift-default-74d94fd969-4nnkx" event={"ID":"49cac5e7-2a6f-493d-b280-31637879b57e","Type":"ContainerStarted","Data":"862d4d88f6ee7ed56a31d607e07d6ff94f60f14a23006311e02012cc21e06baf"} Jul 18 12:19:53.635089 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:19:53.635064 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-3dff9e6d-openshift-default-74d94fd969-4nnkx" event={"ID":"49cac5e7-2a6f-493d-b280-31637879b57e","Type":"ContainerStarted","Data":"ea716dbd1b4134c92136be4966444c64b9f6163d73b1fa350266c94dfdf5e7de"} Jul 18 12:19:53.657915 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:19:53.657851 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-3dff9e6d-openshift-default-74d94fd969-4nnkx" podStartSLOduration=1.657833245 podStartE2EDuration="1.657833245s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-18 12:19:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-18 12:19:53.655236072 +0000 UTC m=+1860.735442430" watchObservedRunningTime="2026-07-18 12:19:53.657833245 +0000 UTC m=+1860.738039602" Jul 18 12:19:54.153959 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:19:54.153926 2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-3dff9e6d-openshift-default-74d94fd969-4nnkx" Jul 18 12:19:54.159742 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:19:54.159706 2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-3dff9e6d-openshift-default-74d94fd969-4nnkx" Jul 18 12:19:54.640827 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:19:54.640797 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-3dff9e6d-openshift-default-74d94fd969-4nnkx" Jul 18 12:19:54.641660 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:19:54.641640 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-3dff9e6d-openshift-default-74d94fd969-4nnkx" Jul 18 12:19:57.347295 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:19:57.347237 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-f1cd424d-gw-openshift-default-d98684879-5kkmq" Jul 18 12:19:57.428093 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:19:57.428063 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9eb24b37-ec67-41b4-a2f1-b69bcced3c4e-workload-certs\") pod \"9eb24b37-ec67-41b4-a2f1-b69bcced3c4e\" (UID: \"9eb24b37-ec67-41b4-a2f1-b69bcced3c4e\") " Jul 18 12:19:57.428260 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:19:57.428109 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9eb24b37-ec67-41b4-a2f1-b69bcced3c4e-istio-token\") pod \"9eb24b37-ec67-41b4-a2f1-b69bcced3c4e\" (UID: \"9eb24b37-ec67-41b4-a2f1-b69bcced3c4e\") " Jul 18 12:19:57.428260 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:19:57.428134 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9eb24b37-ec67-41b4-a2f1-b69bcced3c4e-istio-data\") pod \"9eb24b37-ec67-41b4-a2f1-b69bcced3c4e\" (UID: \"9eb24b37-ec67-41b4-a2f1-b69bcced3c4e\") " Jul 18 12:19:57.428260 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:19:57.428157 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9eb24b37-ec67-41b4-a2f1-b69bcced3c4e-istio-envoy\") pod \"9eb24b37-ec67-41b4-a2f1-b69bcced3c4e\" (UID: \"9eb24b37-ec67-41b4-a2f1-b69bcced3c4e\") " Jul 18 12:19:57.428260 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:19:57.428183 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9eb24b37-ec67-41b4-a2f1-b69bcced3c4e-istiod-ca-cert\") pod \"9eb24b37-ec67-41b4-a2f1-b69bcced3c4e\" (UID: \"9eb24b37-ec67-41b4-a2f1-b69bcced3c4e\") " Jul 18 12:19:57.428260 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:19:57.428211 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9eb24b37-ec67-41b4-a2f1-b69bcced3c4e-istio-podinfo\") pod \"9eb24b37-ec67-41b4-a2f1-b69bcced3c4e\" (UID: \"9eb24b37-ec67-41b4-a2f1-b69bcced3c4e\") " Jul 18 12:19:57.428260 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:19:57.428232 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5tmkb\" (UniqueName: \"kubernetes.io/projected/9eb24b37-ec67-41b4-a2f1-b69bcced3c4e-kube-api-access-5tmkb\") pod \"9eb24b37-ec67-41b4-a2f1-b69bcced3c4e\" (UID: \"9eb24b37-ec67-41b4-a2f1-b69bcced3c4e\") " Jul 18 12:19:57.428576 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:19:57.428350 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9eb24b37-ec67-41b4-a2f1-b69bcced3c4e-workload-socket\") pod \"9eb24b37-ec67-41b4-a2f1-b69bcced3c4e\" (UID: \"9eb24b37-ec67-41b4-a2f1-b69bcced3c4e\") " Jul 18 12:19:57.428576 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:19:57.428360 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9eb24b37-ec67-41b4-a2f1-b69bcced3c4e-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "9eb24b37-ec67-41b4-a2f1-b69bcced3c4e" (UID: "9eb24b37-ec67-41b4-a2f1-b69bcced3c4e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 18 12:19:57.428576 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:19:57.428409 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9eb24b37-ec67-41b4-a2f1-b69bcced3c4e-credential-socket\") pod \"9eb24b37-ec67-41b4-a2f1-b69bcced3c4e\" (UID: \"9eb24b37-ec67-41b4-a2f1-b69bcced3c4e\") " Jul 18 12:19:57.428800 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:19:57.428779 2585 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9eb24b37-ec67-41b4-a2f1-b69bcced3c4e-workload-certs\") on node \"ip-10-0-136-240.ec2.internal\" DevicePath \"\"" Jul 18 12:19:57.428879 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:19:57.428836 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9eb24b37-ec67-41b4-a2f1-b69bcced3c4e-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "9eb24b37-ec67-41b4-a2f1-b69bcced3c4e" (UID: "9eb24b37-ec67-41b4-a2f1-b69bcced3c4e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 18 12:19:57.429014 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:19:57.428992 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9eb24b37-ec67-41b4-a2f1-b69bcced3c4e-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "9eb24b37-ec67-41b4-a2f1-b69bcced3c4e" (UID: "9eb24b37-ec67-41b4-a2f1-b69bcced3c4e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 18 12:19:57.429123 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:19:57.429099 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9eb24b37-ec67-41b4-a2f1-b69bcced3c4e-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "9eb24b37-ec67-41b4-a2f1-b69bcced3c4e" (UID: "9eb24b37-ec67-41b4-a2f1-b69bcced3c4e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 18 12:19:57.429411 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:19:57.429386 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9eb24b37-ec67-41b4-a2f1-b69bcced3c4e-istio-data" (OuterVolumeSpecName: "istio-data") pod "9eb24b37-ec67-41b4-a2f1-b69bcced3c4e" (UID: "9eb24b37-ec67-41b4-a2f1-b69bcced3c4e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 18 12:19:57.430494 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:19:57.430470 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9eb24b37-ec67-41b4-a2f1-b69bcced3c4e-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "9eb24b37-ec67-41b4-a2f1-b69bcced3c4e" (UID: "9eb24b37-ec67-41b4-a2f1-b69bcced3c4e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 18 12:19:57.430730 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:19:57.430713 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9eb24b37-ec67-41b4-a2f1-b69bcced3c4e-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "9eb24b37-ec67-41b4-a2f1-b69bcced3c4e" (UID: "9eb24b37-ec67-41b4-a2f1-b69bcced3c4e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 18 12:19:57.430855 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:19:57.430837 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9eb24b37-ec67-41b4-a2f1-b69bcced3c4e-kube-api-access-5tmkb" (OuterVolumeSpecName: "kube-api-access-5tmkb") pod "9eb24b37-ec67-41b4-a2f1-b69bcced3c4e" (UID: "9eb24b37-ec67-41b4-a2f1-b69bcced3c4e"). InnerVolumeSpecName "kube-api-access-5tmkb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 18 12:19:57.431218 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:19:57.431195 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9eb24b37-ec67-41b4-a2f1-b69bcced3c4e-istio-token" (OuterVolumeSpecName: "istio-token") pod "9eb24b37-ec67-41b4-a2f1-b69bcced3c4e" (UID: "9eb24b37-ec67-41b4-a2f1-b69bcced3c4e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 18 12:19:57.529723 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:19:57.529684 2585 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9eb24b37-ec67-41b4-a2f1-b69bcced3c4e-istio-data\") on node \"ip-10-0-136-240.ec2.internal\" DevicePath \"\"" Jul 18 12:19:57.529723 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:19:57.529716 2585 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9eb24b37-ec67-41b4-a2f1-b69bcced3c4e-istio-envoy\") on node \"ip-10-0-136-240.ec2.internal\" DevicePath \"\"" Jul 18 12:19:57.529926 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:19:57.529730 2585 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9eb24b37-ec67-41b4-a2f1-b69bcced3c4e-istiod-ca-cert\") on node \"ip-10-0-136-240.ec2.internal\" DevicePath \"\"" Jul 18 12:19:57.529926 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:19:57.529743 2585 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9eb24b37-ec67-41b4-a2f1-b69bcced3c4e-istio-podinfo\") on node \"ip-10-0-136-240.ec2.internal\" DevicePath \"\"" Jul 18 12:19:57.529926 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:19:57.529756 2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5tmkb\" (UniqueName: \"kubernetes.io/projected/9eb24b37-ec67-41b4-a2f1-b69bcced3c4e-kube-api-access-5tmkb\") on node \"ip-10-0-136-240.ec2.internal\" DevicePath \"\"" Jul 18 12:19:57.529926 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:19:57.529771 2585 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9eb24b37-ec67-41b4-a2f1-b69bcced3c4e-workload-socket\") on node \"ip-10-0-136-240.ec2.internal\" DevicePath \"\"" Jul 18 12:19:57.529926 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:19:57.529797 2585 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9eb24b37-ec67-41b4-a2f1-b69bcced3c4e-credential-socket\") on node \"ip-10-0-136-240.ec2.internal\" DevicePath \"\"" Jul 18 12:19:57.529926 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:19:57.529819 2585 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9eb24b37-ec67-41b4-a2f1-b69bcced3c4e-istio-token\") on node \"ip-10-0-136-240.ec2.internal\" DevicePath \"\"" Jul 18 12:19:57.659823 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:19:57.659729 2585 generic.go:356] "Generic (PLEG): container finished" podID="9eb24b37-ec67-41b4-a2f1-b69bcced3c4e" containerID="ab7dbd75dd93080aaa4d542256e9a0529a87d1b409ca700c79906671b1a4d8eb" exitCode=0 Jul 18 12:19:57.659823 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:19:57.659781 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-f1cd424d-gw-openshift-default-d98684879-5kkmq" Jul 18 12:19:57.660004 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:19:57.659818 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-f1cd424d-gw-openshift-default-d98684879-5kkmq" event={"ID":"9eb24b37-ec67-41b4-a2f1-b69bcced3c4e","Type":"ContainerDied","Data":"ab7dbd75dd93080aaa4d542256e9a0529a87d1b409ca700c79906671b1a4d8eb"} Jul 18 12:19:57.660004 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:19:57.659868 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-f1cd424d-gw-openshift-default-d98684879-5kkmq" event={"ID":"9eb24b37-ec67-41b4-a2f1-b69bcced3c4e","Type":"ContainerDied","Data":"5279a4b01f6e4090afa49ba761e967e77c5f4b92b66a6f682861ce3b948d785b"} Jul 18 12:19:57.660004 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:19:57.659892 2585 scope.go:117] "RemoveContainer" containerID="ab7dbd75dd93080aaa4d542256e9a0529a87d1b409ca700c79906671b1a4d8eb" Jul 18 12:19:57.669167 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:19:57.669154 2585 scope.go:117] "RemoveContainer" containerID="bbe6bc506612b272116811ab0f5d8e0fa3b80b3ab773c33f0ce51b4983127de9" Jul 18 12:19:57.677282 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:19:57.677261 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-f1cd424d-gw-openshift-default-d98684879-5kkmq"] Jul 18 12:19:57.677361 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:19:57.677293 2585 scope.go:117] "RemoveContainer" containerID="ab7dbd75dd93080aaa4d542256e9a0529a87d1b409ca700c79906671b1a4d8eb" Jul 18 12:19:57.677586 ip-10-0-136-240 kubenswrapper[2585]: E0718 12:19:57.677567 2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ab7dbd75dd93080aaa4d542256e9a0529a87d1b409ca700c79906671b1a4d8eb\": container with ID starting with ab7dbd75dd93080aaa4d542256e9a0529a87d1b409ca700c79906671b1a4d8eb not found: ID does not exist" containerID="ab7dbd75dd93080aaa4d542256e9a0529a87d1b409ca700c79906671b1a4d8eb" Jul 18 12:19:57.677653 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:19:57.677594 2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ab7dbd75dd93080aaa4d542256e9a0529a87d1b409ca700c79906671b1a4d8eb"} err="failed to get container status \"ab7dbd75dd93080aaa4d542256e9a0529a87d1b409ca700c79906671b1a4d8eb\": rpc error: code = NotFound desc = could not find container \"ab7dbd75dd93080aaa4d542256e9a0529a87d1b409ca700c79906671b1a4d8eb\": container with ID starting with ab7dbd75dd93080aaa4d542256e9a0529a87d1b409ca700c79906671b1a4d8eb not found: ID does not exist" Jul 18 12:19:57.677653 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:19:57.677611 2585 scope.go:117] "RemoveContainer" containerID="bbe6bc506612b272116811ab0f5d8e0fa3b80b3ab773c33f0ce51b4983127de9" Jul 18 12:19:57.677843 ip-10-0-136-240 kubenswrapper[2585]: E0718 12:19:57.677825 2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bbe6bc506612b272116811ab0f5d8e0fa3b80b3ab773c33f0ce51b4983127de9\": container with ID starting with bbe6bc506612b272116811ab0f5d8e0fa3b80b3ab773c33f0ce51b4983127de9 not found: ID does not exist" containerID="bbe6bc506612b272116811ab0f5d8e0fa3b80b3ab773c33f0ce51b4983127de9" Jul 18 12:19:57.677893 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:19:57.677851 2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bbe6bc506612b272116811ab0f5d8e0fa3b80b3ab773c33f0ce51b4983127de9"} err="failed to get container status \"bbe6bc506612b272116811ab0f5d8e0fa3b80b3ab773c33f0ce51b4983127de9\": rpc error: code = NotFound desc = could not find container \"bbe6bc506612b272116811ab0f5d8e0fa3b80b3ab773c33f0ce51b4983127de9\": container with ID starting with bbe6bc506612b272116811ab0f5d8e0fa3b80b3ab773c33f0ce51b4983127de9 not found: ID does not exist" Jul 18 12:19:57.681040 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:19:57.681020 2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-migrate-f1cd424d-gw-openshift-default-d98684879-5kkmq"] Jul 18 12:19:59.469751 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:19:59.469712 2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9eb24b37-ec67-41b4-a2f1-b69bcced3c4e" path="/var/lib/kubelet/pods/9eb24b37-ec67-41b4-a2f1-b69bcced3c4e/volumes" Jul 18 12:20:09.642685 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:20:09.642651 2585 patch_prober.go:28] interesting pod/e2e-ait-3dff9e6d-openshift-default-74d94fd969-4nnkx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" start-of-body= Jul 18 12:20:09.643079 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:20:09.642703 2585 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-3dff9e6d-openshift-default-74d94fd969-4nnkx" podUID="49cac5e7-2a6f-493d-b280-31637879b57e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" Jul 18 12:20:09.729826 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:20:09.729797 2585 generic.go:356] "Generic (PLEG): container finished" podID="49cac5e7-2a6f-493d-b280-31637879b57e" containerID="862d4d88f6ee7ed56a31d607e07d6ff94f60f14a23006311e02012cc21e06baf" exitCode=0 Jul 18 12:20:09.729972 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:20:09.729882 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-3dff9e6d-openshift-default-74d94fd969-4nnkx" event={"ID":"49cac5e7-2a6f-493d-b280-31637879b57e","Type":"ContainerDied","Data":"862d4d88f6ee7ed56a31d607e07d6ff94f60f14a23006311e02012cc21e06baf"} Jul 18 12:20:09.730300 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:20:09.730282 2585 scope.go:117] "RemoveContainer" containerID="862d4d88f6ee7ed56a31d607e07d6ff94f60f14a23006311e02012cc21e06baf" Jul 18 12:20:09.731305 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:20:09.731277 2585 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 18 12:20:09.731384 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:20:09.731361 2585 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 18 12:20:09.731420 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:20:09.731405 2585 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 18 12:20:10.737823 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:20:10.737783 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-3dff9e6d-openshift-default-74d94fd969-4nnkx" event={"ID":"49cac5e7-2a6f-493d-b280-31637879b57e","Type":"ContainerStarted","Data":"772637ac9c309f7b46e95ea12972ce5be262bf7c53fd337ad3a52e78fad9c622"} Jul 18 12:20:11.154355 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:20:11.154272 2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-3dff9e6d-openshift-default-74d94fd969-4nnkx" Jul 18 12:20:11.154505 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:20:11.154482 2585 patch_prober.go:28] interesting pod/e2e-ait-3dff9e6d-openshift-default-74d94fd969-4nnkx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" start-of-body= Jul 18 12:20:11.154586 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:20:11.154538 2585 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-3dff9e6d-openshift-default-74d94fd969-4nnkx" podUID="49cac5e7-2a6f-493d-b280-31637879b57e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" Jul 18 12:20:12.154150 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:20:12.154112 2585 patch_prober.go:28] interesting pod/e2e-ait-3dff9e6d-openshift-default-74d94fd969-4nnkx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" start-of-body= Jul 18 12:20:12.154536 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:20:12.154170 2585 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-3dff9e6d-openshift-default-74d94fd969-4nnkx" podUID="49cac5e7-2a6f-493d-b280-31637879b57e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" Jul 18 12:20:13.153979 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:20:13.153940 2585 patch_prober.go:28] interesting pod/e2e-ait-3dff9e6d-openshift-default-74d94fd969-4nnkx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" start-of-body= Jul 18 12:20:13.154165 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:20:13.153998 2585 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-3dff9e6d-openshift-default-74d94fd969-4nnkx" podUID="49cac5e7-2a6f-493d-b280-31637879b57e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" Jul 18 12:20:14.154496 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:20:14.154463 2585 patch_prober.go:28] interesting pod/e2e-ait-3dff9e6d-openshift-default-74d94fd969-4nnkx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" start-of-body= Jul 18 12:20:14.154904 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:20:14.154521 2585 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-3dff9e6d-openshift-default-74d94fd969-4nnkx" podUID="49cac5e7-2a6f-493d-b280-31637879b57e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" Jul 18 12:20:15.158343 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:20:15.158312 2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-3dff9e6d-openshift-default-74d94fd969-4nnkx" Jul 18 12:20:15.158834 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:20:15.158718 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-3dff9e6d-openshift-default-74d94fd969-4nnkx" Jul 18 12:20:15.159422 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:20:15.159406 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-3dff9e6d-openshift-default-74d94fd969-4nnkx" Jul 18 12:20:36.950796 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:20:36.950759 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-3dff9e6d-openshift-default-74d94fd969-4nnkx"] Jul 18 12:20:36.951330 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:20:36.951042 2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-3dff9e6d-openshift-default-74d94fd969-4nnkx" podUID="49cac5e7-2a6f-493d-b280-31637879b57e" containerName="istio-proxy" containerID="cri-o://772637ac9c309f7b46e95ea12972ce5be262bf7c53fd337ad3a52e78fad9c622" gracePeriod=30 Jul 18 12:20:37.764167 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:20:37.764135 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-derive-c5db2665-openshift-default-f7989589f-hp45v"] Jul 18 12:20:37.764658 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:20:37.764638 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9eb24b37-ec67-41b4-a2f1-b69bcced3c4e" containerName="istio-proxy" Jul 18 12:20:37.764658 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:20:37.764658 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="9eb24b37-ec67-41b4-a2f1-b69bcced3c4e" containerName="istio-proxy" Jul 18 12:20:37.764819 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:20:37.764673 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9eb24b37-ec67-41b4-a2f1-b69bcced3c4e" containerName="istio-proxy" Jul 18 12:20:37.764819 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:20:37.764680 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="9eb24b37-ec67-41b4-a2f1-b69bcced3c4e" containerName="istio-proxy" Jul 18 12:20:37.764819 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:20:37.764799 2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="9eb24b37-ec67-41b4-a2f1-b69bcced3c4e" containerName="istio-proxy" Jul 18 12:20:37.764819 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:20:37.764811 2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="9eb24b37-ec67-41b4-a2f1-b69bcced3c4e" containerName="istio-proxy" Jul 18 12:20:37.767967 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:20:37.767945 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-c5db2665-openshift-default-f7989589f-hp45v" Jul 18 12:20:37.775095 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:20:37.775069 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-derive-c5db2665-openshift-default-dockercfg-4hjtf\"" Jul 18 12:20:37.799627 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:20:37.799595 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-c5db2665-openshift-default-f7989589f-hp45v"] Jul 18 12:20:37.911973 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:20:37.911937 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c236787c-3cc1-4370-a8a0-138fb2709902-istio-podinfo\") pod \"e2e-derive-c5db2665-openshift-default-f7989589f-hp45v\" (UID: \"c236787c-3cc1-4370-a8a0-138fb2709902\") " pod="openshift-ingress/e2e-derive-c5db2665-openshift-default-f7989589f-hp45v" Jul 18 12:20:37.912160 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:20:37.912000 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-26hrm\" (UniqueName: \"kubernetes.io/projected/c236787c-3cc1-4370-a8a0-138fb2709902-kube-api-access-26hrm\") pod \"e2e-derive-c5db2665-openshift-default-f7989589f-hp45v\" (UID: \"c236787c-3cc1-4370-a8a0-138fb2709902\") " pod="openshift-ingress/e2e-derive-c5db2665-openshift-default-f7989589f-hp45v" Jul 18 12:20:37.912160 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:20:37.912075 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c236787c-3cc1-4370-a8a0-138fb2709902-workload-socket\") pod \"e2e-derive-c5db2665-openshift-default-f7989589f-hp45v\" (UID: \"c236787c-3cc1-4370-a8a0-138fb2709902\") " pod="openshift-ingress/e2e-derive-c5db2665-openshift-default-f7989589f-hp45v" Jul 18 12:20:37.912160 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:20:37.912109 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c236787c-3cc1-4370-a8a0-138fb2709902-istiod-ca-cert\") pod \"e2e-derive-c5db2665-openshift-default-f7989589f-hp45v\" (UID: \"c236787c-3cc1-4370-a8a0-138fb2709902\") " pod="openshift-ingress/e2e-derive-c5db2665-openshift-default-f7989589f-hp45v" Jul 18 12:20:37.912160 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:20:37.912132 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c236787c-3cc1-4370-a8a0-138fb2709902-istio-token\") pod \"e2e-derive-c5db2665-openshift-default-f7989589f-hp45v\" (UID: \"c236787c-3cc1-4370-a8a0-138fb2709902\") " pod="openshift-ingress/e2e-derive-c5db2665-openshift-default-f7989589f-hp45v" Jul 18 12:20:37.912160 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:20:37.912148 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c236787c-3cc1-4370-a8a0-138fb2709902-istio-envoy\") pod \"e2e-derive-c5db2665-openshift-default-f7989589f-hp45v\" (UID: \"c236787c-3cc1-4370-a8a0-138fb2709902\") " pod="openshift-ingress/e2e-derive-c5db2665-openshift-default-f7989589f-hp45v" Jul 18 12:20:37.912365 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:20:37.912198 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c236787c-3cc1-4370-a8a0-138fb2709902-workload-certs\") pod \"e2e-derive-c5db2665-openshift-default-f7989589f-hp45v\" (UID: \"c236787c-3cc1-4370-a8a0-138fb2709902\") " pod="openshift-ingress/e2e-derive-c5db2665-openshift-default-f7989589f-hp45v" Jul 18 12:20:37.912365 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:20:37.912249 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c236787c-3cc1-4370-a8a0-138fb2709902-credential-socket\") pod \"e2e-derive-c5db2665-openshift-default-f7989589f-hp45v\" (UID: \"c236787c-3cc1-4370-a8a0-138fb2709902\") " pod="openshift-ingress/e2e-derive-c5db2665-openshift-default-f7989589f-hp45v" Jul 18 12:20:37.912365 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:20:37.912271 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c236787c-3cc1-4370-a8a0-138fb2709902-istio-data\") pod \"e2e-derive-c5db2665-openshift-default-f7989589f-hp45v\" (UID: \"c236787c-3cc1-4370-a8a0-138fb2709902\") " pod="openshift-ingress/e2e-derive-c5db2665-openshift-default-f7989589f-hp45v" Jul 18 12:20:38.012979 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:20:38.012949 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c236787c-3cc1-4370-a8a0-138fb2709902-credential-socket\") pod \"e2e-derive-c5db2665-openshift-default-f7989589f-hp45v\" (UID: \"c236787c-3cc1-4370-a8a0-138fb2709902\") " pod="openshift-ingress/e2e-derive-c5db2665-openshift-default-f7989589f-hp45v" Jul 18 12:20:38.013409 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:20:38.012984 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c236787c-3cc1-4370-a8a0-138fb2709902-istio-data\") pod \"e2e-derive-c5db2665-openshift-default-f7989589f-hp45v\" (UID: \"c236787c-3cc1-4370-a8a0-138fb2709902\") " pod="openshift-ingress/e2e-derive-c5db2665-openshift-default-f7989589f-hp45v" Jul 18 12:20:38.013409 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:20:38.013032 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c236787c-3cc1-4370-a8a0-138fb2709902-istio-podinfo\") pod \"e2e-derive-c5db2665-openshift-default-f7989589f-hp45v\" (UID: \"c236787c-3cc1-4370-a8a0-138fb2709902\") " pod="openshift-ingress/e2e-derive-c5db2665-openshift-default-f7989589f-hp45v" Jul 18 12:20:38.013409 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:20:38.013064 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-26hrm\" (UniqueName: \"kubernetes.io/projected/c236787c-3cc1-4370-a8a0-138fb2709902-kube-api-access-26hrm\") pod \"e2e-derive-c5db2665-openshift-default-f7989589f-hp45v\" (UID: \"c236787c-3cc1-4370-a8a0-138fb2709902\") " pod="openshift-ingress/e2e-derive-c5db2665-openshift-default-f7989589f-hp45v" Jul 18 12:20:38.013409 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:20:38.013097 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c236787c-3cc1-4370-a8a0-138fb2709902-workload-socket\") pod \"e2e-derive-c5db2665-openshift-default-f7989589f-hp45v\" (UID: \"c236787c-3cc1-4370-a8a0-138fb2709902\") " pod="openshift-ingress/e2e-derive-c5db2665-openshift-default-f7989589f-hp45v" Jul 18 12:20:38.013409 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:20:38.013121 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c236787c-3cc1-4370-a8a0-138fb2709902-istiod-ca-cert\") pod \"e2e-derive-c5db2665-openshift-default-f7989589f-hp45v\" (UID: \"c236787c-3cc1-4370-a8a0-138fb2709902\") " pod="openshift-ingress/e2e-derive-c5db2665-openshift-default-f7989589f-hp45v" Jul 18 12:20:38.013409 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:20:38.013147 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c236787c-3cc1-4370-a8a0-138fb2709902-istio-token\") pod \"e2e-derive-c5db2665-openshift-default-f7989589f-hp45v\" (UID: \"c236787c-3cc1-4370-a8a0-138fb2709902\") " pod="openshift-ingress/e2e-derive-c5db2665-openshift-default-f7989589f-hp45v" Jul 18 12:20:38.013409 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:20:38.013173 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c236787c-3cc1-4370-a8a0-138fb2709902-istio-envoy\") pod \"e2e-derive-c5db2665-openshift-default-f7989589f-hp45v\" (UID: \"c236787c-3cc1-4370-a8a0-138fb2709902\") " pod="openshift-ingress/e2e-derive-c5db2665-openshift-default-f7989589f-hp45v" Jul 18 12:20:38.013409 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:20:38.013215 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c236787c-3cc1-4370-a8a0-138fb2709902-workload-certs\") pod \"e2e-derive-c5db2665-openshift-default-f7989589f-hp45v\" (UID: \"c236787c-3cc1-4370-a8a0-138fb2709902\") " pod="openshift-ingress/e2e-derive-c5db2665-openshift-default-f7989589f-hp45v" Jul 18 12:20:38.013409 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:20:38.013408 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c236787c-3cc1-4370-a8a0-138fb2709902-credential-socket\") pod \"e2e-derive-c5db2665-openshift-default-f7989589f-hp45v\" (UID: \"c236787c-3cc1-4370-a8a0-138fb2709902\") " pod="openshift-ingress/e2e-derive-c5db2665-openshift-default-f7989589f-hp45v" Jul 18 12:20:38.013899 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:20:38.013618 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c236787c-3cc1-4370-a8a0-138fb2709902-istio-data\") pod \"e2e-derive-c5db2665-openshift-default-f7989589f-hp45v\" (UID: \"c236787c-3cc1-4370-a8a0-138fb2709902\") " pod="openshift-ingress/e2e-derive-c5db2665-openshift-default-f7989589f-hp45v" Jul 18 12:20:38.013899 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:20:38.013672 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c236787c-3cc1-4370-a8a0-138fb2709902-workload-certs\") pod \"e2e-derive-c5db2665-openshift-default-f7989589f-hp45v\" (UID: \"c236787c-3cc1-4370-a8a0-138fb2709902\") " pod="openshift-ingress/e2e-derive-c5db2665-openshift-default-f7989589f-hp45v" Jul 18 12:20:38.013899 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:20:38.013702 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c236787c-3cc1-4370-a8a0-138fb2709902-workload-socket\") pod \"e2e-derive-c5db2665-openshift-default-f7989589f-hp45v\" (UID: \"c236787c-3cc1-4370-a8a0-138fb2709902\") " pod="openshift-ingress/e2e-derive-c5db2665-openshift-default-f7989589f-hp45v" Jul 18 12:20:38.013899 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:20:38.013813 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c236787c-3cc1-4370-a8a0-138fb2709902-istiod-ca-cert\") pod \"e2e-derive-c5db2665-openshift-default-f7989589f-hp45v\" (UID: \"c236787c-3cc1-4370-a8a0-138fb2709902\") " pod="openshift-ingress/e2e-derive-c5db2665-openshift-default-f7989589f-hp45v" Jul 18 12:20:38.015711 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:20:38.015666 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c236787c-3cc1-4370-a8a0-138fb2709902-istio-envoy\") pod \"e2e-derive-c5db2665-openshift-default-f7989589f-hp45v\" (UID: \"c236787c-3cc1-4370-a8a0-138fb2709902\") " pod="openshift-ingress/e2e-derive-c5db2665-openshift-default-f7989589f-hp45v" Jul 18 12:20:38.015778 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:20:38.015715 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c236787c-3cc1-4370-a8a0-138fb2709902-istio-podinfo\") pod \"e2e-derive-c5db2665-openshift-default-f7989589f-hp45v\" (UID: \"c236787c-3cc1-4370-a8a0-138fb2709902\") " pod="openshift-ingress/e2e-derive-c5db2665-openshift-default-f7989589f-hp45v" Jul 18 12:20:38.022475 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:20:38.022447 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c236787c-3cc1-4370-a8a0-138fb2709902-istio-token\") pod \"e2e-derive-c5db2665-openshift-default-f7989589f-hp45v\" (UID: \"c236787c-3cc1-4370-a8a0-138fb2709902\") " pod="openshift-ingress/e2e-derive-c5db2665-openshift-default-f7989589f-hp45v" Jul 18 12:20:38.024361 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:20:38.024342 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-26hrm\" (UniqueName: \"kubernetes.io/projected/c236787c-3cc1-4370-a8a0-138fb2709902-kube-api-access-26hrm\") pod \"e2e-derive-c5db2665-openshift-default-f7989589f-hp45v\" (UID: \"c236787c-3cc1-4370-a8a0-138fb2709902\") " pod="openshift-ingress/e2e-derive-c5db2665-openshift-default-f7989589f-hp45v" Jul 18 12:20:38.078761 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:20:38.078741 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-c5db2665-openshift-default-f7989589f-hp45v" Jul 18 12:20:38.218953 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:20:38.218924 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-c5db2665-openshift-default-f7989589f-hp45v"] Jul 18 12:20:38.220859 ip-10-0-136-240 kubenswrapper[2585]: W0718 12:20:38.220826 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc236787c_3cc1_4370_a8a0_138fb2709902.slice/crio-598bf8d890d23cb0746bb0043bb37ec30f854535bdaa90b7c9b0a3cfe15f0800 WatchSource:0}: Error finding container 598bf8d890d23cb0746bb0043bb37ec30f854535bdaa90b7c9b0a3cfe15f0800: Status 404 returned error can't find the container with id 598bf8d890d23cb0746bb0043bb37ec30f854535bdaa90b7c9b0a3cfe15f0800 Jul 18 12:20:38.223067 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:20:38.223036 2585 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 18 12:20:38.223138 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:20:38.223105 2585 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 18 12:20:38.223138 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:20:38.223132 2585 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 18 12:20:38.888536 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:20:38.888492 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-c5db2665-openshift-default-f7989589f-hp45v" event={"ID":"c236787c-3cc1-4370-a8a0-138fb2709902","Type":"ContainerStarted","Data":"b22a129bc7453518f6fe24cb61ce31d25324fd3431a4965d263e866d1dd426c0"} Jul 18 12:20:38.888965 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:20:38.888581 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-c5db2665-openshift-default-f7989589f-hp45v" event={"ID":"c236787c-3cc1-4370-a8a0-138fb2709902","Type":"ContainerStarted","Data":"598bf8d890d23cb0746bb0043bb37ec30f854535bdaa90b7c9b0a3cfe15f0800"} Jul 18 12:20:38.910799 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:20:38.910750 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-derive-c5db2665-openshift-default-f7989589f-hp45v" podStartSLOduration=1.910732797 podStartE2EDuration="1.910732797s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-18 12:20:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-18 12:20:38.909261459 +0000 UTC m=+1905.989467837" watchObservedRunningTime="2026-07-18 12:20:38.910732797 +0000 UTC m=+1905.990939154" Jul 18 12:20:39.079072 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:20:39.079033 2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-c5db2665-openshift-default-f7989589f-hp45v" Jul 18 12:20:39.083444 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:20:39.083422 2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-c5db2665-openshift-default-f7989589f-hp45v" Jul 18 12:20:39.894421 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:20:39.894395 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-c5db2665-openshift-default-f7989589f-hp45v" Jul 18 12:20:39.895207 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:20:39.895185 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-c5db2665-openshift-default-f7989589f-hp45v" Jul 18 12:20:42.204880 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:20:42.204857 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-3dff9e6d-openshift-default-74d94fd969-4nnkx" Jul 18 12:20:42.356312 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:20:42.356224 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/49cac5e7-2a6f-493d-b280-31637879b57e-istio-data\") pod \"49cac5e7-2a6f-493d-b280-31637879b57e\" (UID: \"49cac5e7-2a6f-493d-b280-31637879b57e\") " Jul 18 12:20:42.356312 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:20:42.356277 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-w9mdk\" (UniqueName: \"kubernetes.io/projected/49cac5e7-2a6f-493d-b280-31637879b57e-kube-api-access-w9mdk\") pod \"49cac5e7-2a6f-493d-b280-31637879b57e\" (UID: \"49cac5e7-2a6f-493d-b280-31637879b57e\") " Jul 18 12:20:42.356312 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:20:42.356303 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/49cac5e7-2a6f-493d-b280-31637879b57e-istio-token\") pod \"49cac5e7-2a6f-493d-b280-31637879b57e\" (UID: \"49cac5e7-2a6f-493d-b280-31637879b57e\") " Jul 18 12:20:42.356312 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:20:42.356320 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/49cac5e7-2a6f-493d-b280-31637879b57e-istio-podinfo\") pod \"49cac5e7-2a6f-493d-b280-31637879b57e\" (UID: \"49cac5e7-2a6f-493d-b280-31637879b57e\") " Jul 18 12:20:42.356680 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:20:42.356374 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/49cac5e7-2a6f-493d-b280-31637879b57e-istio-envoy\") pod \"49cac5e7-2a6f-493d-b280-31637879b57e\" (UID: \"49cac5e7-2a6f-493d-b280-31637879b57e\") " Jul 18 12:20:42.356680 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:20:42.356450 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/49cac5e7-2a6f-493d-b280-31637879b57e-credential-socket\") pod \"49cac5e7-2a6f-493d-b280-31637879b57e\" (UID: \"49cac5e7-2a6f-493d-b280-31637879b57e\") " Jul 18 12:20:42.356680 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:20:42.356501 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/49cac5e7-2a6f-493d-b280-31637879b57e-workload-socket\") pod \"49cac5e7-2a6f-493d-b280-31637879b57e\" (UID: \"49cac5e7-2a6f-493d-b280-31637879b57e\") " Jul 18 12:20:42.356680 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:20:42.356600 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/49cac5e7-2a6f-493d-b280-31637879b57e-workload-certs\") pod \"49cac5e7-2a6f-493d-b280-31637879b57e\" (UID: \"49cac5e7-2a6f-493d-b280-31637879b57e\") " Jul 18 12:20:42.356680 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:20:42.356675 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/49cac5e7-2a6f-493d-b280-31637879b57e-istiod-ca-cert\") pod \"49cac5e7-2a6f-493d-b280-31637879b57e\" (UID: \"49cac5e7-2a6f-493d-b280-31637879b57e\") " Jul 18 12:20:42.356944 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:20:42.356785 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/49cac5e7-2a6f-493d-b280-31637879b57e-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "49cac5e7-2a6f-493d-b280-31637879b57e" (UID: "49cac5e7-2a6f-493d-b280-31637879b57e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 18 12:20:42.356944 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:20:42.356803 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/49cac5e7-2a6f-493d-b280-31637879b57e-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "49cac5e7-2a6f-493d-b280-31637879b57e" (UID: "49cac5e7-2a6f-493d-b280-31637879b57e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 18 12:20:42.357053 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:20:42.357000 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/49cac5e7-2a6f-493d-b280-31637879b57e-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "49cac5e7-2a6f-493d-b280-31637879b57e" (UID: "49cac5e7-2a6f-493d-b280-31637879b57e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 18 12:20:42.357107 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:20:42.357077 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/49cac5e7-2a6f-493d-b280-31637879b57e-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "49cac5e7-2a6f-493d-b280-31637879b57e" (UID: "49cac5e7-2a6f-493d-b280-31637879b57e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 18 12:20:42.357326 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:20:42.357113 2585 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/49cac5e7-2a6f-493d-b280-31637879b57e-credential-socket\") on node \"ip-10-0-136-240.ec2.internal\" DevicePath \"\"" Jul 18 12:20:42.357326 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:20:42.357131 2585 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/49cac5e7-2a6f-493d-b280-31637879b57e-workload-socket\") on node \"ip-10-0-136-240.ec2.internal\" DevicePath \"\"" Jul 18 12:20:42.357326 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:20:42.357144 2585 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/49cac5e7-2a6f-493d-b280-31637879b57e-workload-certs\") on node \"ip-10-0-136-240.ec2.internal\" DevicePath \"\"" Jul 18 12:20:42.357656 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:20:42.357637 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/49cac5e7-2a6f-493d-b280-31637879b57e-istio-data" (OuterVolumeSpecName: "istio-data") pod "49cac5e7-2a6f-493d-b280-31637879b57e" (UID: "49cac5e7-2a6f-493d-b280-31637879b57e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 18 12:20:42.358771 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:20:42.358747 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/49cac5e7-2a6f-493d-b280-31637879b57e-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "49cac5e7-2a6f-493d-b280-31637879b57e" (UID: "49cac5e7-2a6f-493d-b280-31637879b57e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 18 12:20:42.358885 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:20:42.358830 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/49cac5e7-2a6f-493d-b280-31637879b57e-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "49cac5e7-2a6f-493d-b280-31637879b57e" (UID: "49cac5e7-2a6f-493d-b280-31637879b57e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 18 12:20:42.359183 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:20:42.359170 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/49cac5e7-2a6f-493d-b280-31637879b57e-kube-api-access-w9mdk" (OuterVolumeSpecName: "kube-api-access-w9mdk") pod "49cac5e7-2a6f-493d-b280-31637879b57e" (UID: "49cac5e7-2a6f-493d-b280-31637879b57e"). InnerVolumeSpecName "kube-api-access-w9mdk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 18 12:20:42.359226 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:20:42.359206 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/49cac5e7-2a6f-493d-b280-31637879b57e-istio-token" (OuterVolumeSpecName: "istio-token") pod "49cac5e7-2a6f-493d-b280-31637879b57e" (UID: "49cac5e7-2a6f-493d-b280-31637879b57e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 18 12:20:42.457916 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:20:42.457885 2585 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/49cac5e7-2a6f-493d-b280-31637879b57e-istio-envoy\") on node \"ip-10-0-136-240.ec2.internal\" DevicePath \"\"" Jul 18 12:20:42.457916 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:20:42.457911 2585 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/49cac5e7-2a6f-493d-b280-31637879b57e-istiod-ca-cert\") on node \"ip-10-0-136-240.ec2.internal\" DevicePath \"\"" Jul 18 12:20:42.458070 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:20:42.457923 2585 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/49cac5e7-2a6f-493d-b280-31637879b57e-istio-data\") on node \"ip-10-0-136-240.ec2.internal\" DevicePath \"\"" Jul 18 12:20:42.458070 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:20:42.457934 2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w9mdk\" (UniqueName: \"kubernetes.io/projected/49cac5e7-2a6f-493d-b280-31637879b57e-kube-api-access-w9mdk\") on node \"ip-10-0-136-240.ec2.internal\" DevicePath \"\"" Jul 18 12:20:42.458070 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:20:42.457946 2585 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/49cac5e7-2a6f-493d-b280-31637879b57e-istio-token\") on node \"ip-10-0-136-240.ec2.internal\" DevicePath \"\"" Jul 18 12:20:42.458070 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:20:42.457957 2585 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/49cac5e7-2a6f-493d-b280-31637879b57e-istio-podinfo\") on node \"ip-10-0-136-240.ec2.internal\" DevicePath \"\"" Jul 18 12:20:42.910369 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:20:42.910339 2585 generic.go:356] "Generic (PLEG): container finished" podID="49cac5e7-2a6f-493d-b280-31637879b57e" containerID="772637ac9c309f7b46e95ea12972ce5be262bf7c53fd337ad3a52e78fad9c622" exitCode=0 Jul 18 12:20:42.910564 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:20:42.910389 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-3dff9e6d-openshift-default-74d94fd969-4nnkx" Jul 18 12:20:42.910564 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:20:42.910430 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-3dff9e6d-openshift-default-74d94fd969-4nnkx" event={"ID":"49cac5e7-2a6f-493d-b280-31637879b57e","Type":"ContainerDied","Data":"772637ac9c309f7b46e95ea12972ce5be262bf7c53fd337ad3a52e78fad9c622"} Jul 18 12:20:42.910564 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:20:42.910484 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-3dff9e6d-openshift-default-74d94fd969-4nnkx" event={"ID":"49cac5e7-2a6f-493d-b280-31637879b57e","Type":"ContainerDied","Data":"ea716dbd1b4134c92136be4966444c64b9f6163d73b1fa350266c94dfdf5e7de"} Jul 18 12:20:42.910564 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:20:42.910510 2585 scope.go:117] "RemoveContainer" containerID="772637ac9c309f7b46e95ea12972ce5be262bf7c53fd337ad3a52e78fad9c622" Jul 18 12:20:42.920465 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:20:42.920445 2585 scope.go:117] "RemoveContainer" containerID="862d4d88f6ee7ed56a31d607e07d6ff94f60f14a23006311e02012cc21e06baf" Jul 18 12:20:42.928565 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:20:42.928529 2585 scope.go:117] "RemoveContainer" containerID="772637ac9c309f7b46e95ea12972ce5be262bf7c53fd337ad3a52e78fad9c622" Jul 18 12:20:42.928860 ip-10-0-136-240 kubenswrapper[2585]: E0718 12:20:42.928841 2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"772637ac9c309f7b46e95ea12972ce5be262bf7c53fd337ad3a52e78fad9c622\": container with ID starting with 772637ac9c309f7b46e95ea12972ce5be262bf7c53fd337ad3a52e78fad9c622 not found: ID does not exist" containerID="772637ac9c309f7b46e95ea12972ce5be262bf7c53fd337ad3a52e78fad9c622" Jul 18 12:20:42.928988 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:20:42.928871 2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"772637ac9c309f7b46e95ea12972ce5be262bf7c53fd337ad3a52e78fad9c622"} err="failed to get container status \"772637ac9c309f7b46e95ea12972ce5be262bf7c53fd337ad3a52e78fad9c622\": rpc error: code = NotFound desc = could not find container \"772637ac9c309f7b46e95ea12972ce5be262bf7c53fd337ad3a52e78fad9c622\": container with ID starting with 772637ac9c309f7b46e95ea12972ce5be262bf7c53fd337ad3a52e78fad9c622 not found: ID does not exist" Jul 18 12:20:42.928988 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:20:42.928893 2585 scope.go:117] "RemoveContainer" containerID="862d4d88f6ee7ed56a31d607e07d6ff94f60f14a23006311e02012cc21e06baf" Jul 18 12:20:42.929152 ip-10-0-136-240 kubenswrapper[2585]: E0718 12:20:42.929132 2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"862d4d88f6ee7ed56a31d607e07d6ff94f60f14a23006311e02012cc21e06baf\": container with ID starting with 862d4d88f6ee7ed56a31d607e07d6ff94f60f14a23006311e02012cc21e06baf not found: ID does not exist" containerID="862d4d88f6ee7ed56a31d607e07d6ff94f60f14a23006311e02012cc21e06baf" Jul 18 12:20:42.929190 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:20:42.929159 2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"862d4d88f6ee7ed56a31d607e07d6ff94f60f14a23006311e02012cc21e06baf"} err="failed to get container status \"862d4d88f6ee7ed56a31d607e07d6ff94f60f14a23006311e02012cc21e06baf\": rpc error: code = NotFound desc = could not find container \"862d4d88f6ee7ed56a31d607e07d6ff94f60f14a23006311e02012cc21e06baf\": container with ID starting with 862d4d88f6ee7ed56a31d607e07d6ff94f60f14a23006311e02012cc21e06baf not found: ID does not exist" Jul 18 12:20:42.933520 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:20:42.933497 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-3dff9e6d-openshift-default-74d94fd969-4nnkx"] Jul 18 12:20:42.937123 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:20:42.937104 2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-3dff9e6d-openshift-default-74d94fd969-4nnkx"] Jul 18 12:20:43.469002 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:20:43.468971 2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="49cac5e7-2a6f-493d-b280-31637879b57e" path="/var/lib/kubelet/pods/49cac5e7-2a6f-493d-b280-31637879b57e/volumes" Jul 18 12:20:52.966737 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:20:52.966709 2585 generic.go:356] "Generic (PLEG): container finished" podID="c236787c-3cc1-4370-a8a0-138fb2709902" containerID="b22a129bc7453518f6fe24cb61ce31d25324fd3431a4965d263e866d1dd426c0" exitCode=0 Jul 18 12:20:52.967095 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:20:52.966793 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-c5db2665-openshift-default-f7989589f-hp45v" event={"ID":"c236787c-3cc1-4370-a8a0-138fb2709902","Type":"ContainerDied","Data":"b22a129bc7453518f6fe24cb61ce31d25324fd3431a4965d263e866d1dd426c0"} Jul 18 12:20:52.967220 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:20:52.967203 2585 scope.go:117] "RemoveContainer" containerID="b22a129bc7453518f6fe24cb61ce31d25324fd3431a4965d263e866d1dd426c0" Jul 18 12:20:52.968226 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:20:52.968197 2585 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 18 12:20:52.968334 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:20:52.968285 2585 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 18 12:20:52.968402 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:20:52.968332 2585 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 18 12:20:53.978208 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:20:53.978170 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-c5db2665-openshift-default-f7989589f-hp45v" event={"ID":"c236787c-3cc1-4370-a8a0-138fb2709902","Type":"ContainerStarted","Data":"69f051088d09841d6274c23ff1201aa45f80c95e6729d38bbdb869401f6571b8"} Jul 18 12:20:54.079209 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:20:54.079164 2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-c5db2665-openshift-default-f7989589f-hp45v" Jul 18 12:20:54.080454 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:20:54.080420 2585 patch_prober.go:28] interesting pod/e2e-derive-c5db2665-openshift-default-f7989589f-hp45v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" start-of-body= Jul 18 12:20:54.080650 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:20:54.080478 2585 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-c5db2665-openshift-default-f7989589f-hp45v" podUID="c236787c-3cc1-4370-a8a0-138fb2709902" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" Jul 18 12:20:54.895126 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:20:54.895091 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-c5db2665-openshift-default-f7989589f-hp45v" Jul 18 12:20:55.079961 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:20:55.079918 2585 patch_prober.go:28] interesting pod/e2e-derive-c5db2665-openshift-default-f7989589f-hp45v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" start-of-body= Jul 18 12:20:55.080437 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:20:55.079982 2585 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-c5db2665-openshift-default-f7989589f-hp45v" podUID="c236787c-3cc1-4370-a8a0-138fb2709902" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" Jul 18 12:20:56.081339 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:20:56.081296 2585 patch_prober.go:28] interesting pod/e2e-derive-c5db2665-openshift-default-f7989589f-hp45v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" start-of-body= Jul 18 12:20:56.081855 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:20:56.081363 2585 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-c5db2665-openshift-default-f7989589f-hp45v" podUID="c236787c-3cc1-4370-a8a0-138fb2709902" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" Jul 18 12:20:57.080123 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:20:57.080084 2585 patch_prober.go:28] interesting pod/e2e-derive-c5db2665-openshift-default-f7989589f-hp45v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" start-of-body= Jul 18 12:20:57.080402 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:20:57.080371 2585 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-c5db2665-openshift-default-f7989589f-hp45v" podUID="c236787c-3cc1-4370-a8a0-138fb2709902" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" Jul 18 12:20:58.083199 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:20:58.083169 2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-c5db2665-openshift-default-f7989589f-hp45v" Jul 18 12:20:58.084283 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:20:58.084264 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-c5db2665-openshift-default-f7989589f-hp45v" Jul 18 12:21:20.360210 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:21:20.360169 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-c5db2665-openshift-default-f7989589f-hp45v"] Jul 18 12:21:20.361023 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:21:20.360446 2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-derive-c5db2665-openshift-default-f7989589f-hp45v" podUID="c236787c-3cc1-4370-a8a0-138fb2709902" containerName="istio-proxy" containerID="cri-o://69f051088d09841d6274c23ff1201aa45f80c95e6729d38bbdb869401f6571b8" gracePeriod=30 Jul 18 12:21:25.620911 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:21:25.620887 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-c5db2665-openshift-default-f7989589f-hp45v" Jul 18 12:21:25.760682 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:21:25.760657 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c236787c-3cc1-4370-a8a0-138fb2709902-workload-certs\") pod \"c236787c-3cc1-4370-a8a0-138fb2709902\" (UID: \"c236787c-3cc1-4370-a8a0-138fb2709902\") " Jul 18 12:21:25.760843 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:21:25.760701 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c236787c-3cc1-4370-a8a0-138fb2709902-credential-socket\") pod \"c236787c-3cc1-4370-a8a0-138fb2709902\" (UID: \"c236787c-3cc1-4370-a8a0-138fb2709902\") " Jul 18 12:21:25.760843 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:21:25.760741 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c236787c-3cc1-4370-a8a0-138fb2709902-istio-token\") pod \"c236787c-3cc1-4370-a8a0-138fb2709902\" (UID: \"c236787c-3cc1-4370-a8a0-138fb2709902\") " Jul 18 12:21:25.760843 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:21:25.760777 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c236787c-3cc1-4370-a8a0-138fb2709902-istiod-ca-cert\") pod \"c236787c-3cc1-4370-a8a0-138fb2709902\" (UID: \"c236787c-3cc1-4370-a8a0-138fb2709902\") " Jul 18 12:21:25.760843 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:21:25.760793 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c236787c-3cc1-4370-a8a0-138fb2709902-istio-envoy\") pod \"c236787c-3cc1-4370-a8a0-138fb2709902\" (UID: \"c236787c-3cc1-4370-a8a0-138fb2709902\") " Jul 18 12:21:25.760843 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:21:25.760817 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c236787c-3cc1-4370-a8a0-138fb2709902-workload-socket\") pod \"c236787c-3cc1-4370-a8a0-138fb2709902\" (UID: \"c236787c-3cc1-4370-a8a0-138fb2709902\") " Jul 18 12:21:25.761104 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:21:25.760870 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c236787c-3cc1-4370-a8a0-138fb2709902-istio-podinfo\") pod \"c236787c-3cc1-4370-a8a0-138fb2709902\" (UID: \"c236787c-3cc1-4370-a8a0-138fb2709902\") " Jul 18 12:21:25.761104 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:21:25.760908 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-26hrm\" (UniqueName: \"kubernetes.io/projected/c236787c-3cc1-4370-a8a0-138fb2709902-kube-api-access-26hrm\") pod \"c236787c-3cc1-4370-a8a0-138fb2709902\" (UID: \"c236787c-3cc1-4370-a8a0-138fb2709902\") " Jul 18 12:21:25.761104 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:21:25.760943 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c236787c-3cc1-4370-a8a0-138fb2709902-istio-data\") pod \"c236787c-3cc1-4370-a8a0-138fb2709902\" (UID: \"c236787c-3cc1-4370-a8a0-138fb2709902\") " Jul 18 12:21:25.761265 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:21:25.761120 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c236787c-3cc1-4370-a8a0-138fb2709902-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "c236787c-3cc1-4370-a8a0-138fb2709902" (UID: "c236787c-3cc1-4370-a8a0-138fb2709902"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 18 12:21:25.761265 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:21:25.761140 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c236787c-3cc1-4370-a8a0-138fb2709902-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "c236787c-3cc1-4370-a8a0-138fb2709902" (UID: "c236787c-3cc1-4370-a8a0-138fb2709902"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 18 12:21:25.761265 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:21:25.761168 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c236787c-3cc1-4370-a8a0-138fb2709902-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "c236787c-3cc1-4370-a8a0-138fb2709902" (UID: "c236787c-3cc1-4370-a8a0-138fb2709902"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 18 12:21:25.761412 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:21:25.761268 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c236787c-3cc1-4370-a8a0-138fb2709902-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "c236787c-3cc1-4370-a8a0-138fb2709902" (UID: "c236787c-3cc1-4370-a8a0-138fb2709902"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 18 12:21:25.761412 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:21:25.761377 2585 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c236787c-3cc1-4370-a8a0-138fb2709902-istiod-ca-cert\") on node \"ip-10-0-136-240.ec2.internal\" DevicePath \"\"" Jul 18 12:21:25.761412 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:21:25.761390 2585 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c236787c-3cc1-4370-a8a0-138fb2709902-workload-socket\") on node \"ip-10-0-136-240.ec2.internal\" DevicePath \"\"" Jul 18 12:21:25.761412 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:21:25.761402 2585 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c236787c-3cc1-4370-a8a0-138fb2709902-workload-certs\") on node \"ip-10-0-136-240.ec2.internal\" DevicePath \"\"" Jul 18 12:21:25.761807 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:21:25.761420 2585 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c236787c-3cc1-4370-a8a0-138fb2709902-credential-socket\") on node \"ip-10-0-136-240.ec2.internal\" DevicePath \"\"" Jul 18 12:21:25.761867 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:21:25.761837 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c236787c-3cc1-4370-a8a0-138fb2709902-istio-data" (OuterVolumeSpecName: "istio-data") pod "c236787c-3cc1-4370-a8a0-138fb2709902" (UID: "c236787c-3cc1-4370-a8a0-138fb2709902"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 18 12:21:25.763148 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:21:25.763119 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c236787c-3cc1-4370-a8a0-138fb2709902-istio-token" (OuterVolumeSpecName: "istio-token") pod "c236787c-3cc1-4370-a8a0-138fb2709902" (UID: "c236787c-3cc1-4370-a8a0-138fb2709902"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 18 12:21:25.763306 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:21:25.763287 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c236787c-3cc1-4370-a8a0-138fb2709902-kube-api-access-26hrm" (OuterVolumeSpecName: "kube-api-access-26hrm") pod "c236787c-3cc1-4370-a8a0-138fb2709902" (UID: "c236787c-3cc1-4370-a8a0-138fb2709902"). InnerVolumeSpecName "kube-api-access-26hrm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 18 12:21:25.763607 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:21:25.763588 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c236787c-3cc1-4370-a8a0-138fb2709902-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "c236787c-3cc1-4370-a8a0-138fb2709902" (UID: "c236787c-3cc1-4370-a8a0-138fb2709902"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 18 12:21:25.763607 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:21:25.763585 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c236787c-3cc1-4370-a8a0-138fb2709902-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "c236787c-3cc1-4370-a8a0-138fb2709902" (UID: "c236787c-3cc1-4370-a8a0-138fb2709902"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 18 12:21:25.862407 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:21:25.862379 2585 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c236787c-3cc1-4370-a8a0-138fb2709902-istio-envoy\") on node \"ip-10-0-136-240.ec2.internal\" DevicePath \"\"" Jul 18 12:21:25.862407 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:21:25.862404 2585 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c236787c-3cc1-4370-a8a0-138fb2709902-istio-podinfo\") on node \"ip-10-0-136-240.ec2.internal\" DevicePath \"\"" Jul 18 12:21:25.862596 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:21:25.862417 2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-26hrm\" (UniqueName: \"kubernetes.io/projected/c236787c-3cc1-4370-a8a0-138fb2709902-kube-api-access-26hrm\") on node \"ip-10-0-136-240.ec2.internal\" DevicePath \"\"" Jul 18 12:21:25.862596 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:21:25.862431 2585 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c236787c-3cc1-4370-a8a0-138fb2709902-istio-data\") on node \"ip-10-0-136-240.ec2.internal\" DevicePath \"\"" Jul 18 12:21:25.862596 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:21:25.862443 2585 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c236787c-3cc1-4370-a8a0-138fb2709902-istio-token\") on node \"ip-10-0-136-240.ec2.internal\" DevicePath \"\"" Jul 18 12:21:26.143768 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:21:26.143680 2585 generic.go:356] "Generic (PLEG): container finished" podID="c236787c-3cc1-4370-a8a0-138fb2709902" containerID="69f051088d09841d6274c23ff1201aa45f80c95e6729d38bbdb869401f6571b8" exitCode=0 Jul 18 12:21:26.143768 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:21:26.143740 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-c5db2665-openshift-default-f7989589f-hp45v" event={"ID":"c236787c-3cc1-4370-a8a0-138fb2709902","Type":"ContainerDied","Data":"69f051088d09841d6274c23ff1201aa45f80c95e6729d38bbdb869401f6571b8"} Jul 18 12:21:26.143768 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:21:26.143764 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-c5db2665-openshift-default-f7989589f-hp45v" Jul 18 12:21:26.144017 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:21:26.143790 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-c5db2665-openshift-default-f7989589f-hp45v" event={"ID":"c236787c-3cc1-4370-a8a0-138fb2709902","Type":"ContainerDied","Data":"598bf8d890d23cb0746bb0043bb37ec30f854535bdaa90b7c9b0a3cfe15f0800"} Jul 18 12:21:26.144017 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:21:26.143820 2585 scope.go:117] "RemoveContainer" containerID="69f051088d09841d6274c23ff1201aa45f80c95e6729d38bbdb869401f6571b8" Jul 18 12:21:26.159970 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:21:26.159954 2585 scope.go:117] "RemoveContainer" containerID="b22a129bc7453518f6fe24cb61ce31d25324fd3431a4965d263e866d1dd426c0" Jul 18 12:21:26.169126 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:21:26.169105 2585 scope.go:117] "RemoveContainer" containerID="69f051088d09841d6274c23ff1201aa45f80c95e6729d38bbdb869401f6571b8" Jul 18 12:21:26.169417 ip-10-0-136-240 kubenswrapper[2585]: E0718 12:21:26.169398 2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"69f051088d09841d6274c23ff1201aa45f80c95e6729d38bbdb869401f6571b8\": container with ID starting with 69f051088d09841d6274c23ff1201aa45f80c95e6729d38bbdb869401f6571b8 not found: ID does not exist" containerID="69f051088d09841d6274c23ff1201aa45f80c95e6729d38bbdb869401f6571b8" Jul 18 12:21:26.169465 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:21:26.169426 2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"69f051088d09841d6274c23ff1201aa45f80c95e6729d38bbdb869401f6571b8"} err="failed to get container status \"69f051088d09841d6274c23ff1201aa45f80c95e6729d38bbdb869401f6571b8\": rpc error: code = NotFound desc = could not find container \"69f051088d09841d6274c23ff1201aa45f80c95e6729d38bbdb869401f6571b8\": container with ID starting with 69f051088d09841d6274c23ff1201aa45f80c95e6729d38bbdb869401f6571b8 not found: ID does not exist" Jul 18 12:21:26.169465 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:21:26.169444 2585 scope.go:117] "RemoveContainer" containerID="b22a129bc7453518f6fe24cb61ce31d25324fd3431a4965d263e866d1dd426c0" Jul 18 12:21:26.169717 ip-10-0-136-240 kubenswrapper[2585]: E0718 12:21:26.169694 2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b22a129bc7453518f6fe24cb61ce31d25324fd3431a4965d263e866d1dd426c0\": container with ID starting with b22a129bc7453518f6fe24cb61ce31d25324fd3431a4965d263e866d1dd426c0 not found: ID does not exist" containerID="b22a129bc7453518f6fe24cb61ce31d25324fd3431a4965d263e866d1dd426c0" Jul 18 12:21:26.169839 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:21:26.169720 2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b22a129bc7453518f6fe24cb61ce31d25324fd3431a4965d263e866d1dd426c0"} err="failed to get container status \"b22a129bc7453518f6fe24cb61ce31d25324fd3431a4965d263e866d1dd426c0\": rpc error: code = NotFound desc = could not find container \"b22a129bc7453518f6fe24cb61ce31d25324fd3431a4965d263e866d1dd426c0\": container with ID starting with b22a129bc7453518f6fe24cb61ce31d25324fd3431a4965d263e866d1dd426c0 not found: ID does not exist" Jul 18 12:21:26.170297 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:21:26.170276 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-c5db2665-openshift-default-f7989589f-hp45v"] Jul 18 12:21:26.173925 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:21:26.173905 2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-derive-c5db2665-openshift-default-f7989589f-hp45v"] Jul 18 12:21:27.468805 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:21:27.468772 2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c236787c-3cc1-4370-a8a0-138fb2709902" path="/var/lib/kubelet/pods/c236787c-3cc1-4370-a8a0-138fb2709902/volumes" Jul 18 12:23:04.756843 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:23:04.756809 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-a9c69c8d-openshift-default-798666dd8c-cl92h"] Jul 18 12:23:04.757328 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:23:04.757172 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="49cac5e7-2a6f-493d-b280-31637879b57e" containerName="istio-proxy" Jul 18 12:23:04.757328 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:23:04.757185 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="49cac5e7-2a6f-493d-b280-31637879b57e" containerName="istio-proxy" Jul 18 12:23:04.757328 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:23:04.757205 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c236787c-3cc1-4370-a8a0-138fb2709902" containerName="istio-proxy" Jul 18 12:23:04.757328 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:23:04.757211 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="c236787c-3cc1-4370-a8a0-138fb2709902" containerName="istio-proxy" Jul 18 12:23:04.757328 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:23:04.757227 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c236787c-3cc1-4370-a8a0-138fb2709902" containerName="istio-proxy" Jul 18 12:23:04.757328 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:23:04.757233 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="c236787c-3cc1-4370-a8a0-138fb2709902" containerName="istio-proxy" Jul 18 12:23:04.757328 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:23:04.757300 2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="49cac5e7-2a6f-493d-b280-31637879b57e" containerName="istio-proxy" Jul 18 12:23:04.757328 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:23:04.757308 2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="c236787c-3cc1-4370-a8a0-138fb2709902" containerName="istio-proxy" Jul 18 12:23:04.757328 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:23:04.757313 2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="c236787c-3cc1-4370-a8a0-138fb2709902" containerName="istio-proxy" Jul 18 12:23:04.760321 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:23:04.760305 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-a9c69c8d-openshift-default-798666dd8c-cl92h" Jul 18 12:23:04.765756 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:23:04.765724 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-a9c69c8d-openshift-default-dockercfg-ftdlt\"" Jul 18 12:23:04.789929 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:23:04.789906 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-a9c69c8d-openshift-default-798666dd8c-cl92h"] Jul 18 12:23:04.825576 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:23:04.825533 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f45bd5d-ec98-4d74-bcee-9074b403b643-workload-socket\") pod \"e2e-mt-a9c69c8d-openshift-default-798666dd8c-cl92h\" (UID: \"8f45bd5d-ec98-4d74-bcee-9074b403b643\") " pod="openshift-ingress/e2e-mt-a9c69c8d-openshift-default-798666dd8c-cl92h" Jul 18 12:23:04.825687 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:23:04.825583 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8f45bd5d-ec98-4d74-bcee-9074b403b643-workload-certs\") pod \"e2e-mt-a9c69c8d-openshift-default-798666dd8c-cl92h\" (UID: \"8f45bd5d-ec98-4d74-bcee-9074b403b643\") " pod="openshift-ingress/e2e-mt-a9c69c8d-openshift-default-798666dd8c-cl92h" Jul 18 12:23:04.825687 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:23:04.825602 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8f45bd5d-ec98-4d74-bcee-9074b403b643-istio-podinfo\") pod \"e2e-mt-a9c69c8d-openshift-default-798666dd8c-cl92h\" (UID: \"8f45bd5d-ec98-4d74-bcee-9074b403b643\") " pod="openshift-ingress/e2e-mt-a9c69c8d-openshift-default-798666dd8c-cl92h" Jul 18 12:23:04.825687 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:23:04.825638 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8f45bd5d-ec98-4d74-bcee-9074b403b643-istio-data\") pod \"e2e-mt-a9c69c8d-openshift-default-798666dd8c-cl92h\" (UID: \"8f45bd5d-ec98-4d74-bcee-9074b403b643\") " pod="openshift-ingress/e2e-mt-a9c69c8d-openshift-default-798666dd8c-cl92h" Jul 18 12:23:04.825687 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:23:04.825667 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8f45bd5d-ec98-4d74-bcee-9074b403b643-istio-token\") pod \"e2e-mt-a9c69c8d-openshift-default-798666dd8c-cl92h\" (UID: \"8f45bd5d-ec98-4d74-bcee-9074b403b643\") " pod="openshift-ingress/e2e-mt-a9c69c8d-openshift-default-798666dd8c-cl92h" Jul 18 12:23:04.825824 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:23:04.825692 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f45bd5d-ec98-4d74-bcee-9074b403b643-credential-socket\") pod \"e2e-mt-a9c69c8d-openshift-default-798666dd8c-cl92h\" (UID: \"8f45bd5d-ec98-4d74-bcee-9074b403b643\") " pod="openshift-ingress/e2e-mt-a9c69c8d-openshift-default-798666dd8c-cl92h" Jul 18 12:23:04.825824 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:23:04.825730 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8f45bd5d-ec98-4d74-bcee-9074b403b643-istiod-ca-cert\") pod \"e2e-mt-a9c69c8d-openshift-default-798666dd8c-cl92h\" (UID: \"8f45bd5d-ec98-4d74-bcee-9074b403b643\") " pod="openshift-ingress/e2e-mt-a9c69c8d-openshift-default-798666dd8c-cl92h" Jul 18 12:23:04.825824 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:23:04.825783 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8f45bd5d-ec98-4d74-bcee-9074b403b643-istio-envoy\") pod \"e2e-mt-a9c69c8d-openshift-default-798666dd8c-cl92h\" (UID: \"8f45bd5d-ec98-4d74-bcee-9074b403b643\") " pod="openshift-ingress/e2e-mt-a9c69c8d-openshift-default-798666dd8c-cl92h" Jul 18 12:23:04.825917 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:23:04.825822 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gdnxv\" (UniqueName: \"kubernetes.io/projected/8f45bd5d-ec98-4d74-bcee-9074b403b643-kube-api-access-gdnxv\") pod \"e2e-mt-a9c69c8d-openshift-default-798666dd8c-cl92h\" (UID: \"8f45bd5d-ec98-4d74-bcee-9074b403b643\") " pod="openshift-ingress/e2e-mt-a9c69c8d-openshift-default-798666dd8c-cl92h" Jul 18 12:23:04.926430 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:23:04.926390 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8f45bd5d-ec98-4d74-bcee-9074b403b643-istiod-ca-cert\") pod \"e2e-mt-a9c69c8d-openshift-default-798666dd8c-cl92h\" (UID: \"8f45bd5d-ec98-4d74-bcee-9074b403b643\") " pod="openshift-ingress/e2e-mt-a9c69c8d-openshift-default-798666dd8c-cl92h" Jul 18 12:23:04.926621 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:23:04.926455 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8f45bd5d-ec98-4d74-bcee-9074b403b643-istio-envoy\") pod \"e2e-mt-a9c69c8d-openshift-default-798666dd8c-cl92h\" (UID: \"8f45bd5d-ec98-4d74-bcee-9074b403b643\") " pod="openshift-ingress/e2e-mt-a9c69c8d-openshift-default-798666dd8c-cl92h" Jul 18 12:23:04.926621 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:23:04.926474 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gdnxv\" (UniqueName: \"kubernetes.io/projected/8f45bd5d-ec98-4d74-bcee-9074b403b643-kube-api-access-gdnxv\") pod \"e2e-mt-a9c69c8d-openshift-default-798666dd8c-cl92h\" (UID: \"8f45bd5d-ec98-4d74-bcee-9074b403b643\") " pod="openshift-ingress/e2e-mt-a9c69c8d-openshift-default-798666dd8c-cl92h" Jul 18 12:23:04.926621 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:23:04.926511 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f45bd5d-ec98-4d74-bcee-9074b403b643-workload-socket\") pod \"e2e-mt-a9c69c8d-openshift-default-798666dd8c-cl92h\" (UID: \"8f45bd5d-ec98-4d74-bcee-9074b403b643\") " pod="openshift-ingress/e2e-mt-a9c69c8d-openshift-default-798666dd8c-cl92h" Jul 18 12:23:04.926621 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:23:04.926538 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8f45bd5d-ec98-4d74-bcee-9074b403b643-workload-certs\") pod \"e2e-mt-a9c69c8d-openshift-default-798666dd8c-cl92h\" (UID: \"8f45bd5d-ec98-4d74-bcee-9074b403b643\") " pod="openshift-ingress/e2e-mt-a9c69c8d-openshift-default-798666dd8c-cl92h" Jul 18 12:23:04.926621 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:23:04.926612 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8f45bd5d-ec98-4d74-bcee-9074b403b643-istio-podinfo\") pod \"e2e-mt-a9c69c8d-openshift-default-798666dd8c-cl92h\" (UID: \"8f45bd5d-ec98-4d74-bcee-9074b403b643\") " pod="openshift-ingress/e2e-mt-a9c69c8d-openshift-default-798666dd8c-cl92h" Jul 18 12:23:04.926916 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:23:04.926747 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8f45bd5d-ec98-4d74-bcee-9074b403b643-istio-data\") pod \"e2e-mt-a9c69c8d-openshift-default-798666dd8c-cl92h\" (UID: \"8f45bd5d-ec98-4d74-bcee-9074b403b643\") " pod="openshift-ingress/e2e-mt-a9c69c8d-openshift-default-798666dd8c-cl92h" Jul 18 12:23:04.926916 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:23:04.926817 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8f45bd5d-ec98-4d74-bcee-9074b403b643-istio-token\") pod \"e2e-mt-a9c69c8d-openshift-default-798666dd8c-cl92h\" (UID: \"8f45bd5d-ec98-4d74-bcee-9074b403b643\") " pod="openshift-ingress/e2e-mt-a9c69c8d-openshift-default-798666dd8c-cl92h" Jul 18 12:23:04.926916 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:23:04.926869 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f45bd5d-ec98-4d74-bcee-9074b403b643-credential-socket\") pod \"e2e-mt-a9c69c8d-openshift-default-798666dd8c-cl92h\" (UID: \"8f45bd5d-ec98-4d74-bcee-9074b403b643\") " pod="openshift-ingress/e2e-mt-a9c69c8d-openshift-default-798666dd8c-cl92h" Jul 18 12:23:04.927070 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:23:04.926918 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f45bd5d-ec98-4d74-bcee-9074b403b643-workload-socket\") pod \"e2e-mt-a9c69c8d-openshift-default-798666dd8c-cl92h\" (UID: \"8f45bd5d-ec98-4d74-bcee-9074b403b643\") " pod="openshift-ingress/e2e-mt-a9c69c8d-openshift-default-798666dd8c-cl92h" Jul 18 12:23:04.927070 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:23:04.926975 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8f45bd5d-ec98-4d74-bcee-9074b403b643-workload-certs\") pod \"e2e-mt-a9c69c8d-openshift-default-798666dd8c-cl92h\" (UID: \"8f45bd5d-ec98-4d74-bcee-9074b403b643\") " pod="openshift-ingress/e2e-mt-a9c69c8d-openshift-default-798666dd8c-cl92h" Jul 18 12:23:04.927198 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:23:04.927062 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8f45bd5d-ec98-4d74-bcee-9074b403b643-istio-data\") pod \"e2e-mt-a9c69c8d-openshift-default-798666dd8c-cl92h\" (UID: \"8f45bd5d-ec98-4d74-bcee-9074b403b643\") " pod="openshift-ingress/e2e-mt-a9c69c8d-openshift-default-798666dd8c-cl92h" Jul 18 12:23:04.927246 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:23:04.927195 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8f45bd5d-ec98-4d74-bcee-9074b403b643-istiod-ca-cert\") pod \"e2e-mt-a9c69c8d-openshift-default-798666dd8c-cl92h\" (UID: \"8f45bd5d-ec98-4d74-bcee-9074b403b643\") " pod="openshift-ingress/e2e-mt-a9c69c8d-openshift-default-798666dd8c-cl92h" Jul 18 12:23:04.927391 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:23:04.927370 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f45bd5d-ec98-4d74-bcee-9074b403b643-credential-socket\") pod \"e2e-mt-a9c69c8d-openshift-default-798666dd8c-cl92h\" (UID: \"8f45bd5d-ec98-4d74-bcee-9074b403b643\") " pod="openshift-ingress/e2e-mt-a9c69c8d-openshift-default-798666dd8c-cl92h" Jul 18 12:23:04.929058 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:23:04.929035 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8f45bd5d-ec98-4d74-bcee-9074b403b643-istio-envoy\") pod \"e2e-mt-a9c69c8d-openshift-default-798666dd8c-cl92h\" (UID: \"8f45bd5d-ec98-4d74-bcee-9074b403b643\") " pod="openshift-ingress/e2e-mt-a9c69c8d-openshift-default-798666dd8c-cl92h" Jul 18 12:23:04.929191 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:23:04.929173 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8f45bd5d-ec98-4d74-bcee-9074b403b643-istio-podinfo\") pod \"e2e-mt-a9c69c8d-openshift-default-798666dd8c-cl92h\" (UID: \"8f45bd5d-ec98-4d74-bcee-9074b403b643\") " pod="openshift-ingress/e2e-mt-a9c69c8d-openshift-default-798666dd8c-cl92h" Jul 18 12:23:04.935931 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:23:04.935912 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8f45bd5d-ec98-4d74-bcee-9074b403b643-istio-token\") pod \"e2e-mt-a9c69c8d-openshift-default-798666dd8c-cl92h\" (UID: \"8f45bd5d-ec98-4d74-bcee-9074b403b643\") " pod="openshift-ingress/e2e-mt-a9c69c8d-openshift-default-798666dd8c-cl92h" Jul 18 12:23:04.936578 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:23:04.936536 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gdnxv\" (UniqueName: \"kubernetes.io/projected/8f45bd5d-ec98-4d74-bcee-9074b403b643-kube-api-access-gdnxv\") pod \"e2e-mt-a9c69c8d-openshift-default-798666dd8c-cl92h\" (UID: \"8f45bd5d-ec98-4d74-bcee-9074b403b643\") " pod="openshift-ingress/e2e-mt-a9c69c8d-openshift-default-798666dd8c-cl92h" Jul 18 12:23:05.070467 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:23:05.070410 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-a9c69c8d-openshift-default-798666dd8c-cl92h" Jul 18 12:23:05.213988 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:23:05.213962 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-a9c69c8d-openshift-default-798666dd8c-cl92h"] Jul 18 12:23:05.215741 ip-10-0-136-240 kubenswrapper[2585]: W0718 12:23:05.215717 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8f45bd5d_ec98_4d74_bcee_9074b403b643.slice/crio-9197e5f902772863a05286a762be663595545cde0b300210feba57e9b92c7551 WatchSource:0}: Error finding container 9197e5f902772863a05286a762be663595545cde0b300210feba57e9b92c7551: Status 404 returned error can't find the container with id 9197e5f902772863a05286a762be663595545cde0b300210feba57e9b92c7551 Jul 18 12:23:05.218000 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:23:05.217957 2585 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 18 12:23:05.218071 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:23:05.218051 2585 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 18 12:23:05.218109 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:23:05.218094 2585 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 18 12:23:05.630055 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:23:05.630015 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-a9c69c8d-openshift-default-798666dd8c-cl92h" event={"ID":"8f45bd5d-ec98-4d74-bcee-9074b403b643","Type":"ContainerStarted","Data":"ebca501ef00d11d2f99f7ff7e6613abb88c667e982d8643804b5fd7f68722c85"} Jul 18 12:23:05.630225 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:23:05.630077 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-a9c69c8d-openshift-default-798666dd8c-cl92h" event={"ID":"8f45bd5d-ec98-4d74-bcee-9074b403b643","Type":"ContainerStarted","Data":"9197e5f902772863a05286a762be663595545cde0b300210feba57e9b92c7551"} Jul 18 12:23:05.651841 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:23:05.651626 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-a9c69c8d-openshift-default-798666dd8c-cl92h" podStartSLOduration=1.651610882 podStartE2EDuration="1.651610882s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-18 12:23:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-18 12:23:05.651337336 +0000 UTC m=+2052.731543698" watchObservedRunningTime="2026-07-18 12:23:05.651610882 +0000 UTC m=+2052.731817238" Jul 18 12:23:06.070612 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:23:06.070580 2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-a9c69c8d-openshift-default-798666dd8c-cl92h" Jul 18 12:23:06.075210 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:23:06.075187 2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-a9c69c8d-openshift-default-798666dd8c-cl92h" Jul 18 12:23:06.635409 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:23:06.635379 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-a9c69c8d-openshift-default-798666dd8c-cl92h" Jul 18 12:23:06.636248 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:23:06.636228 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-a9c69c8d-openshift-default-798666dd8c-cl92h" Jul 18 12:23:20.722710 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:23:20.722678 2585 generic.go:356] "Generic (PLEG): container finished" podID="8f45bd5d-ec98-4d74-bcee-9074b403b643" containerID="ebca501ef00d11d2f99f7ff7e6613abb88c667e982d8643804b5fd7f68722c85" exitCode=0 Jul 18 12:23:20.723072 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:23:20.722758 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-a9c69c8d-openshift-default-798666dd8c-cl92h" event={"ID":"8f45bd5d-ec98-4d74-bcee-9074b403b643","Type":"ContainerDied","Data":"ebca501ef00d11d2f99f7ff7e6613abb88c667e982d8643804b5fd7f68722c85"} Jul 18 12:23:20.723220 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:23:20.723203 2585 scope.go:117] "RemoveContainer" containerID="ebca501ef00d11d2f99f7ff7e6613abb88c667e982d8643804b5fd7f68722c85" Jul 18 12:23:20.724242 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:23:20.724182 2585 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 18 12:23:20.724303 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:23:20.724248 2585 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 18 12:23:20.724303 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:23:20.724279 2585 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 18 12:23:21.635615 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:23:21.635582 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-a9c69c8d-openshift-default-798666dd8c-cl92h" Jul 18 12:23:21.730940 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:23:21.730901 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-a9c69c8d-openshift-default-798666dd8c-cl92h" event={"ID":"8f45bd5d-ec98-4d74-bcee-9074b403b643","Type":"ContainerStarted","Data":"2308ce0d1914549014ecefe7e1bacc64f0299c3e4de55215a67f33dc4a6d3bd3"} Jul 18 12:23:22.071360 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:23:22.071318 2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-a9c69c8d-openshift-default-798666dd8c-cl92h" Jul 18 12:23:22.072781 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:23:22.072750 2585 patch_prober.go:28] interesting pod/e2e-mt-a9c69c8d-openshift-default-798666dd8c-cl92h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" start-of-body= Jul 18 12:23:22.072994 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:23:22.072969 2585 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-a9c69c8d-openshift-default-798666dd8c-cl92h" podUID="8f45bd5d-ec98-4d74-bcee-9074b403b643" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" Jul 18 12:23:23.071464 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:23:23.071409 2585 patch_prober.go:28] interesting pod/e2e-mt-a9c69c8d-openshift-default-798666dd8c-cl92h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" start-of-body= Jul 18 12:23:23.071976 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:23:23.071498 2585 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-a9c69c8d-openshift-default-798666dd8c-cl92h" podUID="8f45bd5d-ec98-4d74-bcee-9074b403b643" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" Jul 18 12:23:24.071669 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:23:24.071626 2585 patch_prober.go:28] interesting pod/e2e-mt-a9c69c8d-openshift-default-798666dd8c-cl92h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" start-of-body= Jul 18 12:23:24.072159 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:23:24.071692 2585 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-a9c69c8d-openshift-default-798666dd8c-cl92h" podUID="8f45bd5d-ec98-4d74-bcee-9074b403b643" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" Jul 18 12:23:25.071403 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:23:25.071361 2585 patch_prober.go:28] interesting pod/e2e-mt-a9c69c8d-openshift-default-798666dd8c-cl92h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" start-of-body= Jul 18 12:23:25.071628 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:23:25.071439 2585 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-a9c69c8d-openshift-default-798666dd8c-cl92h" podUID="8f45bd5d-ec98-4d74-bcee-9074b403b643" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" Jul 18 12:23:26.075175 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:23:26.075145 2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-a9c69c8d-openshift-default-798666dd8c-cl92h" Jul 18 12:23:26.075535 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:23:26.075396 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-a9c69c8d-openshift-default-798666dd8c-cl92h" Jul 18 12:23:26.076107 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:23:26.076090 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-a9c69c8d-openshift-default-798666dd8c-cl92h" Jul 18 12:23:41.278170 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:23:41.278130 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-a9c69c8d-openshift-default-798666dd8c-cl92h"] Jul 18 12:23:41.278710 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:23:41.278319 2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-a9c69c8d-openshift-default-798666dd8c-cl92h" podUID="8f45bd5d-ec98-4d74-bcee-9074b403b643" containerName="istio-proxy" containerID="cri-o://2308ce0d1914549014ecefe7e1bacc64f0299c3e4de55215a67f33dc4a6d3bd3" gracePeriod=30 Jul 18 12:23:46.540133 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:23:46.540106 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-a9c69c8d-openshift-default-798666dd8c-cl92h" Jul 18 12:23:46.697945 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:23:46.697859 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f45bd5d-ec98-4d74-bcee-9074b403b643-credential-socket\") pod \"8f45bd5d-ec98-4d74-bcee-9074b403b643\" (UID: \"8f45bd5d-ec98-4d74-bcee-9074b403b643\") " Jul 18 12:23:46.697945 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:23:46.697926 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f45bd5d-ec98-4d74-bcee-9074b403b643-workload-socket\") pod \"8f45bd5d-ec98-4d74-bcee-9074b403b643\" (UID: \"8f45bd5d-ec98-4d74-bcee-9074b403b643\") " Jul 18 12:23:46.698156 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:23:46.697958 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gdnxv\" (UniqueName: \"kubernetes.io/projected/8f45bd5d-ec98-4d74-bcee-9074b403b643-kube-api-access-gdnxv\") pod \"8f45bd5d-ec98-4d74-bcee-9074b403b643\" (UID: \"8f45bd5d-ec98-4d74-bcee-9074b403b643\") " Jul 18 12:23:46.698156 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:23:46.698001 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8f45bd5d-ec98-4d74-bcee-9074b403b643-istiod-ca-cert\") pod \"8f45bd5d-ec98-4d74-bcee-9074b403b643\" (UID: \"8f45bd5d-ec98-4d74-bcee-9074b403b643\") " Jul 18 12:23:46.698156 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:23:46.698043 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8f45bd5d-ec98-4d74-bcee-9074b403b643-workload-certs\") pod \"8f45bd5d-ec98-4d74-bcee-9074b403b643\" (UID: \"8f45bd5d-ec98-4d74-bcee-9074b403b643\") " Jul 18 12:23:46.698156 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:23:46.698084 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8f45bd5d-ec98-4d74-bcee-9074b403b643-istio-podinfo\") pod \"8f45bd5d-ec98-4d74-bcee-9074b403b643\" (UID: \"8f45bd5d-ec98-4d74-bcee-9074b403b643\") " Jul 18 12:23:46.698156 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:23:46.698119 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8f45bd5d-ec98-4d74-bcee-9074b403b643-istio-data\") pod \"8f45bd5d-ec98-4d74-bcee-9074b403b643\" (UID: \"8f45bd5d-ec98-4d74-bcee-9074b403b643\") " Jul 18 12:23:46.698156 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:23:46.698144 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8f45bd5d-ec98-4d74-bcee-9074b403b643-istio-token\") pod \"8f45bd5d-ec98-4d74-bcee-9074b403b643\" (UID: \"8f45bd5d-ec98-4d74-bcee-9074b403b643\") " Jul 18 12:23:46.698455 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:23:46.698165 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8f45bd5d-ec98-4d74-bcee-9074b403b643-istio-envoy\") pod \"8f45bd5d-ec98-4d74-bcee-9074b403b643\" (UID: \"8f45bd5d-ec98-4d74-bcee-9074b403b643\") " Jul 18 12:23:46.698455 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:23:46.698155 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8f45bd5d-ec98-4d74-bcee-9074b403b643-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "8f45bd5d-ec98-4d74-bcee-9074b403b643" (UID: "8f45bd5d-ec98-4d74-bcee-9074b403b643"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 18 12:23:46.698455 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:23:46.698206 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8f45bd5d-ec98-4d74-bcee-9074b403b643-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "8f45bd5d-ec98-4d74-bcee-9074b403b643" (UID: "8f45bd5d-ec98-4d74-bcee-9074b403b643"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 18 12:23:46.698642 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:23:46.698534 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8f45bd5d-ec98-4d74-bcee-9074b403b643-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "8f45bd5d-ec98-4d74-bcee-9074b403b643" (UID: "8f45bd5d-ec98-4d74-bcee-9074b403b643"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 18 12:23:46.698692 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:23:46.698653 2585 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f45bd5d-ec98-4d74-bcee-9074b403b643-credential-socket\") on node \"ip-10-0-136-240.ec2.internal\" DevicePath \"\"" Jul 18 12:23:46.698692 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:23:46.698687 2585 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f45bd5d-ec98-4d74-bcee-9074b403b643-workload-socket\") on node \"ip-10-0-136-240.ec2.internal\" DevicePath \"\"" Jul 18 12:23:46.698789 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:23:46.698746 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8f45bd5d-ec98-4d74-bcee-9074b403b643-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "8f45bd5d-ec98-4d74-bcee-9074b403b643" (UID: "8f45bd5d-ec98-4d74-bcee-9074b403b643"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 18 12:23:46.699373 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:23:46.699337 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8f45bd5d-ec98-4d74-bcee-9074b403b643-istio-data" (OuterVolumeSpecName: "istio-data") pod "8f45bd5d-ec98-4d74-bcee-9074b403b643" (UID: "8f45bd5d-ec98-4d74-bcee-9074b403b643"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 18 12:23:46.700645 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:23:46.700614 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8f45bd5d-ec98-4d74-bcee-9074b403b643-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "8f45bd5d-ec98-4d74-bcee-9074b403b643" (UID: "8f45bd5d-ec98-4d74-bcee-9074b403b643"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 18 12:23:46.700645 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:23:46.700628 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8f45bd5d-ec98-4d74-bcee-9074b403b643-istio-token" (OuterVolumeSpecName: "istio-token") pod "8f45bd5d-ec98-4d74-bcee-9074b403b643" (UID: "8f45bd5d-ec98-4d74-bcee-9074b403b643"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 18 12:23:46.700771 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:23:46.700722 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8f45bd5d-ec98-4d74-bcee-9074b403b643-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "8f45bd5d-ec98-4d74-bcee-9074b403b643" (UID: "8f45bd5d-ec98-4d74-bcee-9074b403b643"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 18 12:23:46.700877 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:23:46.700862 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8f45bd5d-ec98-4d74-bcee-9074b403b643-kube-api-access-gdnxv" (OuterVolumeSpecName: "kube-api-access-gdnxv") pod "8f45bd5d-ec98-4d74-bcee-9074b403b643" (UID: "8f45bd5d-ec98-4d74-bcee-9074b403b643"). InnerVolumeSpecName "kube-api-access-gdnxv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 18 12:23:46.799968 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:23:46.799941 2585 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8f45bd5d-ec98-4d74-bcee-9074b403b643-istiod-ca-cert\") on node \"ip-10-0-136-240.ec2.internal\" DevicePath \"\"" Jul 18 12:23:46.799968 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:23:46.799964 2585 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8f45bd5d-ec98-4d74-bcee-9074b403b643-workload-certs\") on node \"ip-10-0-136-240.ec2.internal\" DevicePath \"\"" Jul 18 12:23:46.799968 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:23:46.799973 2585 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8f45bd5d-ec98-4d74-bcee-9074b403b643-istio-podinfo\") on node \"ip-10-0-136-240.ec2.internal\" DevicePath \"\"" Jul 18 12:23:46.800144 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:23:46.799980 2585 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8f45bd5d-ec98-4d74-bcee-9074b403b643-istio-data\") on node \"ip-10-0-136-240.ec2.internal\" DevicePath \"\"" Jul 18 12:23:46.800144 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:23:46.799989 2585 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8f45bd5d-ec98-4d74-bcee-9074b403b643-istio-token\") on node \"ip-10-0-136-240.ec2.internal\" DevicePath \"\"" Jul 18 12:23:46.800144 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:23:46.799996 2585 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8f45bd5d-ec98-4d74-bcee-9074b403b643-istio-envoy\") on node \"ip-10-0-136-240.ec2.internal\" DevicePath \"\"" Jul 18 12:23:46.800144 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:23:46.800004 2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gdnxv\" (UniqueName: \"kubernetes.io/projected/8f45bd5d-ec98-4d74-bcee-9074b403b643-kube-api-access-gdnxv\") on node \"ip-10-0-136-240.ec2.internal\" DevicePath \"\"" Jul 18 12:23:46.853091 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:23:46.853060 2585 generic.go:356] "Generic (PLEG): container finished" podID="8f45bd5d-ec98-4d74-bcee-9074b403b643" containerID="2308ce0d1914549014ecefe7e1bacc64f0299c3e4de55215a67f33dc4a6d3bd3" exitCode=0 Jul 18 12:23:46.853229 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:23:46.853116 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-a9c69c8d-openshift-default-798666dd8c-cl92h" Jul 18 12:23:46.853229 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:23:46.853149 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-a9c69c8d-openshift-default-798666dd8c-cl92h" event={"ID":"8f45bd5d-ec98-4d74-bcee-9074b403b643","Type":"ContainerDied","Data":"2308ce0d1914549014ecefe7e1bacc64f0299c3e4de55215a67f33dc4a6d3bd3"} Jul 18 12:23:46.853229 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:23:46.853202 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-a9c69c8d-openshift-default-798666dd8c-cl92h" event={"ID":"8f45bd5d-ec98-4d74-bcee-9074b403b643","Type":"ContainerDied","Data":"9197e5f902772863a05286a762be663595545cde0b300210feba57e9b92c7551"} Jul 18 12:23:46.853229 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:23:46.853224 2585 scope.go:117] "RemoveContainer" containerID="2308ce0d1914549014ecefe7e1bacc64f0299c3e4de55215a67f33dc4a6d3bd3" Jul 18 12:23:46.862596 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:23:46.862580 2585 scope.go:117] "RemoveContainer" containerID="ebca501ef00d11d2f99f7ff7e6613abb88c667e982d8643804b5fd7f68722c85" Jul 18 12:23:46.870539 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:23:46.870523 2585 scope.go:117] "RemoveContainer" containerID="2308ce0d1914549014ecefe7e1bacc64f0299c3e4de55215a67f33dc4a6d3bd3" Jul 18 12:23:46.870800 ip-10-0-136-240 kubenswrapper[2585]: E0718 12:23:46.870782 2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2308ce0d1914549014ecefe7e1bacc64f0299c3e4de55215a67f33dc4a6d3bd3\": container with ID starting with 2308ce0d1914549014ecefe7e1bacc64f0299c3e4de55215a67f33dc4a6d3bd3 not found: ID does not exist" containerID="2308ce0d1914549014ecefe7e1bacc64f0299c3e4de55215a67f33dc4a6d3bd3" Jul 18 12:23:46.870871 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:23:46.870812 2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2308ce0d1914549014ecefe7e1bacc64f0299c3e4de55215a67f33dc4a6d3bd3"} err="failed to get container status \"2308ce0d1914549014ecefe7e1bacc64f0299c3e4de55215a67f33dc4a6d3bd3\": rpc error: code = NotFound desc = could not find container \"2308ce0d1914549014ecefe7e1bacc64f0299c3e4de55215a67f33dc4a6d3bd3\": container with ID starting with 2308ce0d1914549014ecefe7e1bacc64f0299c3e4de55215a67f33dc4a6d3bd3 not found: ID does not exist" Jul 18 12:23:46.870871 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:23:46.870837 2585 scope.go:117] "RemoveContainer" containerID="ebca501ef00d11d2f99f7ff7e6613abb88c667e982d8643804b5fd7f68722c85" Jul 18 12:23:46.871090 ip-10-0-136-240 kubenswrapper[2585]: E0718 12:23:46.871073 2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ebca501ef00d11d2f99f7ff7e6613abb88c667e982d8643804b5fd7f68722c85\": container with ID starting with ebca501ef00d11d2f99f7ff7e6613abb88c667e982d8643804b5fd7f68722c85 not found: ID does not exist" containerID="ebca501ef00d11d2f99f7ff7e6613abb88c667e982d8643804b5fd7f68722c85" Jul 18 12:23:46.871129 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:23:46.871097 2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ebca501ef00d11d2f99f7ff7e6613abb88c667e982d8643804b5fd7f68722c85"} err="failed to get container status \"ebca501ef00d11d2f99f7ff7e6613abb88c667e982d8643804b5fd7f68722c85\": rpc error: code = NotFound desc = could not find container \"ebca501ef00d11d2f99f7ff7e6613abb88c667e982d8643804b5fd7f68722c85\": container with ID starting with ebca501ef00d11d2f99f7ff7e6613abb88c667e982d8643804b5fd7f68722c85 not found: ID does not exist" Jul 18 12:23:46.877105 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:23:46.877084 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-a9c69c8d-openshift-default-798666dd8c-cl92h"] Jul 18 12:23:46.881816 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:23:46.881798 2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-a9c69c8d-openshift-default-798666dd8c-cl92h"] Jul 18 12:23:47.469274 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:23:47.469239 2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8f45bd5d-ec98-4d74-bcee-9074b403b643" path="/var/lib/kubelet/pods/8f45bd5d-ec98-4d74-bcee-9074b403b643/volumes" Jul 18 12:23:53.553175 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:23:53.553145 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-j9l7p_8ec47eff-5fbc-4130-baef-d8544aa4e786/console-operator/1.log" Jul 18 12:23:53.557802 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:23:53.557767 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-j9l7p_8ec47eff-5fbc-4130-baef-d8544aa4e786/console-operator/1.log" Jul 18 12:23:53.560847 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:23:53.560829 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-wq4lg_247c97c8-982b-4176-ae10-afc396745360/ovn-acl-logging/0.log" Jul 18 12:23:53.565078 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:23:53.565060 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-wq4lg_247c97c8-982b-4176-ae10-afc396745360/ovn-acl-logging/0.log" Jul 18 12:24:01.854266 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:24:01.854226 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-07bfbcac-openshift-default-78fbf5fb95-l986p"] Jul 18 12:24:01.854807 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:24:01.854787 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="49cac5e7-2a6f-493d-b280-31637879b57e" containerName="istio-proxy" Jul 18 12:24:01.854887 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:24:01.854809 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="49cac5e7-2a6f-493d-b280-31637879b57e" containerName="istio-proxy" Jul 18 12:24:01.854887 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:24:01.854826 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8f45bd5d-ec98-4d74-bcee-9074b403b643" containerName="istio-proxy" Jul 18 12:24:01.854887 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:24:01.854834 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="8f45bd5d-ec98-4d74-bcee-9074b403b643" containerName="istio-proxy" Jul 18 12:24:01.854887 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:24:01.854852 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8f45bd5d-ec98-4d74-bcee-9074b403b643" containerName="istio-proxy" Jul 18 12:24:01.854887 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:24:01.854862 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="8f45bd5d-ec98-4d74-bcee-9074b403b643" containerName="istio-proxy" Jul 18 12:24:01.855132 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:24:01.854988 2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="8f45bd5d-ec98-4d74-bcee-9074b403b643" containerName="istio-proxy" Jul 18 12:24:01.855132 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:24:01.855003 2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="49cac5e7-2a6f-493d-b280-31637879b57e" containerName="istio-proxy" Jul 18 12:24:01.858034 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:24:01.858014 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-07bfbcac-openshift-default-78fbf5fb95-l986p" Jul 18 12:24:01.861254 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:24:01.861235 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-07bfbcac-openshift-default-dockercfg-v4chz\"" Jul 18 12:24:01.880388 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:24:01.880365 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-07bfbcac-openshift-default-78fbf5fb95-l986p"] Jul 18 12:24:01.922861 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:24:01.922830 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b5b5c317-e5c5-4d79-bb4e-63559c8c8268-istio-data\") pod \"e2e-mt-07bfbcac-openshift-default-78fbf5fb95-l986p\" (UID: \"b5b5c317-e5c5-4d79-bb4e-63559c8c8268\") " pod="openshift-ingress/e2e-mt-07bfbcac-openshift-default-78fbf5fb95-l986p" Jul 18 12:24:01.923063 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:24:01.922891 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zp6pv\" (UniqueName: \"kubernetes.io/projected/b5b5c317-e5c5-4d79-bb4e-63559c8c8268-kube-api-access-zp6pv\") pod \"e2e-mt-07bfbcac-openshift-default-78fbf5fb95-l986p\" (UID: \"b5b5c317-e5c5-4d79-bb4e-63559c8c8268\") " pod="openshift-ingress/e2e-mt-07bfbcac-openshift-default-78fbf5fb95-l986p" Jul 18 12:24:01.923174 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:24:01.923055 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b5b5c317-e5c5-4d79-bb4e-63559c8c8268-istiod-ca-cert\") pod \"e2e-mt-07bfbcac-openshift-default-78fbf5fb95-l986p\" (UID: \"b5b5c317-e5c5-4d79-bb4e-63559c8c8268\") " pod="openshift-ingress/e2e-mt-07bfbcac-openshift-default-78fbf5fb95-l986p" Jul 18 12:24:01.923174 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:24:01.923122 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5b5c317-e5c5-4d79-bb4e-63559c8c8268-credential-socket\") pod \"e2e-mt-07bfbcac-openshift-default-78fbf5fb95-l986p\" (UID: \"b5b5c317-e5c5-4d79-bb4e-63559c8c8268\") " pod="openshift-ingress/e2e-mt-07bfbcac-openshift-default-78fbf5fb95-l986p" Jul 18 12:24:01.923294 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:24:01.923172 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b5b5c317-e5c5-4d79-bb4e-63559c8c8268-istio-podinfo\") pod \"e2e-mt-07bfbcac-openshift-default-78fbf5fb95-l986p\" (UID: \"b5b5c317-e5c5-4d79-bb4e-63559c8c8268\") " pod="openshift-ingress/e2e-mt-07bfbcac-openshift-default-78fbf5fb95-l986p" Jul 18 12:24:01.923294 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:24:01.923204 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5b5c317-e5c5-4d79-bb4e-63559c8c8268-workload-socket\") pod \"e2e-mt-07bfbcac-openshift-default-78fbf5fb95-l986p\" (UID: \"b5b5c317-e5c5-4d79-bb4e-63559c8c8268\") " pod="openshift-ingress/e2e-mt-07bfbcac-openshift-default-78fbf5fb95-l986p" Jul 18 12:24:01.923294 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:24:01.923253 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b5b5c317-e5c5-4d79-bb4e-63559c8c8268-istio-envoy\") pod \"e2e-mt-07bfbcac-openshift-default-78fbf5fb95-l986p\" (UID: \"b5b5c317-e5c5-4d79-bb4e-63559c8c8268\") " pod="openshift-ingress/e2e-mt-07bfbcac-openshift-default-78fbf5fb95-l986p" Jul 18 12:24:01.923449 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:24:01.923297 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b5b5c317-e5c5-4d79-bb4e-63559c8c8268-istio-token\") pod \"e2e-mt-07bfbcac-openshift-default-78fbf5fb95-l986p\" (UID: \"b5b5c317-e5c5-4d79-bb4e-63559c8c8268\") " pod="openshift-ingress/e2e-mt-07bfbcac-openshift-default-78fbf5fb95-l986p" Jul 18 12:24:01.923449 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:24:01.923374 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b5b5c317-e5c5-4d79-bb4e-63559c8c8268-workload-certs\") pod \"e2e-mt-07bfbcac-openshift-default-78fbf5fb95-l986p\" (UID: \"b5b5c317-e5c5-4d79-bb4e-63559c8c8268\") " pod="openshift-ingress/e2e-mt-07bfbcac-openshift-default-78fbf5fb95-l986p" Jul 18 12:24:02.023975 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:24:02.023942 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b5b5c317-e5c5-4d79-bb4e-63559c8c8268-istio-data\") pod \"e2e-mt-07bfbcac-openshift-default-78fbf5fb95-l986p\" (UID: \"b5b5c317-e5c5-4d79-bb4e-63559c8c8268\") " pod="openshift-ingress/e2e-mt-07bfbcac-openshift-default-78fbf5fb95-l986p" Jul 18 12:24:02.024175 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:24:02.023981 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zp6pv\" (UniqueName: \"kubernetes.io/projected/b5b5c317-e5c5-4d79-bb4e-63559c8c8268-kube-api-access-zp6pv\") pod \"e2e-mt-07bfbcac-openshift-default-78fbf5fb95-l986p\" (UID: \"b5b5c317-e5c5-4d79-bb4e-63559c8c8268\") " pod="openshift-ingress/e2e-mt-07bfbcac-openshift-default-78fbf5fb95-l986p" Jul 18 12:24:02.024175 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:24:02.024003 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b5b5c317-e5c5-4d79-bb4e-63559c8c8268-istiod-ca-cert\") pod \"e2e-mt-07bfbcac-openshift-default-78fbf5fb95-l986p\" (UID: \"b5b5c317-e5c5-4d79-bb4e-63559c8c8268\") " pod="openshift-ingress/e2e-mt-07bfbcac-openshift-default-78fbf5fb95-l986p" Jul 18 12:24:02.024175 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:24:02.024028 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5b5c317-e5c5-4d79-bb4e-63559c8c8268-credential-socket\") pod \"e2e-mt-07bfbcac-openshift-default-78fbf5fb95-l986p\" (UID: \"b5b5c317-e5c5-4d79-bb4e-63559c8c8268\") " pod="openshift-ingress/e2e-mt-07bfbcac-openshift-default-78fbf5fb95-l986p" Jul 18 12:24:02.024175 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:24:02.024049 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b5b5c317-e5c5-4d79-bb4e-63559c8c8268-istio-podinfo\") pod \"e2e-mt-07bfbcac-openshift-default-78fbf5fb95-l986p\" (UID: \"b5b5c317-e5c5-4d79-bb4e-63559c8c8268\") " pod="openshift-ingress/e2e-mt-07bfbcac-openshift-default-78fbf5fb95-l986p" Jul 18 12:24:02.024175 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:24:02.024075 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5b5c317-e5c5-4d79-bb4e-63559c8c8268-workload-socket\") pod \"e2e-mt-07bfbcac-openshift-default-78fbf5fb95-l986p\" (UID: \"b5b5c317-e5c5-4d79-bb4e-63559c8c8268\") " pod="openshift-ingress/e2e-mt-07bfbcac-openshift-default-78fbf5fb95-l986p" Jul 18 12:24:02.024175 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:24:02.024116 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b5b5c317-e5c5-4d79-bb4e-63559c8c8268-istio-envoy\") pod \"e2e-mt-07bfbcac-openshift-default-78fbf5fb95-l986p\" (UID: \"b5b5c317-e5c5-4d79-bb4e-63559c8c8268\") " pod="openshift-ingress/e2e-mt-07bfbcac-openshift-default-78fbf5fb95-l986p" Jul 18 12:24:02.024175 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:24:02.024153 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b5b5c317-e5c5-4d79-bb4e-63559c8c8268-istio-token\") pod \"e2e-mt-07bfbcac-openshift-default-78fbf5fb95-l986p\" (UID: \"b5b5c317-e5c5-4d79-bb4e-63559c8c8268\") " pod="openshift-ingress/e2e-mt-07bfbcac-openshift-default-78fbf5fb95-l986p" Jul 18 12:24:02.024596 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:24:02.024239 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b5b5c317-e5c5-4d79-bb4e-63559c8c8268-workload-certs\") pod \"e2e-mt-07bfbcac-openshift-default-78fbf5fb95-l986p\" (UID: \"b5b5c317-e5c5-4d79-bb4e-63559c8c8268\") " pod="openshift-ingress/e2e-mt-07bfbcac-openshift-default-78fbf5fb95-l986p" Jul 18 12:24:02.024596 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:24:02.024403 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b5b5c317-e5c5-4d79-bb4e-63559c8c8268-istio-data\") pod \"e2e-mt-07bfbcac-openshift-default-78fbf5fb95-l986p\" (UID: \"b5b5c317-e5c5-4d79-bb4e-63559c8c8268\") " pod="openshift-ingress/e2e-mt-07bfbcac-openshift-default-78fbf5fb95-l986p" Jul 18 12:24:02.024596 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:24:02.024437 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5b5c317-e5c5-4d79-bb4e-63559c8c8268-credential-socket\") pod \"e2e-mt-07bfbcac-openshift-default-78fbf5fb95-l986p\" (UID: \"b5b5c317-e5c5-4d79-bb4e-63559c8c8268\") " pod="openshift-ingress/e2e-mt-07bfbcac-openshift-default-78fbf5fb95-l986p" Jul 18 12:24:02.024782 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:24:02.024613 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5b5c317-e5c5-4d79-bb4e-63559c8c8268-workload-socket\") pod \"e2e-mt-07bfbcac-openshift-default-78fbf5fb95-l986p\" (UID: \"b5b5c317-e5c5-4d79-bb4e-63559c8c8268\") " pod="openshift-ingress/e2e-mt-07bfbcac-openshift-default-78fbf5fb95-l986p" Jul 18 12:24:02.024782 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:24:02.024631 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b5b5c317-e5c5-4d79-bb4e-63559c8c8268-workload-certs\") pod \"e2e-mt-07bfbcac-openshift-default-78fbf5fb95-l986p\" (UID: \"b5b5c317-e5c5-4d79-bb4e-63559c8c8268\") " pod="openshift-ingress/e2e-mt-07bfbcac-openshift-default-78fbf5fb95-l986p" Jul 18 12:24:02.024852 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:24:02.024799 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b5b5c317-e5c5-4d79-bb4e-63559c8c8268-istiod-ca-cert\") pod \"e2e-mt-07bfbcac-openshift-default-78fbf5fb95-l986p\" (UID: \"b5b5c317-e5c5-4d79-bb4e-63559c8c8268\") " pod="openshift-ingress/e2e-mt-07bfbcac-openshift-default-78fbf5fb95-l986p" Jul 18 12:24:02.026700 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:24:02.026674 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b5b5c317-e5c5-4d79-bb4e-63559c8c8268-istio-envoy\") pod \"e2e-mt-07bfbcac-openshift-default-78fbf5fb95-l986p\" (UID: \"b5b5c317-e5c5-4d79-bb4e-63559c8c8268\") " pod="openshift-ingress/e2e-mt-07bfbcac-openshift-default-78fbf5fb95-l986p" Jul 18 12:24:02.026802 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:24:02.026703 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b5b5c317-e5c5-4d79-bb4e-63559c8c8268-istio-podinfo\") pod \"e2e-mt-07bfbcac-openshift-default-78fbf5fb95-l986p\" (UID: \"b5b5c317-e5c5-4d79-bb4e-63559c8c8268\") " pod="openshift-ingress/e2e-mt-07bfbcac-openshift-default-78fbf5fb95-l986p" Jul 18 12:24:02.032567 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:24:02.032529 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b5b5c317-e5c5-4d79-bb4e-63559c8c8268-istio-token\") pod \"e2e-mt-07bfbcac-openshift-default-78fbf5fb95-l986p\" (UID: \"b5b5c317-e5c5-4d79-bb4e-63559c8c8268\") " pod="openshift-ingress/e2e-mt-07bfbcac-openshift-default-78fbf5fb95-l986p" Jul 18 12:24:02.032656 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:24:02.032611 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zp6pv\" (UniqueName: \"kubernetes.io/projected/b5b5c317-e5c5-4d79-bb4e-63559c8c8268-kube-api-access-zp6pv\") pod \"e2e-mt-07bfbcac-openshift-default-78fbf5fb95-l986p\" (UID: \"b5b5c317-e5c5-4d79-bb4e-63559c8c8268\") " pod="openshift-ingress/e2e-mt-07bfbcac-openshift-default-78fbf5fb95-l986p" Jul 18 12:24:02.168154 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:24:02.168066 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-07bfbcac-openshift-default-78fbf5fb95-l986p" Jul 18 12:24:02.310597 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:24:02.310571 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-07bfbcac-openshift-default-78fbf5fb95-l986p"] Jul 18 12:24:02.312394 ip-10-0-136-240 kubenswrapper[2585]: W0718 12:24:02.312368 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb5b5c317_e5c5_4d79_bb4e_63559c8c8268.slice/crio-5d98845a07e181d1a1063a88790a201968950e7f240897fd41c26d4eec6e8892 WatchSource:0}: Error finding container 5d98845a07e181d1a1063a88790a201968950e7f240897fd41c26d4eec6e8892: Status 404 returned error can't find the container with id 5d98845a07e181d1a1063a88790a201968950e7f240897fd41c26d4eec6e8892 Jul 18 12:24:02.314628 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:24:02.314600 2585 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 18 12:24:02.314717 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:24:02.314662 2585 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 18 12:24:02.314717 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:24:02.314691 2585 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 18 12:24:02.934837 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:24:02.934795 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-07bfbcac-openshift-default-78fbf5fb95-l986p" event={"ID":"b5b5c317-e5c5-4d79-bb4e-63559c8c8268","Type":"ContainerStarted","Data":"57f6c2183c527bde730527c826752f7d4975ed7717e08d29f73651d833da3a64"} Jul 18 12:24:02.935453 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:24:02.934861 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-07bfbcac-openshift-default-78fbf5fb95-l986p" event={"ID":"b5b5c317-e5c5-4d79-bb4e-63559c8c8268","Type":"ContainerStarted","Data":"5d98845a07e181d1a1063a88790a201968950e7f240897fd41c26d4eec6e8892"} Jul 18 12:24:02.956414 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:24:02.956358 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-07bfbcac-openshift-default-78fbf5fb95-l986p" podStartSLOduration=1.956337276 podStartE2EDuration="1.956337276s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-18 12:24:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-18 12:24:02.955573922 +0000 UTC m=+2110.035780271" watchObservedRunningTime="2026-07-18 12:24:02.956337276 +0000 UTC m=+2110.036543633" Jul 18 12:24:03.168954 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:24:03.168920 2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-07bfbcac-openshift-default-78fbf5fb95-l986p" Jul 18 12:24:03.174310 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:24:03.174286 2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-07bfbcac-openshift-default-78fbf5fb95-l986p" Jul 18 12:24:03.940061 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:24:03.940030 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-07bfbcac-openshift-default-78fbf5fb95-l986p" Jul 18 12:24:03.940942 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:24:03.940922 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-07bfbcac-openshift-default-78fbf5fb95-l986p" Jul 18 12:24:18.033469 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:24:18.033434 2585 generic.go:356] "Generic (PLEG): container finished" podID="b5b5c317-e5c5-4d79-bb4e-63559c8c8268" containerID="57f6c2183c527bde730527c826752f7d4975ed7717e08d29f73651d833da3a64" exitCode=0 Jul 18 12:24:18.033903 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:24:18.033506 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-07bfbcac-openshift-default-78fbf5fb95-l986p" event={"ID":"b5b5c317-e5c5-4d79-bb4e-63559c8c8268","Type":"ContainerDied","Data":"57f6c2183c527bde730527c826752f7d4975ed7717e08d29f73651d833da3a64"} Jul 18 12:24:18.033948 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:24:18.033910 2585 scope.go:117] "RemoveContainer" containerID="57f6c2183c527bde730527c826752f7d4975ed7717e08d29f73651d833da3a64" Jul 18 12:24:18.034938 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:24:18.034903 2585 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 18 12:24:18.035060 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:24:18.034971 2585 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 18 12:24:18.035060 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:24:18.035003 2585 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 18 12:24:18.645424 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:24:18.645394 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-mt-07bfbcac/e2e-lifecycle-model-07bfbcac-kserve-6b5f58f7cf-pnt8r"] Jul 18 12:24:18.645931 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:24:18.645911 2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="8f45bd5d-ec98-4d74-bcee-9074b403b643" containerName="istio-proxy" Jul 18 12:24:18.649044 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:24:18.649019 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-07bfbcac/e2e-lifecycle-model-07bfbcac-kserve-6b5f58f7cf-pnt8r" Jul 18 12:24:18.651689 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:24:18.651665 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-07bfbcac\"/\"kube-root-ca.crt\"" Jul 18 12:24:18.652794 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:24:18.652768 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-07bfbcac\"/\"default-dockercfg-65wzp\"" Jul 18 12:24:18.652871 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:24:18.652769 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-07bfbcac\"/\"e2e-lifecycle-model-07bfbcac-kserve-self-signed-certs\"" Jul 18 12:24:18.652871 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:24:18.652826 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-07bfbcac\"/\"openshift-service-ca.crt\"" Jul 18 12:24:18.661223 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:24:18.661203 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-07bfbcac/e2e-lifecycle-model-07bfbcac-kserve-6b5f58f7cf-pnt8r"] Jul 18 12:24:18.681744 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:24:18.681696 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6e27cfc7-e129-4ed9-bb21-8dea0a55ae04-dshm\") pod \"e2e-lifecycle-model-07bfbcac-kserve-6b5f58f7cf-pnt8r\" (UID: \"6e27cfc7-e129-4ed9-bb21-8dea0a55ae04\") " pod="ai-tenant-e2e-mt-07bfbcac/e2e-lifecycle-model-07bfbcac-kserve-6b5f58f7cf-pnt8r" Jul 18 12:24:18.681894 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:24:18.681826 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6e27cfc7-e129-4ed9-bb21-8dea0a55ae04-home\") pod \"e2e-lifecycle-model-07bfbcac-kserve-6b5f58f7cf-pnt8r\" (UID: \"6e27cfc7-e129-4ed9-bb21-8dea0a55ae04\") " pod="ai-tenant-e2e-mt-07bfbcac/e2e-lifecycle-model-07bfbcac-kserve-6b5f58f7cf-pnt8r" Jul 18 12:24:18.681941 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:24:18.681882 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6e27cfc7-e129-4ed9-bb21-8dea0a55ae04-model-cache\") pod \"e2e-lifecycle-model-07bfbcac-kserve-6b5f58f7cf-pnt8r\" (UID: \"6e27cfc7-e129-4ed9-bb21-8dea0a55ae04\") " pod="ai-tenant-e2e-mt-07bfbcac/e2e-lifecycle-model-07bfbcac-kserve-6b5f58f7cf-pnt8r" Jul 18 12:24:18.681993 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:24:18.681962 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6e27cfc7-e129-4ed9-bb21-8dea0a55ae04-tls-certs\") pod \"e2e-lifecycle-model-07bfbcac-kserve-6b5f58f7cf-pnt8r\" (UID: \"6e27cfc7-e129-4ed9-bb21-8dea0a55ae04\") " pod="ai-tenant-e2e-mt-07bfbcac/e2e-lifecycle-model-07bfbcac-kserve-6b5f58f7cf-pnt8r" Jul 18 12:24:18.682056 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:24:18.682027 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6e27cfc7-e129-4ed9-bb21-8dea0a55ae04-tmp-dir\") pod \"e2e-lifecycle-model-07bfbcac-kserve-6b5f58f7cf-pnt8r\" (UID: \"6e27cfc7-e129-4ed9-bb21-8dea0a55ae04\") " pod="ai-tenant-e2e-mt-07bfbcac/e2e-lifecycle-model-07bfbcac-kserve-6b5f58f7cf-pnt8r" Jul 18 12:24:18.682147 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:24:18.682127 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ms2sq\" (UniqueName: \"kubernetes.io/projected/6e27cfc7-e129-4ed9-bb21-8dea0a55ae04-kube-api-access-ms2sq\") pod \"e2e-lifecycle-model-07bfbcac-kserve-6b5f58f7cf-pnt8r\" (UID: \"6e27cfc7-e129-4ed9-bb21-8dea0a55ae04\") " pod="ai-tenant-e2e-mt-07bfbcac/e2e-lifecycle-model-07bfbcac-kserve-6b5f58f7cf-pnt8r" Jul 18 12:24:18.785764 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:24:18.785718 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6e27cfc7-e129-4ed9-bb21-8dea0a55ae04-tls-certs\") pod \"e2e-lifecycle-model-07bfbcac-kserve-6b5f58f7cf-pnt8r\" (UID: \"6e27cfc7-e129-4ed9-bb21-8dea0a55ae04\") " pod="ai-tenant-e2e-mt-07bfbcac/e2e-lifecycle-model-07bfbcac-kserve-6b5f58f7cf-pnt8r" Jul 18 12:24:18.785960 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:24:18.785787 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6e27cfc7-e129-4ed9-bb21-8dea0a55ae04-tmp-dir\") pod \"e2e-lifecycle-model-07bfbcac-kserve-6b5f58f7cf-pnt8r\" (UID: \"6e27cfc7-e129-4ed9-bb21-8dea0a55ae04\") " pod="ai-tenant-e2e-mt-07bfbcac/e2e-lifecycle-model-07bfbcac-kserve-6b5f58f7cf-pnt8r" Jul 18 12:24:18.785960 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:24:18.785840 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ms2sq\" (UniqueName: \"kubernetes.io/projected/6e27cfc7-e129-4ed9-bb21-8dea0a55ae04-kube-api-access-ms2sq\") pod \"e2e-lifecycle-model-07bfbcac-kserve-6b5f58f7cf-pnt8r\" (UID: \"6e27cfc7-e129-4ed9-bb21-8dea0a55ae04\") " pod="ai-tenant-e2e-mt-07bfbcac/e2e-lifecycle-model-07bfbcac-kserve-6b5f58f7cf-pnt8r" Jul 18 12:24:18.785960 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:24:18.785892 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6e27cfc7-e129-4ed9-bb21-8dea0a55ae04-dshm\") pod \"e2e-lifecycle-model-07bfbcac-kserve-6b5f58f7cf-pnt8r\" (UID: \"6e27cfc7-e129-4ed9-bb21-8dea0a55ae04\") " pod="ai-tenant-e2e-mt-07bfbcac/e2e-lifecycle-model-07bfbcac-kserve-6b5f58f7cf-pnt8r" Jul 18 12:24:18.785960 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:24:18.785955 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6e27cfc7-e129-4ed9-bb21-8dea0a55ae04-home\") pod \"e2e-lifecycle-model-07bfbcac-kserve-6b5f58f7cf-pnt8r\" (UID: \"6e27cfc7-e129-4ed9-bb21-8dea0a55ae04\") " pod="ai-tenant-e2e-mt-07bfbcac/e2e-lifecycle-model-07bfbcac-kserve-6b5f58f7cf-pnt8r" Jul 18 12:24:18.786175 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:24:18.785993 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6e27cfc7-e129-4ed9-bb21-8dea0a55ae04-model-cache\") pod \"e2e-lifecycle-model-07bfbcac-kserve-6b5f58f7cf-pnt8r\" (UID: \"6e27cfc7-e129-4ed9-bb21-8dea0a55ae04\") " pod="ai-tenant-e2e-mt-07bfbcac/e2e-lifecycle-model-07bfbcac-kserve-6b5f58f7cf-pnt8r" Jul 18 12:24:18.787079 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:24:18.787046 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6e27cfc7-e129-4ed9-bb21-8dea0a55ae04-home\") pod \"e2e-lifecycle-model-07bfbcac-kserve-6b5f58f7cf-pnt8r\" (UID: \"6e27cfc7-e129-4ed9-bb21-8dea0a55ae04\") " pod="ai-tenant-e2e-mt-07bfbcac/e2e-lifecycle-model-07bfbcac-kserve-6b5f58f7cf-pnt8r" Jul 18 12:24:18.788354 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:24:18.787585 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6e27cfc7-e129-4ed9-bb21-8dea0a55ae04-tmp-dir\") pod \"e2e-lifecycle-model-07bfbcac-kserve-6b5f58f7cf-pnt8r\" (UID: \"6e27cfc7-e129-4ed9-bb21-8dea0a55ae04\") " pod="ai-tenant-e2e-mt-07bfbcac/e2e-lifecycle-model-07bfbcac-kserve-6b5f58f7cf-pnt8r" Jul 18 12:24:18.788354 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:24:18.787825 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6e27cfc7-e129-4ed9-bb21-8dea0a55ae04-model-cache\") pod \"e2e-lifecycle-model-07bfbcac-kserve-6b5f58f7cf-pnt8r\" (UID: \"6e27cfc7-e129-4ed9-bb21-8dea0a55ae04\") " pod="ai-tenant-e2e-mt-07bfbcac/e2e-lifecycle-model-07bfbcac-kserve-6b5f58f7cf-pnt8r" Jul 18 12:24:18.791836 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:24:18.791785 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6e27cfc7-e129-4ed9-bb21-8dea0a55ae04-dshm\") pod \"e2e-lifecycle-model-07bfbcac-kserve-6b5f58f7cf-pnt8r\" (UID: \"6e27cfc7-e129-4ed9-bb21-8dea0a55ae04\") " pod="ai-tenant-e2e-mt-07bfbcac/e2e-lifecycle-model-07bfbcac-kserve-6b5f58f7cf-pnt8r" Jul 18 12:24:18.792764 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:24:18.792709 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6e27cfc7-e129-4ed9-bb21-8dea0a55ae04-tls-certs\") pod \"e2e-lifecycle-model-07bfbcac-kserve-6b5f58f7cf-pnt8r\" (UID: \"6e27cfc7-e129-4ed9-bb21-8dea0a55ae04\") " pod="ai-tenant-e2e-mt-07bfbcac/e2e-lifecycle-model-07bfbcac-kserve-6b5f58f7cf-pnt8r" Jul 18 12:24:18.795754 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:24:18.795424 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ms2sq\" (UniqueName: \"kubernetes.io/projected/6e27cfc7-e129-4ed9-bb21-8dea0a55ae04-kube-api-access-ms2sq\") pod \"e2e-lifecycle-model-07bfbcac-kserve-6b5f58f7cf-pnt8r\" (UID: \"6e27cfc7-e129-4ed9-bb21-8dea0a55ae04\") " pod="ai-tenant-e2e-mt-07bfbcac/e2e-lifecycle-model-07bfbcac-kserve-6b5f58f7cf-pnt8r" Jul 18 12:24:18.940254 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:24:18.940168 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-07bfbcac-openshift-default-78fbf5fb95-l986p" Jul 18 12:24:18.958493 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:24:18.958461 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-07bfbcac/e2e-lifecycle-model-07bfbcac-kserve-6b5f58f7cf-pnt8r" Jul 18 12:24:19.042694 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:24:19.042023 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-07bfbcac-openshift-default-78fbf5fb95-l986p" event={"ID":"b5b5c317-e5c5-4d79-bb4e-63559c8c8268","Type":"ContainerStarted","Data":"e478bd27fd0e96b03e5bd71dca6d8c11841ca40322748a6c0fcd78189cd617bb"} Jul 18 12:24:19.148394 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:24:19.148371 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-07bfbcac/e2e-lifecycle-model-07bfbcac-kserve-6b5f58f7cf-pnt8r"] Jul 18 12:24:19.150261 ip-10-0-136-240 kubenswrapper[2585]: W0718 12:24:19.150227 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6e27cfc7_e129_4ed9_bb21_8dea0a55ae04.slice/crio-be0e2100a569ab872dbded4976b47d155fad38a1a0640ce2d291090160761d96 WatchSource:0}: Error finding container be0e2100a569ab872dbded4976b47d155fad38a1a0640ce2d291090160761d96: Status 404 returned error can't find the container with id be0e2100a569ab872dbded4976b47d155fad38a1a0640ce2d291090160761d96 Jul 18 12:24:19.176003 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:24:19.170168 2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-07bfbcac-openshift-default-78fbf5fb95-l986p" Jul 18 12:24:19.176003 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:24:19.170281 2585 patch_prober.go:28] interesting pod/e2e-mt-07bfbcac-openshift-default-78fbf5fb95-l986p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" start-of-body= Jul 18 12:24:19.176003 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:24:19.170324 2585 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-07bfbcac-openshift-default-78fbf5fb95-l986p" podUID="b5b5c317-e5c5-4d79-bb4e-63559c8c8268" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" Jul 18 12:24:20.049473 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:24:20.049439 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-07bfbcac/e2e-lifecycle-model-07bfbcac-kserve-6b5f58f7cf-pnt8r" event={"ID":"6e27cfc7-e129-4ed9-bb21-8dea0a55ae04","Type":"ContainerStarted","Data":"ca000d4fde25748d12c398a34c4380e909d856a40478d175acd58d065ee17a45"} Jul 18 12:24:20.049473 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:24:20.049485 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-07bfbcac/e2e-lifecycle-model-07bfbcac-kserve-6b5f58f7cf-pnt8r" event={"ID":"6e27cfc7-e129-4ed9-bb21-8dea0a55ae04","Type":"ContainerStarted","Data":"be0e2100a569ab872dbded4976b47d155fad38a1a0640ce2d291090160761d96"} Jul 18 12:24:20.092371 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:24:20.092318 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-mt-07bfbcac/e2e-lifecycle-model-07bfbcac-kserve-6b5f58f7cf-pnt8r" podStartSLOduration=2.09229974 podStartE2EDuration="2.09229974s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-18 12:24:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-18 12:24:20.091090757 +0000 UTC m=+2127.171297142" watchObservedRunningTime="2026-07-18 12:24:20.09229974 +0000 UTC m=+2127.172506096" Jul 18 12:24:20.169322 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:24:20.168894 2585 patch_prober.go:28] interesting pod/e2e-mt-07bfbcac-openshift-default-78fbf5fb95-l986p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" start-of-body= Jul 18 12:24:20.169322 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:24:20.168957 2585 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-07bfbcac-openshift-default-78fbf5fb95-l986p" podUID="b5b5c317-e5c5-4d79-bb4e-63559c8c8268" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" Jul 18 12:24:21.169001 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:24:21.168963 2585 patch_prober.go:28] interesting pod/e2e-mt-07bfbcac-openshift-default-78fbf5fb95-l986p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" start-of-body= Jul 18 12:24:21.169807 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:24:21.169775 2585 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-07bfbcac-openshift-default-78fbf5fb95-l986p" podUID="b5b5c317-e5c5-4d79-bb4e-63559c8c8268" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" Jul 18 12:24:22.169061 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:24:22.168601 2585 patch_prober.go:28] interesting pod/e2e-mt-07bfbcac-openshift-default-78fbf5fb95-l986p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" start-of-body= Jul 18 12:24:22.169061 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:24:22.168666 2585 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-07bfbcac-openshift-default-78fbf5fb95-l986p" podUID="b5b5c317-e5c5-4d79-bb4e-63559c8c8268" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" Jul 18 12:24:23.172363 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:24:23.172332 2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-07bfbcac-openshift-default-78fbf5fb95-l986p" Jul 18 12:24:23.172761 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:24:23.172635 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-07bfbcac-openshift-default-78fbf5fb95-l986p" Jul 18 12:24:23.173302 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:24:23.173279 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-07bfbcac-openshift-default-78fbf5fb95-l986p" Jul 18 12:24:28.959199 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:24:28.959165 2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-mt-07bfbcac/e2e-lifecycle-model-07bfbcac-kserve-6b5f58f7cf-pnt8r" Jul 18 12:24:28.959604 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:24:28.959217 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-07bfbcac/e2e-lifecycle-model-07bfbcac-kserve-6b5f58f7cf-pnt8r" Jul 18 12:24:28.971761 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:24:28.971739 2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-mt-07bfbcac/e2e-lifecycle-model-07bfbcac-kserve-6b5f58f7cf-pnt8r" Jul 18 12:24:29.106957 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:24:29.106928 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-07bfbcac/e2e-lifecycle-model-07bfbcac-kserve-6b5f58f7cf-pnt8r" Jul 18 12:25:14.839611 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:25:14.839576 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-07bfbcac/e2e-lifecycle-model-07bfbcac-kserve-6b5f58f7cf-pnt8r"] Jul 18 12:25:29.860545 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:25:29.860481 2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-mt-07bfbcac/e2e-lifecycle-model-07bfbcac-kserve-6b5f58f7cf-pnt8r" podUID="6e27cfc7-e129-4ed9-bb21-8dea0a55ae04" containerName="main" containerID="cri-o://ca000d4fde25748d12c398a34c4380e909d856a40478d175acd58d065ee17a45" gracePeriod=45 Jul 18 12:25:30.112818 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:25:30.112767 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-07bfbcac/e2e-lifecycle-model-07bfbcac-kserve-6b5f58f7cf-pnt8r" Jul 18 12:25:30.132723 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:25:30.132693 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6e27cfc7-e129-4ed9-bb21-8dea0a55ae04-home\") pod \"6e27cfc7-e129-4ed9-bb21-8dea0a55ae04\" (UID: \"6e27cfc7-e129-4ed9-bb21-8dea0a55ae04\") " Jul 18 12:25:30.132857 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:25:30.132735 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6e27cfc7-e129-4ed9-bb21-8dea0a55ae04-tls-certs\") pod \"6e27cfc7-e129-4ed9-bb21-8dea0a55ae04\" (UID: \"6e27cfc7-e129-4ed9-bb21-8dea0a55ae04\") " Jul 18 12:25:30.132857 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:25:30.132772 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6e27cfc7-e129-4ed9-bb21-8dea0a55ae04-tmp-dir\") pod \"6e27cfc7-e129-4ed9-bb21-8dea0a55ae04\" (UID: \"6e27cfc7-e129-4ed9-bb21-8dea0a55ae04\") " Jul 18 12:25:30.132857 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:25:30.132796 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6e27cfc7-e129-4ed9-bb21-8dea0a55ae04-dshm\") pod \"6e27cfc7-e129-4ed9-bb21-8dea0a55ae04\" (UID: \"6e27cfc7-e129-4ed9-bb21-8dea0a55ae04\") " Jul 18 12:25:30.132857 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:25:30.132822 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ms2sq\" (UniqueName: \"kubernetes.io/projected/6e27cfc7-e129-4ed9-bb21-8dea0a55ae04-kube-api-access-ms2sq\") pod \"6e27cfc7-e129-4ed9-bb21-8dea0a55ae04\" (UID: \"6e27cfc7-e129-4ed9-bb21-8dea0a55ae04\") " Jul 18 12:25:30.133080 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:25:30.132900 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6e27cfc7-e129-4ed9-bb21-8dea0a55ae04-model-cache\") pod \"6e27cfc7-e129-4ed9-bb21-8dea0a55ae04\" (UID: \"6e27cfc7-e129-4ed9-bb21-8dea0a55ae04\") " Jul 18 12:25:30.133080 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:25:30.132970 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6e27cfc7-e129-4ed9-bb21-8dea0a55ae04-home" (OuterVolumeSpecName: "home") pod "6e27cfc7-e129-4ed9-bb21-8dea0a55ae04" (UID: "6e27cfc7-e129-4ed9-bb21-8dea0a55ae04"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 18 12:25:30.133080 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:25:30.133048 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6e27cfc7-e129-4ed9-bb21-8dea0a55ae04-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "6e27cfc7-e129-4ed9-bb21-8dea0a55ae04" (UID: "6e27cfc7-e129-4ed9-bb21-8dea0a55ae04"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 18 12:25:30.133353 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:25:30.133295 2585 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6e27cfc7-e129-4ed9-bb21-8dea0a55ae04-home\") on node \"ip-10-0-136-240.ec2.internal\" DevicePath \"\"" Jul 18 12:25:30.133353 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:25:30.133338 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6e27cfc7-e129-4ed9-bb21-8dea0a55ae04-model-cache" (OuterVolumeSpecName: "model-cache") pod "6e27cfc7-e129-4ed9-bb21-8dea0a55ae04" (UID: "6e27cfc7-e129-4ed9-bb21-8dea0a55ae04"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 18 12:25:30.133353 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:25:30.133352 2585 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6e27cfc7-e129-4ed9-bb21-8dea0a55ae04-tmp-dir\") on node \"ip-10-0-136-240.ec2.internal\" DevicePath \"\"" Jul 18 12:25:30.135316 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:25:30.135252 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6e27cfc7-e129-4ed9-bb21-8dea0a55ae04-dshm" (OuterVolumeSpecName: "dshm") pod "6e27cfc7-e129-4ed9-bb21-8dea0a55ae04" (UID: "6e27cfc7-e129-4ed9-bb21-8dea0a55ae04"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 18 12:25:30.135569 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:25:30.135509 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6e27cfc7-e129-4ed9-bb21-8dea0a55ae04-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "6e27cfc7-e129-4ed9-bb21-8dea0a55ae04" (UID: "6e27cfc7-e129-4ed9-bb21-8dea0a55ae04"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 18 12:25:30.136873 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:25:30.136843 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6e27cfc7-e129-4ed9-bb21-8dea0a55ae04-kube-api-access-ms2sq" (OuterVolumeSpecName: "kube-api-access-ms2sq") pod "6e27cfc7-e129-4ed9-bb21-8dea0a55ae04" (UID: "6e27cfc7-e129-4ed9-bb21-8dea0a55ae04"). InnerVolumeSpecName "kube-api-access-ms2sq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 18 12:25:30.234282 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:25:30.234256 2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ms2sq\" (UniqueName: \"kubernetes.io/projected/6e27cfc7-e129-4ed9-bb21-8dea0a55ae04-kube-api-access-ms2sq\") on node \"ip-10-0-136-240.ec2.internal\" DevicePath \"\"" Jul 18 12:25:30.234282 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:25:30.234282 2585 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6e27cfc7-e129-4ed9-bb21-8dea0a55ae04-model-cache\") on node \"ip-10-0-136-240.ec2.internal\" DevicePath \"\"" Jul 18 12:25:30.234431 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:25:30.234294 2585 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6e27cfc7-e129-4ed9-bb21-8dea0a55ae04-tls-certs\") on node \"ip-10-0-136-240.ec2.internal\" DevicePath \"\"" Jul 18 12:25:30.234431 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:25:30.234302 2585 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6e27cfc7-e129-4ed9-bb21-8dea0a55ae04-dshm\") on node \"ip-10-0-136-240.ec2.internal\" DevicePath \"\"" Jul 18 12:25:30.399387 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:25:30.399285 2585 generic.go:356] "Generic (PLEG): container finished" podID="6e27cfc7-e129-4ed9-bb21-8dea0a55ae04" containerID="ca000d4fde25748d12c398a34c4380e909d856a40478d175acd58d065ee17a45" exitCode=0 Jul 18 12:25:30.399387 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:25:30.399323 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-07bfbcac/e2e-lifecycle-model-07bfbcac-kserve-6b5f58f7cf-pnt8r" Jul 18 12:25:30.399387 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:25:30.399380 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-07bfbcac/e2e-lifecycle-model-07bfbcac-kserve-6b5f58f7cf-pnt8r" event={"ID":"6e27cfc7-e129-4ed9-bb21-8dea0a55ae04","Type":"ContainerDied","Data":"ca000d4fde25748d12c398a34c4380e909d856a40478d175acd58d065ee17a45"} Jul 18 12:25:30.399668 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:25:30.399415 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-07bfbcac/e2e-lifecycle-model-07bfbcac-kserve-6b5f58f7cf-pnt8r" event={"ID":"6e27cfc7-e129-4ed9-bb21-8dea0a55ae04","Type":"ContainerDied","Data":"be0e2100a569ab872dbded4976b47d155fad38a1a0640ce2d291090160761d96"} Jul 18 12:25:30.399668 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:25:30.399438 2585 scope.go:117] "RemoveContainer" containerID="ca000d4fde25748d12c398a34c4380e909d856a40478d175acd58d065ee17a45" Jul 18 12:25:30.409426 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:25:30.409405 2585 scope.go:117] "RemoveContainer" containerID="ca000d4fde25748d12c398a34c4380e909d856a40478d175acd58d065ee17a45" Jul 18 12:25:30.409744 ip-10-0-136-240 kubenswrapper[2585]: E0718 12:25:30.409721 2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ca000d4fde25748d12c398a34c4380e909d856a40478d175acd58d065ee17a45\": container with ID starting with ca000d4fde25748d12c398a34c4380e909d856a40478d175acd58d065ee17a45 not found: ID does not exist" containerID="ca000d4fde25748d12c398a34c4380e909d856a40478d175acd58d065ee17a45" Jul 18 12:25:30.409801 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:25:30.409752 2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ca000d4fde25748d12c398a34c4380e909d856a40478d175acd58d065ee17a45"} err="failed to get container status \"ca000d4fde25748d12c398a34c4380e909d856a40478d175acd58d065ee17a45\": rpc error: code = NotFound desc = could not find container \"ca000d4fde25748d12c398a34c4380e909d856a40478d175acd58d065ee17a45\": container with ID starting with ca000d4fde25748d12c398a34c4380e909d856a40478d175acd58d065ee17a45 not found: ID does not exist" Jul 18 12:25:30.420120 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:25:30.420097 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-07bfbcac/e2e-lifecycle-model-07bfbcac-kserve-6b5f58f7cf-pnt8r"] Jul 18 12:25:30.424471 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:25:30.424444 2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-mt-07bfbcac/e2e-lifecycle-model-07bfbcac-kserve-6b5f58f7cf-pnt8r"] Jul 18 12:25:31.469658 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:25:31.469625 2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6e27cfc7-e129-4ed9-bb21-8dea0a55ae04" path="/var/lib/kubelet/pods/6e27cfc7-e129-4ed9-bb21-8dea0a55ae04/volumes" Jul 18 12:25:57.714473 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:25:57.714438 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-07bfbcac-openshift-default-78fbf5fb95-l986p"] Jul 18 12:25:57.714965 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:25:57.714674 2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-07bfbcac-openshift-default-78fbf5fb95-l986p" podUID="b5b5c317-e5c5-4d79-bb4e-63559c8c8268" containerName="istio-proxy" containerID="cri-o://e478bd27fd0e96b03e5bd71dca6d8c11841ca40322748a6c0fcd78189cd617bb" gracePeriod=30 Jul 18 12:26:02.966056 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:26:02.966032 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-07bfbcac-openshift-default-78fbf5fb95-l986p" Jul 18 12:26:03.140105 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:26:03.140007 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5b5c317-e5c5-4d79-bb4e-63559c8c8268-workload-socket\") pod \"b5b5c317-e5c5-4d79-bb4e-63559c8c8268\" (UID: \"b5b5c317-e5c5-4d79-bb4e-63559c8c8268\") " Jul 18 12:26:03.140105 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:26:03.140067 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b5b5c317-e5c5-4d79-bb4e-63559c8c8268-istio-podinfo\") pod \"b5b5c317-e5c5-4d79-bb4e-63559c8c8268\" (UID: \"b5b5c317-e5c5-4d79-bb4e-63559c8c8268\") " Jul 18 12:26:03.140105 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:26:03.140095 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zp6pv\" (UniqueName: \"kubernetes.io/projected/b5b5c317-e5c5-4d79-bb4e-63559c8c8268-kube-api-access-zp6pv\") pod \"b5b5c317-e5c5-4d79-bb4e-63559c8c8268\" (UID: \"b5b5c317-e5c5-4d79-bb4e-63559c8c8268\") " Jul 18 12:26:03.140382 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:26:03.140140 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b5b5c317-e5c5-4d79-bb4e-63559c8c8268-istio-envoy\") pod \"b5b5c317-e5c5-4d79-bb4e-63559c8c8268\" (UID: \"b5b5c317-e5c5-4d79-bb4e-63559c8c8268\") " Jul 18 12:26:03.140382 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:26:03.140197 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b5b5c317-e5c5-4d79-bb4e-63559c8c8268-workload-certs\") pod \"b5b5c317-e5c5-4d79-bb4e-63559c8c8268\" (UID: \"b5b5c317-e5c5-4d79-bb4e-63559c8c8268\") " Jul 18 12:26:03.140382 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:26:03.140255 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b5b5c317-e5c5-4d79-bb4e-63559c8c8268-istio-token\") pod \"b5b5c317-e5c5-4d79-bb4e-63559c8c8268\" (UID: \"b5b5c317-e5c5-4d79-bb4e-63559c8c8268\") " Jul 18 12:26:03.140382 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:26:03.140302 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b5b5c317-e5c5-4d79-bb4e-63559c8c8268-istio-data\") pod \"b5b5c317-e5c5-4d79-bb4e-63559c8c8268\" (UID: \"b5b5c317-e5c5-4d79-bb4e-63559c8c8268\") " Jul 18 12:26:03.140382 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:26:03.140314 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b5b5c317-e5c5-4d79-bb4e-63559c8c8268-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "b5b5c317-e5c5-4d79-bb4e-63559c8c8268" (UID: "b5b5c317-e5c5-4d79-bb4e-63559c8c8268"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 18 12:26:03.140382 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:26:03.140329 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b5b5c317-e5c5-4d79-bb4e-63559c8c8268-istiod-ca-cert\") pod \"b5b5c317-e5c5-4d79-bb4e-63559c8c8268\" (UID: \"b5b5c317-e5c5-4d79-bb4e-63559c8c8268\") " Jul 18 12:26:03.140752 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:26:03.140386 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5b5c317-e5c5-4d79-bb4e-63559c8c8268-credential-socket\") pod \"b5b5c317-e5c5-4d79-bb4e-63559c8c8268\" (UID: \"b5b5c317-e5c5-4d79-bb4e-63559c8c8268\") " Jul 18 12:26:03.140752 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:26:03.140635 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b5b5c317-e5c5-4d79-bb4e-63559c8c8268-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "b5b5c317-e5c5-4d79-bb4e-63559c8c8268" (UID: "b5b5c317-e5c5-4d79-bb4e-63559c8c8268"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 18 12:26:03.140752 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:26:03.140739 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b5b5c317-e5c5-4d79-bb4e-63559c8c8268-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "b5b5c317-e5c5-4d79-bb4e-63559c8c8268" (UID: "b5b5c317-e5c5-4d79-bb4e-63559c8c8268"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 18 12:26:03.140892 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:26:03.140822 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b5b5c317-e5c5-4d79-bb4e-63559c8c8268-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "b5b5c317-e5c5-4d79-bb4e-63559c8c8268" (UID: "b5b5c317-e5c5-4d79-bb4e-63559c8c8268"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 18 12:26:03.141191 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:26:03.141172 2585 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5b5c317-e5c5-4d79-bb4e-63559c8c8268-workload-socket\") on node \"ip-10-0-136-240.ec2.internal\" DevicePath \"\"" Jul 18 12:26:03.141236 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:26:03.141197 2585 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b5b5c317-e5c5-4d79-bb4e-63559c8c8268-workload-certs\") on node \"ip-10-0-136-240.ec2.internal\" DevicePath \"\"" Jul 18 12:26:03.141236 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:26:03.141212 2585 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b5b5c317-e5c5-4d79-bb4e-63559c8c8268-istiod-ca-cert\") on node \"ip-10-0-136-240.ec2.internal\" DevicePath \"\"" Jul 18 12:26:03.141236 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:26:03.141224 2585 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5b5c317-e5c5-4d79-bb4e-63559c8c8268-credential-socket\") on node \"ip-10-0-136-240.ec2.internal\" DevicePath \"\"" Jul 18 12:26:03.141582 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:26:03.141490 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b5b5c317-e5c5-4d79-bb4e-63559c8c8268-istio-data" (OuterVolumeSpecName: "istio-data") pod "b5b5c317-e5c5-4d79-bb4e-63559c8c8268" (UID: "b5b5c317-e5c5-4d79-bb4e-63559c8c8268"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 18 12:26:03.142686 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:26:03.142661 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b5b5c317-e5c5-4d79-bb4e-63559c8c8268-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "b5b5c317-e5c5-4d79-bb4e-63559c8c8268" (UID: "b5b5c317-e5c5-4d79-bb4e-63559c8c8268"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 18 12:26:03.142686 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:26:03.142677 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b5b5c317-e5c5-4d79-bb4e-63559c8c8268-kube-api-access-zp6pv" (OuterVolumeSpecName: "kube-api-access-zp6pv") pod "b5b5c317-e5c5-4d79-bb4e-63559c8c8268" (UID: "b5b5c317-e5c5-4d79-bb4e-63559c8c8268"). InnerVolumeSpecName "kube-api-access-zp6pv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 18 12:26:03.142855 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:26:03.142776 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b5b5c317-e5c5-4d79-bb4e-63559c8c8268-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "b5b5c317-e5c5-4d79-bb4e-63559c8c8268" (UID: "b5b5c317-e5c5-4d79-bb4e-63559c8c8268"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 18 12:26:03.143033 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:26:03.143015 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b5b5c317-e5c5-4d79-bb4e-63559c8c8268-istio-token" (OuterVolumeSpecName: "istio-token") pod "b5b5c317-e5c5-4d79-bb4e-63559c8c8268" (UID: "b5b5c317-e5c5-4d79-bb4e-63559c8c8268"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 18 12:26:03.242360 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:26:03.242331 2585 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b5b5c317-e5c5-4d79-bb4e-63559c8c8268-istio-token\") on node \"ip-10-0-136-240.ec2.internal\" DevicePath \"\"" Jul 18 12:26:03.242360 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:26:03.242356 2585 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b5b5c317-e5c5-4d79-bb4e-63559c8c8268-istio-data\") on node \"ip-10-0-136-240.ec2.internal\" DevicePath \"\"" Jul 18 12:26:03.242360 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:26:03.242365 2585 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b5b5c317-e5c5-4d79-bb4e-63559c8c8268-istio-podinfo\") on node \"ip-10-0-136-240.ec2.internal\" DevicePath \"\"" Jul 18 12:26:03.242618 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:26:03.242374 2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zp6pv\" (UniqueName: \"kubernetes.io/projected/b5b5c317-e5c5-4d79-bb4e-63559c8c8268-kube-api-access-zp6pv\") on node \"ip-10-0-136-240.ec2.internal\" DevicePath \"\"" Jul 18 12:26:03.242618 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:26:03.242383 2585 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b5b5c317-e5c5-4d79-bb4e-63559c8c8268-istio-envoy\") on node \"ip-10-0-136-240.ec2.internal\" DevicePath \"\"" Jul 18 12:26:03.573068 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:26:03.573032 2585 generic.go:356] "Generic (PLEG): container finished" podID="b5b5c317-e5c5-4d79-bb4e-63559c8c8268" containerID="e478bd27fd0e96b03e5bd71dca6d8c11841ca40322748a6c0fcd78189cd617bb" exitCode=0 Jul 18 12:26:03.573246 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:26:03.573101 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-07bfbcac-openshift-default-78fbf5fb95-l986p" Jul 18 12:26:03.573246 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:26:03.573106 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-07bfbcac-openshift-default-78fbf5fb95-l986p" event={"ID":"b5b5c317-e5c5-4d79-bb4e-63559c8c8268","Type":"ContainerDied","Data":"e478bd27fd0e96b03e5bd71dca6d8c11841ca40322748a6c0fcd78189cd617bb"} Jul 18 12:26:03.573246 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:26:03.573160 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-07bfbcac-openshift-default-78fbf5fb95-l986p" event={"ID":"b5b5c317-e5c5-4d79-bb4e-63559c8c8268","Type":"ContainerDied","Data":"5d98845a07e181d1a1063a88790a201968950e7f240897fd41c26d4eec6e8892"} Jul 18 12:26:03.573246 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:26:03.573190 2585 scope.go:117] "RemoveContainer" containerID="e478bd27fd0e96b03e5bd71dca6d8c11841ca40322748a6c0fcd78189cd617bb" Jul 18 12:26:03.581969 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:26:03.581892 2585 scope.go:117] "RemoveContainer" containerID="57f6c2183c527bde730527c826752f7d4975ed7717e08d29f73651d833da3a64" Jul 18 12:26:03.592078 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:26:03.592053 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-07bfbcac-openshift-default-78fbf5fb95-l986p"] Jul 18 12:26:03.593872 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:26:03.593755 2585 scope.go:117] "RemoveContainer" containerID="e478bd27fd0e96b03e5bd71dca6d8c11841ca40322748a6c0fcd78189cd617bb" Jul 18 12:26:03.594202 ip-10-0-136-240 kubenswrapper[2585]: E0718 12:26:03.594178 2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e478bd27fd0e96b03e5bd71dca6d8c11841ca40322748a6c0fcd78189cd617bb\": container with ID starting with e478bd27fd0e96b03e5bd71dca6d8c11841ca40322748a6c0fcd78189cd617bb not found: ID does not exist" containerID="e478bd27fd0e96b03e5bd71dca6d8c11841ca40322748a6c0fcd78189cd617bb" Jul 18 12:26:03.594269 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:26:03.594211 2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e478bd27fd0e96b03e5bd71dca6d8c11841ca40322748a6c0fcd78189cd617bb"} err="failed to get container status \"e478bd27fd0e96b03e5bd71dca6d8c11841ca40322748a6c0fcd78189cd617bb\": rpc error: code = NotFound desc = could not find container \"e478bd27fd0e96b03e5bd71dca6d8c11841ca40322748a6c0fcd78189cd617bb\": container with ID starting with e478bd27fd0e96b03e5bd71dca6d8c11841ca40322748a6c0fcd78189cd617bb not found: ID does not exist" Jul 18 12:26:03.594269 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:26:03.594231 2585 scope.go:117] "RemoveContainer" containerID="57f6c2183c527bde730527c826752f7d4975ed7717e08d29f73651d833da3a64" Jul 18 12:26:03.594509 ip-10-0-136-240 kubenswrapper[2585]: E0718 12:26:03.594489 2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"57f6c2183c527bde730527c826752f7d4975ed7717e08d29f73651d833da3a64\": container with ID starting with 57f6c2183c527bde730527c826752f7d4975ed7717e08d29f73651d833da3a64 not found: ID does not exist" containerID="57f6c2183c527bde730527c826752f7d4975ed7717e08d29f73651d833da3a64" Jul 18 12:26:03.594617 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:26:03.594517 2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"57f6c2183c527bde730527c826752f7d4975ed7717e08d29f73651d833da3a64"} err="failed to get container status \"57f6c2183c527bde730527c826752f7d4975ed7717e08d29f73651d833da3a64\": rpc error: code = NotFound desc = could not find container \"57f6c2183c527bde730527c826752f7d4975ed7717e08d29f73651d833da3a64\": container with ID starting with 57f6c2183c527bde730527c826752f7d4975ed7717e08d29f73651d833da3a64 not found: ID does not exist" Jul 18 12:26:03.595705 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:26:03.595687 2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-07bfbcac-openshift-default-78fbf5fb95-l986p"] Jul 18 12:26:05.468609 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:26:05.468539 2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b5b5c317-e5c5-4d79-bb4e-63559c8c8268" path="/var/lib/kubelet/pods/b5b5c317-e5c5-4d79-bb4e-63559c8c8268/volumes" Jul 18 12:26:52.390000 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:26:52.389966 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-3be5a3-openshift-default-76dcf5b787-89wjz"] Jul 18 12:26:52.390447 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:26:52.390339 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6e27cfc7-e129-4ed9-bb21-8dea0a55ae04" containerName="main" Jul 18 12:26:52.390447 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:26:52.390349 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="6e27cfc7-e129-4ed9-bb21-8dea0a55ae04" containerName="main" Jul 18 12:26:52.390447 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:26:52.390371 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b5b5c317-e5c5-4d79-bb4e-63559c8c8268" containerName="istio-proxy" Jul 18 12:26:52.390447 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:26:52.390377 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="b5b5c317-e5c5-4d79-bb4e-63559c8c8268" containerName="istio-proxy" Jul 18 12:26:52.390447 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:26:52.390385 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b5b5c317-e5c5-4d79-bb4e-63559c8c8268" containerName="istio-proxy" Jul 18 12:26:52.390447 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:26:52.390391 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="b5b5c317-e5c5-4d79-bb4e-63559c8c8268" containerName="istio-proxy" Jul 18 12:26:52.390447 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:26:52.390446 2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="b5b5c317-e5c5-4d79-bb4e-63559c8c8268" containerName="istio-proxy" Jul 18 12:26:52.390705 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:26:52.390455 2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="6e27cfc7-e129-4ed9-bb21-8dea0a55ae04" containerName="main" Jul 18 12:26:52.390705 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:26:52.390461 2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="b5b5c317-e5c5-4d79-bb4e-63559c8c8268" containerName="istio-proxy" Jul 18 12:26:52.393544 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:26:52.393527 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-3be5a3-openshift-default-76dcf5b787-89wjz" Jul 18 12:26:52.396062 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:26:52.396041 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-3be5a3-openshift-default-dockercfg-b7zgp\"" Jul 18 12:26:52.407162 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:26:52.407139 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-3be5a3-openshift-default-76dcf5b787-89wjz"] Jul 18 12:26:52.571746 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:26:52.571709 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f2a380d1-804a-4e1c-9853-2773e962a49c-istio-token\") pod \"e2e-inf-a-3be5a3-openshift-default-76dcf5b787-89wjz\" (UID: \"f2a380d1-804a-4e1c-9853-2773e962a49c\") " pod="openshift-ingress/e2e-inf-a-3be5a3-openshift-default-76dcf5b787-89wjz" Jul 18 12:26:52.571919 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:26:52.571752 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f2a380d1-804a-4e1c-9853-2773e962a49c-istio-envoy\") pod \"e2e-inf-a-3be5a3-openshift-default-76dcf5b787-89wjz\" (UID: \"f2a380d1-804a-4e1c-9853-2773e962a49c\") " pod="openshift-ingress/e2e-inf-a-3be5a3-openshift-default-76dcf5b787-89wjz" Jul 18 12:26:52.571919 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:26:52.571831 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f2a380d1-804a-4e1c-9853-2773e962a49c-istio-data\") pod \"e2e-inf-a-3be5a3-openshift-default-76dcf5b787-89wjz\" (UID: \"f2a380d1-804a-4e1c-9853-2773e962a49c\") " pod="openshift-ingress/e2e-inf-a-3be5a3-openshift-default-76dcf5b787-89wjz" Jul 18 12:26:52.572049 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:26:52.571941 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7xm6h\" (UniqueName: \"kubernetes.io/projected/f2a380d1-804a-4e1c-9853-2773e962a49c-kube-api-access-7xm6h\") pod \"e2e-inf-a-3be5a3-openshift-default-76dcf5b787-89wjz\" (UID: \"f2a380d1-804a-4e1c-9853-2773e962a49c\") " pod="openshift-ingress/e2e-inf-a-3be5a3-openshift-default-76dcf5b787-89wjz" Jul 18 12:26:52.572152 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:26:52.572131 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f2a380d1-804a-4e1c-9853-2773e962a49c-istiod-ca-cert\") pod \"e2e-inf-a-3be5a3-openshift-default-76dcf5b787-89wjz\" (UID: \"f2a380d1-804a-4e1c-9853-2773e962a49c\") " pod="openshift-ingress/e2e-inf-a-3be5a3-openshift-default-76dcf5b787-89wjz" Jul 18 12:26:52.572209 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:26:52.572168 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2a380d1-804a-4e1c-9853-2773e962a49c-credential-socket\") pod \"e2e-inf-a-3be5a3-openshift-default-76dcf5b787-89wjz\" (UID: \"f2a380d1-804a-4e1c-9853-2773e962a49c\") " pod="openshift-ingress/e2e-inf-a-3be5a3-openshift-default-76dcf5b787-89wjz" Jul 18 12:26:52.572266 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:26:52.572221 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f2a380d1-804a-4e1c-9853-2773e962a49c-workload-certs\") pod \"e2e-inf-a-3be5a3-openshift-default-76dcf5b787-89wjz\" (UID: \"f2a380d1-804a-4e1c-9853-2773e962a49c\") " pod="openshift-ingress/e2e-inf-a-3be5a3-openshift-default-76dcf5b787-89wjz" Jul 18 12:26:52.572457 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:26:52.572434 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2a380d1-804a-4e1c-9853-2773e962a49c-workload-socket\") pod \"e2e-inf-a-3be5a3-openshift-default-76dcf5b787-89wjz\" (UID: \"f2a380d1-804a-4e1c-9853-2773e962a49c\") " pod="openshift-ingress/e2e-inf-a-3be5a3-openshift-default-76dcf5b787-89wjz" Jul 18 12:26:52.572617 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:26:52.572525 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f2a380d1-804a-4e1c-9853-2773e962a49c-istio-podinfo\") pod \"e2e-inf-a-3be5a3-openshift-default-76dcf5b787-89wjz\" (UID: \"f2a380d1-804a-4e1c-9853-2773e962a49c\") " pod="openshift-ingress/e2e-inf-a-3be5a3-openshift-default-76dcf5b787-89wjz" Jul 18 12:26:52.673608 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:26:52.673525 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2a380d1-804a-4e1c-9853-2773e962a49c-workload-socket\") pod \"e2e-inf-a-3be5a3-openshift-default-76dcf5b787-89wjz\" (UID: \"f2a380d1-804a-4e1c-9853-2773e962a49c\") " pod="openshift-ingress/e2e-inf-a-3be5a3-openshift-default-76dcf5b787-89wjz" Jul 18 12:26:52.673608 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:26:52.673586 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f2a380d1-804a-4e1c-9853-2773e962a49c-istio-podinfo\") pod \"e2e-inf-a-3be5a3-openshift-default-76dcf5b787-89wjz\" (UID: \"f2a380d1-804a-4e1c-9853-2773e962a49c\") " pod="openshift-ingress/e2e-inf-a-3be5a3-openshift-default-76dcf5b787-89wjz" Jul 18 12:26:52.673819 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:26:52.673632 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f2a380d1-804a-4e1c-9853-2773e962a49c-istio-token\") pod \"e2e-inf-a-3be5a3-openshift-default-76dcf5b787-89wjz\" (UID: \"f2a380d1-804a-4e1c-9853-2773e962a49c\") " pod="openshift-ingress/e2e-inf-a-3be5a3-openshift-default-76dcf5b787-89wjz" Jul 18 12:26:52.673819 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:26:52.673649 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f2a380d1-804a-4e1c-9853-2773e962a49c-istio-envoy\") pod \"e2e-inf-a-3be5a3-openshift-default-76dcf5b787-89wjz\" (UID: \"f2a380d1-804a-4e1c-9853-2773e962a49c\") " pod="openshift-ingress/e2e-inf-a-3be5a3-openshift-default-76dcf5b787-89wjz" Jul 18 12:26:52.673819 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:26:52.673676 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f2a380d1-804a-4e1c-9853-2773e962a49c-istio-data\") pod \"e2e-inf-a-3be5a3-openshift-default-76dcf5b787-89wjz\" (UID: \"f2a380d1-804a-4e1c-9853-2773e962a49c\") " pod="openshift-ingress/e2e-inf-a-3be5a3-openshift-default-76dcf5b787-89wjz" Jul 18 12:26:52.673819 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:26:52.673727 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7xm6h\" (UniqueName: \"kubernetes.io/projected/f2a380d1-804a-4e1c-9853-2773e962a49c-kube-api-access-7xm6h\") pod \"e2e-inf-a-3be5a3-openshift-default-76dcf5b787-89wjz\" (UID: \"f2a380d1-804a-4e1c-9853-2773e962a49c\") " pod="openshift-ingress/e2e-inf-a-3be5a3-openshift-default-76dcf5b787-89wjz" Jul 18 12:26:52.673819 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:26:52.673768 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f2a380d1-804a-4e1c-9853-2773e962a49c-istiod-ca-cert\") pod \"e2e-inf-a-3be5a3-openshift-default-76dcf5b787-89wjz\" (UID: \"f2a380d1-804a-4e1c-9853-2773e962a49c\") " pod="openshift-ingress/e2e-inf-a-3be5a3-openshift-default-76dcf5b787-89wjz" Jul 18 12:26:52.673819 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:26:52.673792 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2a380d1-804a-4e1c-9853-2773e962a49c-credential-socket\") pod \"e2e-inf-a-3be5a3-openshift-default-76dcf5b787-89wjz\" (UID: \"f2a380d1-804a-4e1c-9853-2773e962a49c\") " pod="openshift-ingress/e2e-inf-a-3be5a3-openshift-default-76dcf5b787-89wjz" Jul 18 12:26:52.673819 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:26:52.673817 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f2a380d1-804a-4e1c-9853-2773e962a49c-workload-certs\") pod \"e2e-inf-a-3be5a3-openshift-default-76dcf5b787-89wjz\" (UID: \"f2a380d1-804a-4e1c-9853-2773e962a49c\") " pod="openshift-ingress/e2e-inf-a-3be5a3-openshift-default-76dcf5b787-89wjz" Jul 18 12:26:52.674242 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:26:52.673946 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2a380d1-804a-4e1c-9853-2773e962a49c-workload-socket\") pod \"e2e-inf-a-3be5a3-openshift-default-76dcf5b787-89wjz\" (UID: \"f2a380d1-804a-4e1c-9853-2773e962a49c\") " pod="openshift-ingress/e2e-inf-a-3be5a3-openshift-default-76dcf5b787-89wjz" Jul 18 12:26:52.674242 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:26:52.674146 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f2a380d1-804a-4e1c-9853-2773e962a49c-istio-data\") pod \"e2e-inf-a-3be5a3-openshift-default-76dcf5b787-89wjz\" (UID: \"f2a380d1-804a-4e1c-9853-2773e962a49c\") " pod="openshift-ingress/e2e-inf-a-3be5a3-openshift-default-76dcf5b787-89wjz" Jul 18 12:26:52.674506 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:26:52.674247 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f2a380d1-804a-4e1c-9853-2773e962a49c-workload-certs\") pod \"e2e-inf-a-3be5a3-openshift-default-76dcf5b787-89wjz\" (UID: \"f2a380d1-804a-4e1c-9853-2773e962a49c\") " pod="openshift-ingress/e2e-inf-a-3be5a3-openshift-default-76dcf5b787-89wjz" Jul 18 12:26:52.674506 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:26:52.674278 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2a380d1-804a-4e1c-9853-2773e962a49c-credential-socket\") pod \"e2e-inf-a-3be5a3-openshift-default-76dcf5b787-89wjz\" (UID: \"f2a380d1-804a-4e1c-9853-2773e962a49c\") " pod="openshift-ingress/e2e-inf-a-3be5a3-openshift-default-76dcf5b787-89wjz" Jul 18 12:26:52.674506 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:26:52.674478 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f2a380d1-804a-4e1c-9853-2773e962a49c-istiod-ca-cert\") pod \"e2e-inf-a-3be5a3-openshift-default-76dcf5b787-89wjz\" (UID: \"f2a380d1-804a-4e1c-9853-2773e962a49c\") " pod="openshift-ingress/e2e-inf-a-3be5a3-openshift-default-76dcf5b787-89wjz" Jul 18 12:26:52.676023 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:26:52.676002 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f2a380d1-804a-4e1c-9853-2773e962a49c-istio-envoy\") pod \"e2e-inf-a-3be5a3-openshift-default-76dcf5b787-89wjz\" (UID: \"f2a380d1-804a-4e1c-9853-2773e962a49c\") " pod="openshift-ingress/e2e-inf-a-3be5a3-openshift-default-76dcf5b787-89wjz" Jul 18 12:26:52.676359 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:26:52.676338 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f2a380d1-804a-4e1c-9853-2773e962a49c-istio-podinfo\") pod \"e2e-inf-a-3be5a3-openshift-default-76dcf5b787-89wjz\" (UID: \"f2a380d1-804a-4e1c-9853-2773e962a49c\") " pod="openshift-ingress/e2e-inf-a-3be5a3-openshift-default-76dcf5b787-89wjz" Jul 18 12:26:52.682648 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:26:52.682629 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f2a380d1-804a-4e1c-9853-2773e962a49c-istio-token\") pod \"e2e-inf-a-3be5a3-openshift-default-76dcf5b787-89wjz\" (UID: \"f2a380d1-804a-4e1c-9853-2773e962a49c\") " pod="openshift-ingress/e2e-inf-a-3be5a3-openshift-default-76dcf5b787-89wjz" Jul 18 12:26:52.684959 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:26:52.684936 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7xm6h\" (UniqueName: \"kubernetes.io/projected/f2a380d1-804a-4e1c-9853-2773e962a49c-kube-api-access-7xm6h\") pod \"e2e-inf-a-3be5a3-openshift-default-76dcf5b787-89wjz\" (UID: \"f2a380d1-804a-4e1c-9853-2773e962a49c\") " pod="openshift-ingress/e2e-inf-a-3be5a3-openshift-default-76dcf5b787-89wjz" Jul 18 12:26:52.704714 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:26:52.704689 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-3be5a3-openshift-default-76dcf5b787-89wjz" Jul 18 12:26:52.843674 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:26:52.843650 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-3be5a3-openshift-default-76dcf5b787-89wjz"] Jul 18 12:26:52.845284 ip-10-0-136-240 kubenswrapper[2585]: W0718 12:26:52.845254 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf2a380d1_804a_4e1c_9853_2773e962a49c.slice/crio-dbf3c0573dd3cc29154861f05a461185cfdab8abf4d7a2342d88e5657e4ba2cb WatchSource:0}: Error finding container dbf3c0573dd3cc29154861f05a461185cfdab8abf4d7a2342d88e5657e4ba2cb: Status 404 returned error can't find the container with id dbf3c0573dd3cc29154861f05a461185cfdab8abf4d7a2342d88e5657e4ba2cb Jul 18 12:26:52.847591 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:26:52.847532 2585 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 18 12:26:52.847761 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:26:52.847640 2585 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 18 12:26:52.847761 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:26:52.847687 2585 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 18 12:26:53.824101 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:26:53.824060 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-3be5a3-openshift-default-76dcf5b787-89wjz" event={"ID":"f2a380d1-804a-4e1c-9853-2773e962a49c","Type":"ContainerStarted","Data":"0b1e75fa281636e01674b4f960d5db2a9a34cec889b72a83aaceac8870b5b365"} Jul 18 12:26:53.824486 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:26:53.824142 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-3be5a3-openshift-default-76dcf5b787-89wjz" event={"ID":"f2a380d1-804a-4e1c-9853-2773e962a49c","Type":"ContainerStarted","Data":"dbf3c0573dd3cc29154861f05a461185cfdab8abf4d7a2342d88e5657e4ba2cb"} Jul 18 12:26:54.705469 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:26:54.705437 2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-3be5a3-openshift-default-76dcf5b787-89wjz" Jul 18 12:26:54.710500 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:26:54.710472 2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-3be5a3-openshift-default-76dcf5b787-89wjz" Jul 18 12:26:54.733523 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:26:54.733477 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-3be5a3-openshift-default-76dcf5b787-89wjz" podStartSLOduration=2.7334603790000003 podStartE2EDuration="2.733460379s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-18 12:26:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-18 12:26:53.843493655 +0000 UTC m=+2280.923700013" watchObservedRunningTime="2026-07-18 12:26:54.733460379 +0000 UTC m=+2281.813666735" Jul 18 12:26:54.829008 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:26:54.828982 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-3be5a3-openshift-default-76dcf5b787-89wjz" Jul 18 12:26:54.830026 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:26:54.830007 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-3be5a3-openshift-default-76dcf5b787-89wjz" Jul 18 12:27:00.836647 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:27:00.836610 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-f133b3-openshift-default-7c787bdc9f-ccdbc"] Jul 18 12:27:00.840462 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:27:00.840440 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-f133b3-openshift-default-7c787bdc9f-ccdbc" Jul 18 12:27:00.842697 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:27:00.842678 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-f133b3-openshift-default-dockercfg-cgntx\"" Jul 18 12:27:00.851672 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:27:00.851650 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-f133b3-openshift-default-7c787bdc9f-ccdbc"] Jul 18 12:27:00.852164 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:27:00.852140 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m6hdg\" (UniqueName: \"kubernetes.io/projected/2eede87b-bd0d-4e01-962b-583b7c7522cd-kube-api-access-m6hdg\") pod \"e2e-inf-b-f133b3-openshift-default-7c787bdc9f-ccdbc\" (UID: \"2eede87b-bd0d-4e01-962b-583b7c7522cd\") " pod="openshift-ingress/e2e-inf-b-f133b3-openshift-default-7c787bdc9f-ccdbc" Jul 18 12:27:00.852254 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:27:00.852212 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2eede87b-bd0d-4e01-962b-583b7c7522cd-workload-socket\") pod \"e2e-inf-b-f133b3-openshift-default-7c787bdc9f-ccdbc\" (UID: \"2eede87b-bd0d-4e01-962b-583b7c7522cd\") " pod="openshift-ingress/e2e-inf-b-f133b3-openshift-default-7c787bdc9f-ccdbc" Jul 18 12:27:00.852323 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:27:00.852256 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2eede87b-bd0d-4e01-962b-583b7c7522cd-istio-token\") pod \"e2e-inf-b-f133b3-openshift-default-7c787bdc9f-ccdbc\" (UID: \"2eede87b-bd0d-4e01-962b-583b7c7522cd\") " pod="openshift-ingress/e2e-inf-b-f133b3-openshift-default-7c787bdc9f-ccdbc" Jul 18 12:27:00.852323 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:27:00.852298 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2eede87b-bd0d-4e01-962b-583b7c7522cd-istiod-ca-cert\") pod \"e2e-inf-b-f133b3-openshift-default-7c787bdc9f-ccdbc\" (UID: \"2eede87b-bd0d-4e01-962b-583b7c7522cd\") " pod="openshift-ingress/e2e-inf-b-f133b3-openshift-default-7c787bdc9f-ccdbc" Jul 18 12:27:00.852432 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:27:00.852334 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2eede87b-bd0d-4e01-962b-583b7c7522cd-istio-podinfo\") pod \"e2e-inf-b-f133b3-openshift-default-7c787bdc9f-ccdbc\" (UID: \"2eede87b-bd0d-4e01-962b-583b7c7522cd\") " pod="openshift-ingress/e2e-inf-b-f133b3-openshift-default-7c787bdc9f-ccdbc" Jul 18 12:27:00.852432 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:27:00.852361 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2eede87b-bd0d-4e01-962b-583b7c7522cd-workload-certs\") pod \"e2e-inf-b-f133b3-openshift-default-7c787bdc9f-ccdbc\" (UID: \"2eede87b-bd0d-4e01-962b-583b7c7522cd\") " pod="openshift-ingress/e2e-inf-b-f133b3-openshift-default-7c787bdc9f-ccdbc" Jul 18 12:27:00.852432 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:27:00.852383 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2eede87b-bd0d-4e01-962b-583b7c7522cd-istio-data\") pod \"e2e-inf-b-f133b3-openshift-default-7c787bdc9f-ccdbc\" (UID: \"2eede87b-bd0d-4e01-962b-583b7c7522cd\") " pod="openshift-ingress/e2e-inf-b-f133b3-openshift-default-7c787bdc9f-ccdbc" Jul 18 12:27:00.852432 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:27:00.852409 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2eede87b-bd0d-4e01-962b-583b7c7522cd-istio-envoy\") pod \"e2e-inf-b-f133b3-openshift-default-7c787bdc9f-ccdbc\" (UID: \"2eede87b-bd0d-4e01-962b-583b7c7522cd\") " pod="openshift-ingress/e2e-inf-b-f133b3-openshift-default-7c787bdc9f-ccdbc" Jul 18 12:27:00.852617 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:27:00.852601 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2eede87b-bd0d-4e01-962b-583b7c7522cd-credential-socket\") pod \"e2e-inf-b-f133b3-openshift-default-7c787bdc9f-ccdbc\" (UID: \"2eede87b-bd0d-4e01-962b-583b7c7522cd\") " pod="openshift-ingress/e2e-inf-b-f133b3-openshift-default-7c787bdc9f-ccdbc" Jul 18 12:27:00.953237 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:27:00.953204 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2eede87b-bd0d-4e01-962b-583b7c7522cd-workload-socket\") pod \"e2e-inf-b-f133b3-openshift-default-7c787bdc9f-ccdbc\" (UID: \"2eede87b-bd0d-4e01-962b-583b7c7522cd\") " pod="openshift-ingress/e2e-inf-b-f133b3-openshift-default-7c787bdc9f-ccdbc" Jul 18 12:27:00.953388 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:27:00.953245 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2eede87b-bd0d-4e01-962b-583b7c7522cd-istio-token\") pod \"e2e-inf-b-f133b3-openshift-default-7c787bdc9f-ccdbc\" (UID: \"2eede87b-bd0d-4e01-962b-583b7c7522cd\") " pod="openshift-ingress/e2e-inf-b-f133b3-openshift-default-7c787bdc9f-ccdbc" Jul 18 12:27:00.953388 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:27:00.953279 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2eede87b-bd0d-4e01-962b-583b7c7522cd-istiod-ca-cert\") pod \"e2e-inf-b-f133b3-openshift-default-7c787bdc9f-ccdbc\" (UID: \"2eede87b-bd0d-4e01-962b-583b7c7522cd\") " pod="openshift-ingress/e2e-inf-b-f133b3-openshift-default-7c787bdc9f-ccdbc" Jul 18 12:27:00.953388 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:27:00.953314 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2eede87b-bd0d-4e01-962b-583b7c7522cd-istio-podinfo\") pod \"e2e-inf-b-f133b3-openshift-default-7c787bdc9f-ccdbc\" (UID: \"2eede87b-bd0d-4e01-962b-583b7c7522cd\") " pod="openshift-ingress/e2e-inf-b-f133b3-openshift-default-7c787bdc9f-ccdbc" Jul 18 12:27:00.953388 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:27:00.953331 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2eede87b-bd0d-4e01-962b-583b7c7522cd-workload-certs\") pod \"e2e-inf-b-f133b3-openshift-default-7c787bdc9f-ccdbc\" (UID: \"2eede87b-bd0d-4e01-962b-583b7c7522cd\") " pod="openshift-ingress/e2e-inf-b-f133b3-openshift-default-7c787bdc9f-ccdbc" Jul 18 12:27:00.953388 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:27:00.953355 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2eede87b-bd0d-4e01-962b-583b7c7522cd-istio-data\") pod \"e2e-inf-b-f133b3-openshift-default-7c787bdc9f-ccdbc\" (UID: \"2eede87b-bd0d-4e01-962b-583b7c7522cd\") " pod="openshift-ingress/e2e-inf-b-f133b3-openshift-default-7c787bdc9f-ccdbc" Jul 18 12:27:00.953388 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:27:00.953379 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2eede87b-bd0d-4e01-962b-583b7c7522cd-istio-envoy\") pod \"e2e-inf-b-f133b3-openshift-default-7c787bdc9f-ccdbc\" (UID: \"2eede87b-bd0d-4e01-962b-583b7c7522cd\") " pod="openshift-ingress/e2e-inf-b-f133b3-openshift-default-7c787bdc9f-ccdbc" Jul 18 12:27:00.953758 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:27:00.953496 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2eede87b-bd0d-4e01-962b-583b7c7522cd-credential-socket\") pod \"e2e-inf-b-f133b3-openshift-default-7c787bdc9f-ccdbc\" (UID: \"2eede87b-bd0d-4e01-962b-583b7c7522cd\") " pod="openshift-ingress/e2e-inf-b-f133b3-openshift-default-7c787bdc9f-ccdbc" Jul 18 12:27:00.953758 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:27:00.953538 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m6hdg\" (UniqueName: \"kubernetes.io/projected/2eede87b-bd0d-4e01-962b-583b7c7522cd-kube-api-access-m6hdg\") pod \"e2e-inf-b-f133b3-openshift-default-7c787bdc9f-ccdbc\" (UID: \"2eede87b-bd0d-4e01-962b-583b7c7522cd\") " pod="openshift-ingress/e2e-inf-b-f133b3-openshift-default-7c787bdc9f-ccdbc" Jul 18 12:27:00.953758 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:27:00.953693 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2eede87b-bd0d-4e01-962b-583b7c7522cd-workload-socket\") pod \"e2e-inf-b-f133b3-openshift-default-7c787bdc9f-ccdbc\" (UID: \"2eede87b-bd0d-4e01-962b-583b7c7522cd\") " pod="openshift-ingress/e2e-inf-b-f133b3-openshift-default-7c787bdc9f-ccdbc" Jul 18 12:27:00.953917 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:27:00.953815 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2eede87b-bd0d-4e01-962b-583b7c7522cd-istio-data\") pod \"e2e-inf-b-f133b3-openshift-default-7c787bdc9f-ccdbc\" (UID: \"2eede87b-bd0d-4e01-962b-583b7c7522cd\") " pod="openshift-ingress/e2e-inf-b-f133b3-openshift-default-7c787bdc9f-ccdbc" Jul 18 12:27:00.954026 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:27:00.953986 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2eede87b-bd0d-4e01-962b-583b7c7522cd-credential-socket\") pod \"e2e-inf-b-f133b3-openshift-default-7c787bdc9f-ccdbc\" (UID: \"2eede87b-bd0d-4e01-962b-583b7c7522cd\") " pod="openshift-ingress/e2e-inf-b-f133b3-openshift-default-7c787bdc9f-ccdbc" Jul 18 12:27:00.954167 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:27:00.954149 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2eede87b-bd0d-4e01-962b-583b7c7522cd-workload-certs\") pod \"e2e-inf-b-f133b3-openshift-default-7c787bdc9f-ccdbc\" (UID: \"2eede87b-bd0d-4e01-962b-583b7c7522cd\") " pod="openshift-ingress/e2e-inf-b-f133b3-openshift-default-7c787bdc9f-ccdbc" Jul 18 12:27:00.954239 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:27:00.954170 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2eede87b-bd0d-4e01-962b-583b7c7522cd-istiod-ca-cert\") pod \"e2e-inf-b-f133b3-openshift-default-7c787bdc9f-ccdbc\" (UID: \"2eede87b-bd0d-4e01-962b-583b7c7522cd\") " pod="openshift-ingress/e2e-inf-b-f133b3-openshift-default-7c787bdc9f-ccdbc" Jul 18 12:27:00.956391 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:27:00.956349 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2eede87b-bd0d-4e01-962b-583b7c7522cd-istio-envoy\") pod \"e2e-inf-b-f133b3-openshift-default-7c787bdc9f-ccdbc\" (UID: \"2eede87b-bd0d-4e01-962b-583b7c7522cd\") " pod="openshift-ingress/e2e-inf-b-f133b3-openshift-default-7c787bdc9f-ccdbc" Jul 18 12:27:00.956957 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:27:00.956933 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2eede87b-bd0d-4e01-962b-583b7c7522cd-istio-podinfo\") pod \"e2e-inf-b-f133b3-openshift-default-7c787bdc9f-ccdbc\" (UID: \"2eede87b-bd0d-4e01-962b-583b7c7522cd\") " pod="openshift-ingress/e2e-inf-b-f133b3-openshift-default-7c787bdc9f-ccdbc" Jul 18 12:27:00.961951 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:27:00.961929 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2eede87b-bd0d-4e01-962b-583b7c7522cd-istio-token\") pod \"e2e-inf-b-f133b3-openshift-default-7c787bdc9f-ccdbc\" (UID: \"2eede87b-bd0d-4e01-962b-583b7c7522cd\") " pod="openshift-ingress/e2e-inf-b-f133b3-openshift-default-7c787bdc9f-ccdbc" Jul 18 12:27:00.962034 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:27:00.961948 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m6hdg\" (UniqueName: \"kubernetes.io/projected/2eede87b-bd0d-4e01-962b-583b7c7522cd-kube-api-access-m6hdg\") pod \"e2e-inf-b-f133b3-openshift-default-7c787bdc9f-ccdbc\" (UID: \"2eede87b-bd0d-4e01-962b-583b7c7522cd\") " pod="openshift-ingress/e2e-inf-b-f133b3-openshift-default-7c787bdc9f-ccdbc" Jul 18 12:27:01.152679 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:27:01.152604 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-f133b3-openshift-default-7c787bdc9f-ccdbc" Jul 18 12:27:01.298806 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:27:01.298776 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-f133b3-openshift-default-7c787bdc9f-ccdbc"] Jul 18 12:27:01.300286 ip-10-0-136-240 kubenswrapper[2585]: W0718 12:27:01.300255 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2eede87b_bd0d_4e01_962b_583b7c7522cd.slice/crio-07c5ed0ba1252673cce0c661ae40dc5822dd74fcfd2b151ee56d177205d8059d WatchSource:0}: Error finding container 07c5ed0ba1252673cce0c661ae40dc5822dd74fcfd2b151ee56d177205d8059d: Status 404 returned error can't find the container with id 07c5ed0ba1252673cce0c661ae40dc5822dd74fcfd2b151ee56d177205d8059d Jul 18 12:27:01.302603 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:27:01.302573 2585 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 18 12:27:01.302692 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:27:01.302640 2585 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 18 12:27:01.309521 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:27:01.303176 2585 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 18 12:27:01.864243 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:27:01.864203 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-f133b3-openshift-default-7c787bdc9f-ccdbc" event={"ID":"2eede87b-bd0d-4e01-962b-583b7c7522cd","Type":"ContainerStarted","Data":"e14206932686cc084d2c1a44b1ef18bc7b5f287976f0826d4530eadea0630d21"} Jul 18 12:27:01.864771 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:27:01.864270 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-f133b3-openshift-default-7c787bdc9f-ccdbc" event={"ID":"2eede87b-bd0d-4e01-962b-583b7c7522cd","Type":"ContainerStarted","Data":"07c5ed0ba1252673cce0c661ae40dc5822dd74fcfd2b151ee56d177205d8059d"} Jul 18 12:27:01.883149 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:27:01.883098 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-f133b3-openshift-default-7c787bdc9f-ccdbc" podStartSLOduration=1.8830824229999998 podStartE2EDuration="1.883082423s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-18 12:27:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-18 12:27:01.881074425 +0000 UTC m=+2288.961280784" watchObservedRunningTime="2026-07-18 12:27:01.883082423 +0000 UTC m=+2288.963288774" Jul 18 12:27:02.153696 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:27:02.153618 2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-f133b3-openshift-default-7c787bdc9f-ccdbc" Jul 18 12:27:02.158669 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:27:02.158643 2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-f133b3-openshift-default-7c787bdc9f-ccdbc" Jul 18 12:27:02.872424 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:27:02.872038 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-f133b3-openshift-default-7c787bdc9f-ccdbc" Jul 18 12:27:02.874774 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:27:02.874750 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-f133b3-openshift-default-7c787bdc9f-ccdbc" Jul 18 12:27:09.831139 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:27:09.831108 2585 patch_prober.go:28] interesting pod/e2e-inf-a-3be5a3-openshift-default-76dcf5b787-89wjz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" start-of-body= Jul 18 12:27:09.831527 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:27:09.831164 2585 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-3be5a3-openshift-default-76dcf5b787-89wjz" podUID="f2a380d1-804a-4e1c-9853-2773e962a49c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" Jul 18 12:27:09.921386 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:27:09.921346 2585 generic.go:356] "Generic (PLEG): container finished" podID="f2a380d1-804a-4e1c-9853-2773e962a49c" containerID="0b1e75fa281636e01674b4f960d5db2a9a34cec889b72a83aaceac8870b5b365" exitCode=0 Jul 18 12:27:09.921536 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:27:09.921421 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-3be5a3-openshift-default-76dcf5b787-89wjz" event={"ID":"f2a380d1-804a-4e1c-9853-2773e962a49c","Type":"ContainerDied","Data":"0b1e75fa281636e01674b4f960d5db2a9a34cec889b72a83aaceac8870b5b365"} Jul 18 12:27:09.921834 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:27:09.921820 2585 scope.go:117] "RemoveContainer" containerID="0b1e75fa281636e01674b4f960d5db2a9a34cec889b72a83aaceac8870b5b365" Jul 18 12:27:09.922936 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:27:09.922911 2585 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 18 12:27:09.922992 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:27:09.922978 2585 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 18 12:27:09.923025 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:27:09.923010 2585 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 18 12:27:10.928022 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:27:10.927986 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-3be5a3-openshift-default-76dcf5b787-89wjz" event={"ID":"f2a380d1-804a-4e1c-9853-2773e962a49c","Type":"ContainerStarted","Data":"71770bc294b156f828122fea5f2582d7c7a25b830d038567ed48cc5bd9bbd111"} Jul 18 12:27:11.706049 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:27:11.706011 2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-3be5a3-openshift-default-76dcf5b787-89wjz" Jul 18 12:27:11.706262 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:27:11.706166 2585 patch_prober.go:28] interesting pod/e2e-inf-a-3be5a3-openshift-default-76dcf5b787-89wjz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" start-of-body= Jul 18 12:27:11.706262 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:27:11.706224 2585 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-3be5a3-openshift-default-76dcf5b787-89wjz" podUID="f2a380d1-804a-4e1c-9853-2773e962a49c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" Jul 18 12:27:12.705463 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:27:12.705421 2585 patch_prober.go:28] interesting pod/e2e-inf-a-3be5a3-openshift-default-76dcf5b787-89wjz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" start-of-body= Jul 18 12:27:12.705970 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:27:12.705497 2585 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-3be5a3-openshift-default-76dcf5b787-89wjz" podUID="f2a380d1-804a-4e1c-9853-2773e962a49c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" Jul 18 12:27:13.705451 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:27:13.705413 2585 patch_prober.go:28] interesting pod/e2e-inf-a-3be5a3-openshift-default-76dcf5b787-89wjz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" start-of-body= Jul 18 12:27:13.705784 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:27:13.705739 2585 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-3be5a3-openshift-default-76dcf5b787-89wjz" podUID="f2a380d1-804a-4e1c-9853-2773e962a49c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" Jul 18 12:27:14.705841 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:27:14.705796 2585 patch_prober.go:28] interesting pod/e2e-inf-a-3be5a3-openshift-default-76dcf5b787-89wjz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" start-of-body= Jul 18 12:27:14.706289 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:27:14.705864 2585 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-3be5a3-openshift-default-76dcf5b787-89wjz" podUID="f2a380d1-804a-4e1c-9853-2773e962a49c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" Jul 18 12:27:15.709717 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:27:15.709684 2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-3be5a3-openshift-default-76dcf5b787-89wjz" Jul 18 12:27:15.710148 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:27:15.709952 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-3be5a3-openshift-default-76dcf5b787-89wjz" Jul 18 12:27:15.710720 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:27:15.710700 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-3be5a3-openshift-default-76dcf5b787-89wjz" Jul 18 12:27:17.873528 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:27:17.873445 2585 patch_prober.go:28] interesting pod/e2e-inf-b-f133b3-openshift-default-7c787bdc9f-ccdbc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" start-of-body= Jul 18 12:27:17.873528 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:27:17.873501 2585 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-f133b3-openshift-default-7c787bdc9f-ccdbc" podUID="2eede87b-bd0d-4e01-962b-583b7c7522cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" Jul 18 12:27:17.965500 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:27:17.965470 2585 generic.go:356] "Generic (PLEG): container finished" podID="2eede87b-bd0d-4e01-962b-583b7c7522cd" containerID="e14206932686cc084d2c1a44b1ef18bc7b5f287976f0826d4530eadea0630d21" exitCode=0 Jul 18 12:27:17.965682 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:27:17.965543 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-f133b3-openshift-default-7c787bdc9f-ccdbc" event={"ID":"2eede87b-bd0d-4e01-962b-583b7c7522cd","Type":"ContainerDied","Data":"e14206932686cc084d2c1a44b1ef18bc7b5f287976f0826d4530eadea0630d21"} Jul 18 12:27:17.965938 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:27:17.965925 2585 scope.go:117] "RemoveContainer" containerID="e14206932686cc084d2c1a44b1ef18bc7b5f287976f0826d4530eadea0630d21" Jul 18 12:27:17.966921 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:27:17.966900 2585 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 18 12:27:17.966984 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:27:17.966959 2585 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 18 12:27:17.967021 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:27:17.966993 2585 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 18 12:27:18.976170 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:27:18.976122 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-f133b3-openshift-default-7c787bdc9f-ccdbc" event={"ID":"2eede87b-bd0d-4e01-962b-583b7c7522cd","Type":"ContainerStarted","Data":"72a191de19230a8eb7016ed473068591e013b18df50e2220b4f14e0607b5df58"} Jul 18 12:27:19.153187 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:27:19.153149 2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-f133b3-openshift-default-7c787bdc9f-ccdbc" Jul 18 12:27:19.153383 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:27:19.153326 2585 patch_prober.go:28] interesting pod/e2e-inf-b-f133b3-openshift-default-7c787bdc9f-ccdbc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" start-of-body= Jul 18 12:27:19.153458 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:27:19.153379 2585 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-f133b3-openshift-default-7c787bdc9f-ccdbc" podUID="2eede87b-bd0d-4e01-962b-583b7c7522cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" Jul 18 12:27:20.153753 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:27:20.153720 2585 patch_prober.go:28] interesting pod/e2e-inf-b-f133b3-openshift-default-7c787bdc9f-ccdbc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" start-of-body= Jul 18 12:27:20.154125 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:27:20.153778 2585 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-f133b3-openshift-default-7c787bdc9f-ccdbc" podUID="2eede87b-bd0d-4e01-962b-583b7c7522cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" Jul 18 12:27:21.153647 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:27:21.153611 2585 patch_prober.go:28] interesting pod/e2e-inf-b-f133b3-openshift-default-7c787bdc9f-ccdbc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" start-of-body= Jul 18 12:27:21.153853 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:27:21.153674 2585 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-f133b3-openshift-default-7c787bdc9f-ccdbc" podUID="2eede87b-bd0d-4e01-962b-583b7c7522cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" Jul 18 12:27:22.153683 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:27:22.153646 2585 patch_prober.go:28] interesting pod/e2e-inf-b-f133b3-openshift-default-7c787bdc9f-ccdbc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" start-of-body= Jul 18 12:27:22.153855 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:27:22.153700 2585 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-f133b3-openshift-default-7c787bdc9f-ccdbc" podUID="2eede87b-bd0d-4e01-962b-583b7c7522cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" Jul 18 12:27:23.157709 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:27:23.157680 2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-f133b3-openshift-default-7c787bdc9f-ccdbc" Jul 18 12:27:23.158211 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:27:23.157911 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-f133b3-openshift-default-7c787bdc9f-ccdbc" Jul 18 12:27:23.158542 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:27:23.158524 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-f133b3-openshift-default-7c787bdc9f-ccdbc" Jul 18 12:27:23.762040 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:27:23.762007 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-3be5a3/test-model-3be5a3-kserve-74b7b6ccd7-dp459"] Jul 18 12:27:23.765883 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:27:23.765860 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-3be5a3/test-model-3be5a3-kserve-74b7b6ccd7-dp459" Jul 18 12:27:23.768410 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:27:23.768384 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-3be5a3\"/\"kube-root-ca.crt\"" Jul 18 12:27:23.769221 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:27:23.769201 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-3be5a3\"/\"openshift-service-ca.crt\"" Jul 18 12:27:23.769221 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:27:23.769202 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-3be5a3\"/\"default-dockercfg-2626w\"" Jul 18 12:27:23.769388 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:27:23.769362 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-3be5a3\"/\"test-model-3be5a3-kserve-self-signed-certs\"" Jul 18 12:27:23.776927 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:27:23.776908 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-3be5a3/test-model-3be5a3-kserve-74b7b6ccd7-dp459"] Jul 18 12:27:23.898001 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:27:23.897970 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-whp5z\" (UniqueName: \"kubernetes.io/projected/0e799b0b-cf6c-490d-a40b-b796e9f723f3-kube-api-access-whp5z\") pod \"test-model-3be5a3-kserve-74b7b6ccd7-dp459\" (UID: \"0e799b0b-cf6c-490d-a40b-b796e9f723f3\") " pod="ai-tenant-e2e-inf-a-3be5a3/test-model-3be5a3-kserve-74b7b6ccd7-dp459" Jul 18 12:27:23.898144 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:27:23.898008 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0e799b0b-cf6c-490d-a40b-b796e9f723f3-tmp-dir\") pod \"test-model-3be5a3-kserve-74b7b6ccd7-dp459\" (UID: \"0e799b0b-cf6c-490d-a40b-b796e9f723f3\") " pod="ai-tenant-e2e-inf-a-3be5a3/test-model-3be5a3-kserve-74b7b6ccd7-dp459" Jul 18 12:27:23.898144 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:27:23.898027 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0e799b0b-cf6c-490d-a40b-b796e9f723f3-tls-certs\") pod \"test-model-3be5a3-kserve-74b7b6ccd7-dp459\" (UID: \"0e799b0b-cf6c-490d-a40b-b796e9f723f3\") " pod="ai-tenant-e2e-inf-a-3be5a3/test-model-3be5a3-kserve-74b7b6ccd7-dp459" Jul 18 12:27:23.898144 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:27:23.898093 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0e799b0b-cf6c-490d-a40b-b796e9f723f3-dshm\") pod \"test-model-3be5a3-kserve-74b7b6ccd7-dp459\" (UID: \"0e799b0b-cf6c-490d-a40b-b796e9f723f3\") " pod="ai-tenant-e2e-inf-a-3be5a3/test-model-3be5a3-kserve-74b7b6ccd7-dp459" Jul 18 12:27:23.898254 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:27:23.898166 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0e799b0b-cf6c-490d-a40b-b796e9f723f3-model-cache\") pod \"test-model-3be5a3-kserve-74b7b6ccd7-dp459\" (UID: \"0e799b0b-cf6c-490d-a40b-b796e9f723f3\") " pod="ai-tenant-e2e-inf-a-3be5a3/test-model-3be5a3-kserve-74b7b6ccd7-dp459" Jul 18 12:27:23.898254 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:27:23.898228 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0e799b0b-cf6c-490d-a40b-b796e9f723f3-home\") pod \"test-model-3be5a3-kserve-74b7b6ccd7-dp459\" (UID: \"0e799b0b-cf6c-490d-a40b-b796e9f723f3\") " pod="ai-tenant-e2e-inf-a-3be5a3/test-model-3be5a3-kserve-74b7b6ccd7-dp459" Jul 18 12:27:23.999522 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:27:23.999483 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0e799b0b-cf6c-490d-a40b-b796e9f723f3-model-cache\") pod \"test-model-3be5a3-kserve-74b7b6ccd7-dp459\" (UID: \"0e799b0b-cf6c-490d-a40b-b796e9f723f3\") " pod="ai-tenant-e2e-inf-a-3be5a3/test-model-3be5a3-kserve-74b7b6ccd7-dp459" Jul 18 12:27:23.999705 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:27:23.999541 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0e799b0b-cf6c-490d-a40b-b796e9f723f3-home\") pod \"test-model-3be5a3-kserve-74b7b6ccd7-dp459\" (UID: \"0e799b0b-cf6c-490d-a40b-b796e9f723f3\") " pod="ai-tenant-e2e-inf-a-3be5a3/test-model-3be5a3-kserve-74b7b6ccd7-dp459" Jul 18 12:27:23.999705 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:27:23.999626 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-whp5z\" (UniqueName: \"kubernetes.io/projected/0e799b0b-cf6c-490d-a40b-b796e9f723f3-kube-api-access-whp5z\") pod \"test-model-3be5a3-kserve-74b7b6ccd7-dp459\" (UID: \"0e799b0b-cf6c-490d-a40b-b796e9f723f3\") " pod="ai-tenant-e2e-inf-a-3be5a3/test-model-3be5a3-kserve-74b7b6ccd7-dp459" Jul 18 12:27:23.999705 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:27:23.999657 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0e799b0b-cf6c-490d-a40b-b796e9f723f3-tmp-dir\") pod \"test-model-3be5a3-kserve-74b7b6ccd7-dp459\" (UID: \"0e799b0b-cf6c-490d-a40b-b796e9f723f3\") " pod="ai-tenant-e2e-inf-a-3be5a3/test-model-3be5a3-kserve-74b7b6ccd7-dp459" Jul 18 12:27:23.999705 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:27:23.999682 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0e799b0b-cf6c-490d-a40b-b796e9f723f3-tls-certs\") pod \"test-model-3be5a3-kserve-74b7b6ccd7-dp459\" (UID: \"0e799b0b-cf6c-490d-a40b-b796e9f723f3\") " pod="ai-tenant-e2e-inf-a-3be5a3/test-model-3be5a3-kserve-74b7b6ccd7-dp459" Jul 18 12:27:23.999868 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:27:23.999712 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0e799b0b-cf6c-490d-a40b-b796e9f723f3-dshm\") pod \"test-model-3be5a3-kserve-74b7b6ccd7-dp459\" (UID: \"0e799b0b-cf6c-490d-a40b-b796e9f723f3\") " pod="ai-tenant-e2e-inf-a-3be5a3/test-model-3be5a3-kserve-74b7b6ccd7-dp459" Jul 18 12:27:23.999978 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:27:23.999961 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0e799b0b-cf6c-490d-a40b-b796e9f723f3-model-cache\") pod \"test-model-3be5a3-kserve-74b7b6ccd7-dp459\" (UID: \"0e799b0b-cf6c-490d-a40b-b796e9f723f3\") " pod="ai-tenant-e2e-inf-a-3be5a3/test-model-3be5a3-kserve-74b7b6ccd7-dp459" Jul 18 12:27:24.000017 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:27:24.000001 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0e799b0b-cf6c-490d-a40b-b796e9f723f3-home\") pod \"test-model-3be5a3-kserve-74b7b6ccd7-dp459\" (UID: \"0e799b0b-cf6c-490d-a40b-b796e9f723f3\") " pod="ai-tenant-e2e-inf-a-3be5a3/test-model-3be5a3-kserve-74b7b6ccd7-dp459" Jul 18 12:27:24.000053 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:27:24.000028 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0e799b0b-cf6c-490d-a40b-b796e9f723f3-tmp-dir\") pod \"test-model-3be5a3-kserve-74b7b6ccd7-dp459\" (UID: \"0e799b0b-cf6c-490d-a40b-b796e9f723f3\") " pod="ai-tenant-e2e-inf-a-3be5a3/test-model-3be5a3-kserve-74b7b6ccd7-dp459" Jul 18 12:27:24.002152 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:27:24.002127 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0e799b0b-cf6c-490d-a40b-b796e9f723f3-dshm\") pod \"test-model-3be5a3-kserve-74b7b6ccd7-dp459\" (UID: \"0e799b0b-cf6c-490d-a40b-b796e9f723f3\") " pod="ai-tenant-e2e-inf-a-3be5a3/test-model-3be5a3-kserve-74b7b6ccd7-dp459" Jul 18 12:27:24.002423 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:27:24.002406 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0e799b0b-cf6c-490d-a40b-b796e9f723f3-tls-certs\") pod \"test-model-3be5a3-kserve-74b7b6ccd7-dp459\" (UID: \"0e799b0b-cf6c-490d-a40b-b796e9f723f3\") " pod="ai-tenant-e2e-inf-a-3be5a3/test-model-3be5a3-kserve-74b7b6ccd7-dp459" Jul 18 12:27:24.008928 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:27:24.008908 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-whp5z\" (UniqueName: \"kubernetes.io/projected/0e799b0b-cf6c-490d-a40b-b796e9f723f3-kube-api-access-whp5z\") pod \"test-model-3be5a3-kserve-74b7b6ccd7-dp459\" (UID: \"0e799b0b-cf6c-490d-a40b-b796e9f723f3\") " pod="ai-tenant-e2e-inf-a-3be5a3/test-model-3be5a3-kserve-74b7b6ccd7-dp459" Jul 18 12:27:24.076219 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:27:24.076139 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-3be5a3/test-model-3be5a3-kserve-74b7b6ccd7-dp459" Jul 18 12:27:24.228650 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:27:24.228581 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-3be5a3/test-model-3be5a3-kserve-74b7b6ccd7-dp459"] Jul 18 12:27:24.231104 ip-10-0-136-240 kubenswrapper[2585]: W0718 12:27:24.231074 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0e799b0b_cf6c_490d_a40b_b796e9f723f3.slice/crio-ace2372b5149b5c69f096305387144d91ce0af43fc6bda6a3beefae789b64e1d WatchSource:0}: Error finding container ace2372b5149b5c69f096305387144d91ce0af43fc6bda6a3beefae789b64e1d: Status 404 returned error can't find the container with id ace2372b5149b5c69f096305387144d91ce0af43fc6bda6a3beefae789b64e1d Jul 18 12:27:25.014436 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:27:25.014395 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-3be5a3/test-model-3be5a3-kserve-74b7b6ccd7-dp459" event={"ID":"0e799b0b-cf6c-490d-a40b-b796e9f723f3","Type":"ContainerStarted","Data":"8fa6a17532bff59d9a060e774fa596b10bb5164288314296e3e2a06c276dd129"} Jul 18 12:27:25.014613 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:27:25.014449 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-3be5a3/test-model-3be5a3-kserve-74b7b6ccd7-dp459" event={"ID":"0e799b0b-cf6c-490d-a40b-b796e9f723f3","Type":"ContainerStarted","Data":"ace2372b5149b5c69f096305387144d91ce0af43fc6bda6a3beefae789b64e1d"} Jul 18 12:27:25.033838 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:27:25.033793 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-3be5a3/test-model-3be5a3-kserve-74b7b6ccd7-dp459" podStartSLOduration=2.033776486 podStartE2EDuration="2.033776486s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-18 12:27:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-18 12:27:25.031145434 +0000 UTC m=+2312.111351792" watchObservedRunningTime="2026-07-18 12:27:25.033776486 +0000 UTC m=+2312.113982842" Jul 18 12:27:34.077095 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:27:34.077061 2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-3be5a3/test-model-3be5a3-kserve-74b7b6ccd7-dp459" Jul 18 12:27:34.077494 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:27:34.077137 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-3be5a3/test-model-3be5a3-kserve-74b7b6ccd7-dp459" Jul 18 12:27:34.089837 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:27:34.089814 2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-3be5a3/test-model-3be5a3-kserve-74b7b6ccd7-dp459" Jul 18 12:27:35.082668 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:27:35.082629 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-3be5a3/test-model-3be5a3-kserve-74b7b6ccd7-dp459" Jul 18 12:27:48.439806 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:27:48.439597 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-f133b3/test-model-f133b3-kserve-5c674d9685-bwmnd"] Jul 18 12:27:48.443440 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:27:48.443418 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-f133b3/test-model-f133b3-kserve-5c674d9685-bwmnd" Jul 18 12:27:48.445826 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:27:48.445806 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-f133b3\"/\"kube-root-ca.crt\"" Jul 18 12:27:48.446662 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:27:48.446642 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-f133b3\"/\"default-dockercfg-9255b\"" Jul 18 12:27:48.446756 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:27:48.446688 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-f133b3\"/\"openshift-service-ca.crt\"" Jul 18 12:27:48.446816 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:27:48.446770 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-f133b3\"/\"test-model-f133b3-kserve-self-signed-certs\"" Jul 18 12:27:48.451898 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:27:48.451875 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-f133b3/test-model-f133b3-kserve-5c674d9685-bwmnd"] Jul 18 12:27:48.528921 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:27:48.528891 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/45ab15c2-6649-4a37-a41b-6883fa6ba6ae-tls-certs\") pod \"test-model-f133b3-kserve-5c674d9685-bwmnd\" (UID: \"45ab15c2-6649-4a37-a41b-6883fa6ba6ae\") " pod="ai-tenant-e2e-inf-b-f133b3/test-model-f133b3-kserve-5c674d9685-bwmnd" Jul 18 12:27:48.529051 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:27:48.528925 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/45ab15c2-6649-4a37-a41b-6883fa6ba6ae-tmp-dir\") pod \"test-model-f133b3-kserve-5c674d9685-bwmnd\" (UID: \"45ab15c2-6649-4a37-a41b-6883fa6ba6ae\") " pod="ai-tenant-e2e-inf-b-f133b3/test-model-f133b3-kserve-5c674d9685-bwmnd" Jul 18 12:27:48.529051 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:27:48.528955 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sc6r4\" (UniqueName: \"kubernetes.io/projected/45ab15c2-6649-4a37-a41b-6883fa6ba6ae-kube-api-access-sc6r4\") pod \"test-model-f133b3-kserve-5c674d9685-bwmnd\" (UID: \"45ab15c2-6649-4a37-a41b-6883fa6ba6ae\") " pod="ai-tenant-e2e-inf-b-f133b3/test-model-f133b3-kserve-5c674d9685-bwmnd" Jul 18 12:27:48.529051 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:27:48.529032 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/45ab15c2-6649-4a37-a41b-6883fa6ba6ae-dshm\") pod \"test-model-f133b3-kserve-5c674d9685-bwmnd\" (UID: \"45ab15c2-6649-4a37-a41b-6883fa6ba6ae\") " pod="ai-tenant-e2e-inf-b-f133b3/test-model-f133b3-kserve-5c674d9685-bwmnd" Jul 18 12:27:48.529161 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:27:48.529086 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/45ab15c2-6649-4a37-a41b-6883fa6ba6ae-model-cache\") pod \"test-model-f133b3-kserve-5c674d9685-bwmnd\" (UID: \"45ab15c2-6649-4a37-a41b-6883fa6ba6ae\") " pod="ai-tenant-e2e-inf-b-f133b3/test-model-f133b3-kserve-5c674d9685-bwmnd" Jul 18 12:27:48.529161 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:27:48.529119 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/45ab15c2-6649-4a37-a41b-6883fa6ba6ae-home\") pod \"test-model-f133b3-kserve-5c674d9685-bwmnd\" (UID: \"45ab15c2-6649-4a37-a41b-6883fa6ba6ae\") " pod="ai-tenant-e2e-inf-b-f133b3/test-model-f133b3-kserve-5c674d9685-bwmnd" Jul 18 12:27:48.630358 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:27:48.630326 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/45ab15c2-6649-4a37-a41b-6883fa6ba6ae-tls-certs\") pod \"test-model-f133b3-kserve-5c674d9685-bwmnd\" (UID: \"45ab15c2-6649-4a37-a41b-6883fa6ba6ae\") " pod="ai-tenant-e2e-inf-b-f133b3/test-model-f133b3-kserve-5c674d9685-bwmnd" Jul 18 12:27:48.630473 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:27:48.630360 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/45ab15c2-6649-4a37-a41b-6883fa6ba6ae-tmp-dir\") pod \"test-model-f133b3-kserve-5c674d9685-bwmnd\" (UID: \"45ab15c2-6649-4a37-a41b-6883fa6ba6ae\") " pod="ai-tenant-e2e-inf-b-f133b3/test-model-f133b3-kserve-5c674d9685-bwmnd" Jul 18 12:27:48.630473 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:27:48.630387 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sc6r4\" (UniqueName: \"kubernetes.io/projected/45ab15c2-6649-4a37-a41b-6883fa6ba6ae-kube-api-access-sc6r4\") pod \"test-model-f133b3-kserve-5c674d9685-bwmnd\" (UID: \"45ab15c2-6649-4a37-a41b-6883fa6ba6ae\") " pod="ai-tenant-e2e-inf-b-f133b3/test-model-f133b3-kserve-5c674d9685-bwmnd" Jul 18 12:27:48.630577 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:27:48.630511 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/45ab15c2-6649-4a37-a41b-6883fa6ba6ae-dshm\") pod \"test-model-f133b3-kserve-5c674d9685-bwmnd\" (UID: \"45ab15c2-6649-4a37-a41b-6883fa6ba6ae\") " pod="ai-tenant-e2e-inf-b-f133b3/test-model-f133b3-kserve-5c674d9685-bwmnd" Jul 18 12:27:48.630626 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:27:48.630586 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/45ab15c2-6649-4a37-a41b-6883fa6ba6ae-model-cache\") pod \"test-model-f133b3-kserve-5c674d9685-bwmnd\" (UID: \"45ab15c2-6649-4a37-a41b-6883fa6ba6ae\") " pod="ai-tenant-e2e-inf-b-f133b3/test-model-f133b3-kserve-5c674d9685-bwmnd" Jul 18 12:27:48.630684 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:27:48.630637 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/45ab15c2-6649-4a37-a41b-6883fa6ba6ae-home\") pod \"test-model-f133b3-kserve-5c674d9685-bwmnd\" (UID: \"45ab15c2-6649-4a37-a41b-6883fa6ba6ae\") " pod="ai-tenant-e2e-inf-b-f133b3/test-model-f133b3-kserve-5c674d9685-bwmnd" Jul 18 12:27:48.630985 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:27:48.630964 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/45ab15c2-6649-4a37-a41b-6883fa6ba6ae-tmp-dir\") pod \"test-model-f133b3-kserve-5c674d9685-bwmnd\" (UID: \"45ab15c2-6649-4a37-a41b-6883fa6ba6ae\") " pod="ai-tenant-e2e-inf-b-f133b3/test-model-f133b3-kserve-5c674d9685-bwmnd" Jul 18 12:27:48.631078 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:27:48.630997 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/45ab15c2-6649-4a37-a41b-6883fa6ba6ae-home\") pod \"test-model-f133b3-kserve-5c674d9685-bwmnd\" (UID: \"45ab15c2-6649-4a37-a41b-6883fa6ba6ae\") " pod="ai-tenant-e2e-inf-b-f133b3/test-model-f133b3-kserve-5c674d9685-bwmnd" Jul 18 12:27:48.631078 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:27:48.631069 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/45ab15c2-6649-4a37-a41b-6883fa6ba6ae-model-cache\") pod \"test-model-f133b3-kserve-5c674d9685-bwmnd\" (UID: \"45ab15c2-6649-4a37-a41b-6883fa6ba6ae\") " pod="ai-tenant-e2e-inf-b-f133b3/test-model-f133b3-kserve-5c674d9685-bwmnd" Jul 18 12:27:48.633473 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:27:48.633453 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/45ab15c2-6649-4a37-a41b-6883fa6ba6ae-dshm\") pod \"test-model-f133b3-kserve-5c674d9685-bwmnd\" (UID: \"45ab15c2-6649-4a37-a41b-6883fa6ba6ae\") " pod="ai-tenant-e2e-inf-b-f133b3/test-model-f133b3-kserve-5c674d9685-bwmnd" Jul 18 12:27:48.633760 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:27:48.633741 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/45ab15c2-6649-4a37-a41b-6883fa6ba6ae-tls-certs\") pod \"test-model-f133b3-kserve-5c674d9685-bwmnd\" (UID: \"45ab15c2-6649-4a37-a41b-6883fa6ba6ae\") " pod="ai-tenant-e2e-inf-b-f133b3/test-model-f133b3-kserve-5c674d9685-bwmnd" Jul 18 12:27:48.639432 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:27:48.639414 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sc6r4\" (UniqueName: \"kubernetes.io/projected/45ab15c2-6649-4a37-a41b-6883fa6ba6ae-kube-api-access-sc6r4\") pod \"test-model-f133b3-kserve-5c674d9685-bwmnd\" (UID: \"45ab15c2-6649-4a37-a41b-6883fa6ba6ae\") " pod="ai-tenant-e2e-inf-b-f133b3/test-model-f133b3-kserve-5c674d9685-bwmnd" Jul 18 12:27:48.754159 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:27:48.754125 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-f133b3/test-model-f133b3-kserve-5c674d9685-bwmnd" Jul 18 12:27:48.887501 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:27:48.887475 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-f133b3/test-model-f133b3-kserve-5c674d9685-bwmnd"] Jul 18 12:27:48.889118 ip-10-0-136-240 kubenswrapper[2585]: W0718 12:27:48.889090 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod45ab15c2_6649_4a37_a41b_6883fa6ba6ae.slice/crio-b21ecc3fe82da8aab00a1f08ff37147b38585f3a500387ca1735b9e7ff69bbb0 WatchSource:0}: Error finding container b21ecc3fe82da8aab00a1f08ff37147b38585f3a500387ca1735b9e7ff69bbb0: Status 404 returned error can't find the container with id b21ecc3fe82da8aab00a1f08ff37147b38585f3a500387ca1735b9e7ff69bbb0 Jul 18 12:27:49.140459 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:27:49.140379 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-f133b3/test-model-f133b3-kserve-5c674d9685-bwmnd" event={"ID":"45ab15c2-6649-4a37-a41b-6883fa6ba6ae","Type":"ContainerStarted","Data":"55002a9696a67157d0da646b489efb246e8193732977963dcff657fd2a0fef25"} Jul 18 12:27:49.140459 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:27:49.140417 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-f133b3/test-model-f133b3-kserve-5c674d9685-bwmnd" event={"ID":"45ab15c2-6649-4a37-a41b-6883fa6ba6ae","Type":"ContainerStarted","Data":"b21ecc3fe82da8aab00a1f08ff37147b38585f3a500387ca1735b9e7ff69bbb0"} Jul 18 12:27:49.158227 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:27:49.158172 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-f133b3/test-model-f133b3-kserve-5c674d9685-bwmnd" podStartSLOduration=1.158157766 podStartE2EDuration="1.158157766s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-18 12:27:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-18 12:27:49.156699836 +0000 UTC m=+2336.236906194" watchObservedRunningTime="2026-07-18 12:27:49.158157766 +0000 UTC m=+2336.238364164" Jul 18 12:27:58.755113 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:27:58.755074 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-f133b3/test-model-f133b3-kserve-5c674d9685-bwmnd" Jul 18 12:27:58.755605 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:27:58.755134 2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-f133b3/test-model-f133b3-kserve-5c674d9685-bwmnd" Jul 18 12:27:58.767793 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:27:58.767769 2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-f133b3/test-model-f133b3-kserve-5c674d9685-bwmnd" Jul 18 12:27:59.204677 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:27:59.204582 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-f133b3/test-model-f133b3-kserve-5c674d9685-bwmnd" Jul 18 12:28:22.248578 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:28:22.248531 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-3be5a3/test-model-3be5a3-kserve-74b7b6ccd7-dp459"] Jul 18 12:28:23.180600 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:28:23.180541 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-f133b3/test-model-f133b3-kserve-5c674d9685-bwmnd"] Jul 18 12:28:37.269287 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:28:37.269252 2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-3be5a3/test-model-3be5a3-kserve-74b7b6ccd7-dp459" podUID="0e799b0b-cf6c-490d-a40b-b796e9f723f3" containerName="main" containerID="cri-o://8fa6a17532bff59d9a060e774fa596b10bb5164288314296e3e2a06c276dd129" gracePeriod=45 Jul 18 12:28:37.387741 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:28:37.387716 2585 generic.go:356] "Generic (PLEG): container finished" podID="0e799b0b-cf6c-490d-a40b-b796e9f723f3" containerID="8fa6a17532bff59d9a060e774fa596b10bb5164288314296e3e2a06c276dd129" exitCode=0 Jul 18 12:28:37.387830 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:28:37.387786 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-3be5a3/test-model-3be5a3-kserve-74b7b6ccd7-dp459" event={"ID":"0e799b0b-cf6c-490d-a40b-b796e9f723f3","Type":"ContainerDied","Data":"8fa6a17532bff59d9a060e774fa596b10bb5164288314296e3e2a06c276dd129"} Jul 18 12:28:37.528491 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:28:37.528422 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-3be5a3/test-model-3be5a3-kserve-74b7b6ccd7-dp459" Jul 18 12:28:37.579777 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:28:37.579752 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-whp5z\" (UniqueName: \"kubernetes.io/projected/0e799b0b-cf6c-490d-a40b-b796e9f723f3-kube-api-access-whp5z\") pod \"0e799b0b-cf6c-490d-a40b-b796e9f723f3\" (UID: \"0e799b0b-cf6c-490d-a40b-b796e9f723f3\") " Jul 18 12:28:37.579943 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:28:37.579841 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0e799b0b-cf6c-490d-a40b-b796e9f723f3-tmp-dir\") pod \"0e799b0b-cf6c-490d-a40b-b796e9f723f3\" (UID: \"0e799b0b-cf6c-490d-a40b-b796e9f723f3\") " Jul 18 12:28:37.579943 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:28:37.579871 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0e799b0b-cf6c-490d-a40b-b796e9f723f3-home\") pod \"0e799b0b-cf6c-490d-a40b-b796e9f723f3\" (UID: \"0e799b0b-cf6c-490d-a40b-b796e9f723f3\") " Jul 18 12:28:37.579943 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:28:37.579907 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0e799b0b-cf6c-490d-a40b-b796e9f723f3-dshm\") pod \"0e799b0b-cf6c-490d-a40b-b796e9f723f3\" (UID: \"0e799b0b-cf6c-490d-a40b-b796e9f723f3\") " Jul 18 12:28:37.579943 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:28:37.579942 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0e799b0b-cf6c-490d-a40b-b796e9f723f3-model-cache\") pod \"0e799b0b-cf6c-490d-a40b-b796e9f723f3\" (UID: \"0e799b0b-cf6c-490d-a40b-b796e9f723f3\") " Jul 18 12:28:37.580183 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:28:37.579984 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0e799b0b-cf6c-490d-a40b-b796e9f723f3-tls-certs\") pod \"0e799b0b-cf6c-490d-a40b-b796e9f723f3\" (UID: \"0e799b0b-cf6c-490d-a40b-b796e9f723f3\") " Jul 18 12:28:37.580183 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:28:37.580119 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e799b0b-cf6c-490d-a40b-b796e9f723f3-home" (OuterVolumeSpecName: "home") pod "0e799b0b-cf6c-490d-a40b-b796e9f723f3" (UID: "0e799b0b-cf6c-490d-a40b-b796e9f723f3"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 18 12:28:37.580183 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:28:37.580135 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e799b0b-cf6c-490d-a40b-b796e9f723f3-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "0e799b0b-cf6c-490d-a40b-b796e9f723f3" (UID: "0e799b0b-cf6c-490d-a40b-b796e9f723f3"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 18 12:28:37.580403 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:28:37.580257 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e799b0b-cf6c-490d-a40b-b796e9f723f3-model-cache" (OuterVolumeSpecName: "model-cache") pod "0e799b0b-cf6c-490d-a40b-b796e9f723f3" (UID: "0e799b0b-cf6c-490d-a40b-b796e9f723f3"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 18 12:28:37.580403 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:28:37.580367 2585 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0e799b0b-cf6c-490d-a40b-b796e9f723f3-tmp-dir\") on node \"ip-10-0-136-240.ec2.internal\" DevicePath \"\"" Jul 18 12:28:37.580403 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:28:37.580382 2585 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0e799b0b-cf6c-490d-a40b-b796e9f723f3-home\") on node \"ip-10-0-136-240.ec2.internal\" DevicePath \"\"" Jul 18 12:28:37.580403 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:28:37.580394 2585 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0e799b0b-cf6c-490d-a40b-b796e9f723f3-model-cache\") on node \"ip-10-0-136-240.ec2.internal\" DevicePath \"\"" Jul 18 12:28:37.582370 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:28:37.582347 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0e799b0b-cf6c-490d-a40b-b796e9f723f3-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "0e799b0b-cf6c-490d-a40b-b796e9f723f3" (UID: "0e799b0b-cf6c-490d-a40b-b796e9f723f3"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 18 12:28:37.582657 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:28:37.582637 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0e799b0b-cf6c-490d-a40b-b796e9f723f3-kube-api-access-whp5z" (OuterVolumeSpecName: "kube-api-access-whp5z") pod "0e799b0b-cf6c-490d-a40b-b796e9f723f3" (UID: "0e799b0b-cf6c-490d-a40b-b796e9f723f3"). InnerVolumeSpecName "kube-api-access-whp5z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 18 12:28:37.582745 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:28:37.582671 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e799b0b-cf6c-490d-a40b-b796e9f723f3-dshm" (OuterVolumeSpecName: "dshm") pod "0e799b0b-cf6c-490d-a40b-b796e9f723f3" (UID: "0e799b0b-cf6c-490d-a40b-b796e9f723f3"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 18 12:28:37.681013 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:28:37.680991 2585 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0e799b0b-cf6c-490d-a40b-b796e9f723f3-dshm\") on node \"ip-10-0-136-240.ec2.internal\" DevicePath \"\"" Jul 18 12:28:37.681013 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:28:37.681013 2585 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0e799b0b-cf6c-490d-a40b-b796e9f723f3-tls-certs\") on node \"ip-10-0-136-240.ec2.internal\" DevicePath \"\"" Jul 18 12:28:37.681138 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:28:37.681023 2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-whp5z\" (UniqueName: \"kubernetes.io/projected/0e799b0b-cf6c-490d-a40b-b796e9f723f3-kube-api-access-whp5z\") on node \"ip-10-0-136-240.ec2.internal\" DevicePath \"\"" Jul 18 12:28:38.201573 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:28:38.201513 2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-f133b3/test-model-f133b3-kserve-5c674d9685-bwmnd" podUID="45ab15c2-6649-4a37-a41b-6883fa6ba6ae" containerName="main" containerID="cri-o://55002a9696a67157d0da646b489efb246e8193732977963dcff657fd2a0fef25" gracePeriod=45 Jul 18 12:28:38.395320 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:28:38.395294 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-3be5a3/test-model-3be5a3-kserve-74b7b6ccd7-dp459" Jul 18 12:28:38.395779 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:28:38.395329 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-3be5a3/test-model-3be5a3-kserve-74b7b6ccd7-dp459" event={"ID":"0e799b0b-cf6c-490d-a40b-b796e9f723f3","Type":"ContainerDied","Data":"ace2372b5149b5c69f096305387144d91ce0af43fc6bda6a3beefae789b64e1d"} Jul 18 12:28:38.395779 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:28:38.395389 2585 scope.go:117] "RemoveContainer" containerID="8fa6a17532bff59d9a060e774fa596b10bb5164288314296e3e2a06c276dd129" Jul 18 12:28:38.397474 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:28:38.397447 2585 generic.go:356] "Generic (PLEG): container finished" podID="45ab15c2-6649-4a37-a41b-6883fa6ba6ae" containerID="55002a9696a67157d0da646b489efb246e8193732977963dcff657fd2a0fef25" exitCode=0 Jul 18 12:28:38.397649 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:28:38.397521 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-f133b3/test-model-f133b3-kserve-5c674d9685-bwmnd" event={"ID":"45ab15c2-6649-4a37-a41b-6883fa6ba6ae","Type":"ContainerDied","Data":"55002a9696a67157d0da646b489efb246e8193732977963dcff657fd2a0fef25"} Jul 18 12:28:38.427442 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:28:38.427418 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-3be5a3/test-model-3be5a3-kserve-74b7b6ccd7-dp459"] Jul 18 12:28:38.432870 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:28:38.432850 2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-3be5a3/test-model-3be5a3-kserve-74b7b6ccd7-dp459"] Jul 18 12:28:38.456928 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:28:38.456880 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-f133b3/test-model-f133b3-kserve-5c674d9685-bwmnd" Jul 18 12:28:38.486852 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:28:38.486833 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/45ab15c2-6649-4a37-a41b-6883fa6ba6ae-tmp-dir\") pod \"45ab15c2-6649-4a37-a41b-6883fa6ba6ae\" (UID: \"45ab15c2-6649-4a37-a41b-6883fa6ba6ae\") " Jul 18 12:28:38.486946 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:28:38.486875 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sc6r4\" (UniqueName: \"kubernetes.io/projected/45ab15c2-6649-4a37-a41b-6883fa6ba6ae-kube-api-access-sc6r4\") pod \"45ab15c2-6649-4a37-a41b-6883fa6ba6ae\" (UID: \"45ab15c2-6649-4a37-a41b-6883fa6ba6ae\") " Jul 18 12:28:38.486946 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:28:38.486926 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/45ab15c2-6649-4a37-a41b-6883fa6ba6ae-model-cache\") pod \"45ab15c2-6649-4a37-a41b-6883fa6ba6ae\" (UID: \"45ab15c2-6649-4a37-a41b-6883fa6ba6ae\") " Jul 18 12:28:38.487142 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:28:38.486945 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/45ab15c2-6649-4a37-a41b-6883fa6ba6ae-dshm\") pod \"45ab15c2-6649-4a37-a41b-6883fa6ba6ae\" (UID: \"45ab15c2-6649-4a37-a41b-6883fa6ba6ae\") " Jul 18 12:28:38.487142 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:28:38.487004 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/45ab15c2-6649-4a37-a41b-6883fa6ba6ae-tls-certs\") pod \"45ab15c2-6649-4a37-a41b-6883fa6ba6ae\" (UID: \"45ab15c2-6649-4a37-a41b-6883fa6ba6ae\") " Jul 18 12:28:38.487142 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:28:38.487035 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/45ab15c2-6649-4a37-a41b-6883fa6ba6ae-home\") pod \"45ab15c2-6649-4a37-a41b-6883fa6ba6ae\" (UID: \"45ab15c2-6649-4a37-a41b-6883fa6ba6ae\") " Jul 18 12:28:38.487142 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:28:38.487091 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/45ab15c2-6649-4a37-a41b-6883fa6ba6ae-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "45ab15c2-6649-4a37-a41b-6883fa6ba6ae" (UID: "45ab15c2-6649-4a37-a41b-6883fa6ba6ae"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 18 12:28:38.487364 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:28:38.487342 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/45ab15c2-6649-4a37-a41b-6883fa6ba6ae-home" (OuterVolumeSpecName: "home") pod "45ab15c2-6649-4a37-a41b-6883fa6ba6ae" (UID: "45ab15c2-6649-4a37-a41b-6883fa6ba6ae"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 18 12:28:38.487436 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:28:38.487391 2585 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/45ab15c2-6649-4a37-a41b-6883fa6ba6ae-tmp-dir\") on node \"ip-10-0-136-240.ec2.internal\" DevicePath \"\"" Jul 18 12:28:38.487564 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:28:38.487282 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/45ab15c2-6649-4a37-a41b-6883fa6ba6ae-model-cache" (OuterVolumeSpecName: "model-cache") pod "45ab15c2-6649-4a37-a41b-6883fa6ba6ae" (UID: "45ab15c2-6649-4a37-a41b-6883fa6ba6ae"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 18 12:28:38.489430 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:28:38.489408 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/45ab15c2-6649-4a37-a41b-6883fa6ba6ae-kube-api-access-sc6r4" (OuterVolumeSpecName: "kube-api-access-sc6r4") pod "45ab15c2-6649-4a37-a41b-6883fa6ba6ae" (UID: "45ab15c2-6649-4a37-a41b-6883fa6ba6ae"). InnerVolumeSpecName "kube-api-access-sc6r4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 18 12:28:38.489525 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:28:38.489441 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/45ab15c2-6649-4a37-a41b-6883fa6ba6ae-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "45ab15c2-6649-4a37-a41b-6883fa6ba6ae" (UID: "45ab15c2-6649-4a37-a41b-6883fa6ba6ae"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 18 12:28:38.489604 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:28:38.489580 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/45ab15c2-6649-4a37-a41b-6883fa6ba6ae-dshm" (OuterVolumeSpecName: "dshm") pod "45ab15c2-6649-4a37-a41b-6883fa6ba6ae" (UID: "45ab15c2-6649-4a37-a41b-6883fa6ba6ae"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 18 12:28:38.587844 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:28:38.587816 2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sc6r4\" (UniqueName: \"kubernetes.io/projected/45ab15c2-6649-4a37-a41b-6883fa6ba6ae-kube-api-access-sc6r4\") on node \"ip-10-0-136-240.ec2.internal\" DevicePath \"\"" Jul 18 12:28:38.587844 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:28:38.587842 2585 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/45ab15c2-6649-4a37-a41b-6883fa6ba6ae-model-cache\") on node \"ip-10-0-136-240.ec2.internal\" DevicePath \"\"" Jul 18 12:28:38.587991 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:28:38.587856 2585 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/45ab15c2-6649-4a37-a41b-6883fa6ba6ae-dshm\") on node \"ip-10-0-136-240.ec2.internal\" DevicePath \"\"" Jul 18 12:28:38.587991 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:28:38.587867 2585 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/45ab15c2-6649-4a37-a41b-6883fa6ba6ae-tls-certs\") on node \"ip-10-0-136-240.ec2.internal\" DevicePath \"\"" Jul 18 12:28:38.587991 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:28:38.587877 2585 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/45ab15c2-6649-4a37-a41b-6883fa6ba6ae-home\") on node \"ip-10-0-136-240.ec2.internal\" DevicePath \"\"" Jul 18 12:28:39.405274 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:28:39.405247 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-f133b3/test-model-f133b3-kserve-5c674d9685-bwmnd" Jul 18 12:28:39.405709 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:28:39.405249 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-f133b3/test-model-f133b3-kserve-5c674d9685-bwmnd" event={"ID":"45ab15c2-6649-4a37-a41b-6883fa6ba6ae","Type":"ContainerDied","Data":"b21ecc3fe82da8aab00a1f08ff37147b38585f3a500387ca1735b9e7ff69bbb0"} Jul 18 12:28:39.405709 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:28:39.405377 2585 scope.go:117] "RemoveContainer" containerID="55002a9696a67157d0da646b489efb246e8193732977963dcff657fd2a0fef25" Jul 18 12:28:39.429058 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:28:39.429035 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-f133b3/test-model-f133b3-kserve-5c674d9685-bwmnd"] Jul 18 12:28:39.430895 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:28:39.430874 2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-f133b3/test-model-f133b3-kserve-5c674d9685-bwmnd"] Jul 18 12:28:39.469454 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:28:39.469433 2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0e799b0b-cf6c-490d-a40b-b796e9f723f3" path="/var/lib/kubelet/pods/0e799b0b-cf6c-490d-a40b-b796e9f723f3/volumes" Jul 18 12:28:39.469955 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:28:39.469934 2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="45ab15c2-6649-4a37-a41b-6883fa6ba6ae" path="/var/lib/kubelet/pods/45ab15c2-6649-4a37-a41b-6883fa6ba6ae/volumes" Jul 18 12:28:45.711037 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:28:45.711006 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-3be5a3-openshift-default-76dcf5b787-89wjz"] Jul 18 12:28:45.711863 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:28:45.711263 2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-3be5a3-openshift-default-76dcf5b787-89wjz" podUID="f2a380d1-804a-4e1c-9853-2773e962a49c" containerName="istio-proxy" containerID="cri-o://71770bc294b156f828122fea5f2582d7c7a25b830d038567ed48cc5bd9bbd111" gracePeriod=30 Jul 18 12:28:50.961717 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:28:50.961689 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-3be5a3-openshift-default-76dcf5b787-89wjz" Jul 18 12:28:51.099428 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:28:51.099350 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f2a380d1-804a-4e1c-9853-2773e962a49c-istiod-ca-cert\") pod \"f2a380d1-804a-4e1c-9853-2773e962a49c\" (UID: \"f2a380d1-804a-4e1c-9853-2773e962a49c\") " Jul 18 12:28:51.099428 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:28:51.099395 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f2a380d1-804a-4e1c-9853-2773e962a49c-istio-token\") pod \"f2a380d1-804a-4e1c-9853-2773e962a49c\" (UID: \"f2a380d1-804a-4e1c-9853-2773e962a49c\") " Jul 18 12:28:51.099668 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:28:51.099436 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f2a380d1-804a-4e1c-9853-2773e962a49c-istio-envoy\") pod \"f2a380d1-804a-4e1c-9853-2773e962a49c\" (UID: \"f2a380d1-804a-4e1c-9853-2773e962a49c\") " Jul 18 12:28:51.099668 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:28:51.099480 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2a380d1-804a-4e1c-9853-2773e962a49c-credential-socket\") pod \"f2a380d1-804a-4e1c-9853-2773e962a49c\" (UID: \"f2a380d1-804a-4e1c-9853-2773e962a49c\") " Jul 18 12:28:51.099668 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:28:51.099505 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f2a380d1-804a-4e1c-9853-2773e962a49c-istio-podinfo\") pod \"f2a380d1-804a-4e1c-9853-2773e962a49c\" (UID: \"f2a380d1-804a-4e1c-9853-2773e962a49c\") " Jul 18 12:28:51.099668 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:28:51.099587 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f2a380d1-804a-4e1c-9853-2773e962a49c-workload-certs\") pod \"f2a380d1-804a-4e1c-9853-2773e962a49c\" (UID: \"f2a380d1-804a-4e1c-9853-2773e962a49c\") " Jul 18 12:28:51.099668 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:28:51.099630 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7xm6h\" (UniqueName: \"kubernetes.io/projected/f2a380d1-804a-4e1c-9853-2773e962a49c-kube-api-access-7xm6h\") pod \"f2a380d1-804a-4e1c-9853-2773e962a49c\" (UID: \"f2a380d1-804a-4e1c-9853-2773e962a49c\") " Jul 18 12:28:51.099929 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:28:51.099679 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2a380d1-804a-4e1c-9853-2773e962a49c-workload-socket\") pod \"f2a380d1-804a-4e1c-9853-2773e962a49c\" (UID: \"f2a380d1-804a-4e1c-9853-2773e962a49c\") " Jul 18 12:28:51.099929 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:28:51.099754 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f2a380d1-804a-4e1c-9853-2773e962a49c-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "f2a380d1-804a-4e1c-9853-2773e962a49c" (UID: "f2a380d1-804a-4e1c-9853-2773e962a49c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 18 12:28:51.099929 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:28:51.099762 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f2a380d1-804a-4e1c-9853-2773e962a49c-istio-data\") pod \"f2a380d1-804a-4e1c-9853-2773e962a49c\" (UID: \"f2a380d1-804a-4e1c-9853-2773e962a49c\") " Jul 18 12:28:51.099929 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:28:51.099803 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f2a380d1-804a-4e1c-9853-2773e962a49c-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "f2a380d1-804a-4e1c-9853-2773e962a49c" (UID: "f2a380d1-804a-4e1c-9853-2773e962a49c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 18 12:28:51.100140 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:28:51.100014 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f2a380d1-804a-4e1c-9853-2773e962a49c-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "f2a380d1-804a-4e1c-9853-2773e962a49c" (UID: "f2a380d1-804a-4e1c-9853-2773e962a49c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 18 12:28:51.100140 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:28:51.100122 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f2a380d1-804a-4e1c-9853-2773e962a49c-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "f2a380d1-804a-4e1c-9853-2773e962a49c" (UID: "f2a380d1-804a-4e1c-9853-2773e962a49c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 18 12:28:51.100241 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:28:51.100222 2585 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f2a380d1-804a-4e1c-9853-2773e962a49c-istiod-ca-cert\") on node \"ip-10-0-136-240.ec2.internal\" DevicePath \"\"" Jul 18 12:28:51.100292 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:28:51.100240 2585 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2a380d1-804a-4e1c-9853-2773e962a49c-credential-socket\") on node \"ip-10-0-136-240.ec2.internal\" DevicePath \"\"" Jul 18 12:28:51.100292 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:28:51.100253 2585 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f2a380d1-804a-4e1c-9853-2773e962a49c-workload-certs\") on node \"ip-10-0-136-240.ec2.internal\" DevicePath \"\"" Jul 18 12:28:51.100292 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:28:51.100266 2585 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2a380d1-804a-4e1c-9853-2773e962a49c-workload-socket\") on node \"ip-10-0-136-240.ec2.internal\" DevicePath \"\"" Jul 18 12:28:51.100646 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:28:51.100624 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f2a380d1-804a-4e1c-9853-2773e962a49c-istio-data" (OuterVolumeSpecName: "istio-data") pod "f2a380d1-804a-4e1c-9853-2773e962a49c" (UID: "f2a380d1-804a-4e1c-9853-2773e962a49c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 18 12:28:51.101856 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:28:51.101828 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f2a380d1-804a-4e1c-9853-2773e962a49c-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "f2a380d1-804a-4e1c-9853-2773e962a49c" (UID: "f2a380d1-804a-4e1c-9853-2773e962a49c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 18 12:28:51.102083 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:28:51.102065 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f2a380d1-804a-4e1c-9853-2773e962a49c-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "f2a380d1-804a-4e1c-9853-2773e962a49c" (UID: "f2a380d1-804a-4e1c-9853-2773e962a49c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 18 12:28:51.102320 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:28:51.102295 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f2a380d1-804a-4e1c-9853-2773e962a49c-istio-token" (OuterVolumeSpecName: "istio-token") pod "f2a380d1-804a-4e1c-9853-2773e962a49c" (UID: "f2a380d1-804a-4e1c-9853-2773e962a49c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 18 12:28:51.102420 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:28:51.102323 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f2a380d1-804a-4e1c-9853-2773e962a49c-kube-api-access-7xm6h" (OuterVolumeSpecName: "kube-api-access-7xm6h") pod "f2a380d1-804a-4e1c-9853-2773e962a49c" (UID: "f2a380d1-804a-4e1c-9853-2773e962a49c"). InnerVolumeSpecName "kube-api-access-7xm6h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 18 12:28:51.201338 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:28:51.201312 2585 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f2a380d1-804a-4e1c-9853-2773e962a49c-istio-data\") on node \"ip-10-0-136-240.ec2.internal\" DevicePath \"\"" Jul 18 12:28:51.201338 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:28:51.201334 2585 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f2a380d1-804a-4e1c-9853-2773e962a49c-istio-token\") on node \"ip-10-0-136-240.ec2.internal\" DevicePath \"\"" Jul 18 12:28:51.201338 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:28:51.201343 2585 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f2a380d1-804a-4e1c-9853-2773e962a49c-istio-envoy\") on node \"ip-10-0-136-240.ec2.internal\" DevicePath \"\"" Jul 18 12:28:51.201523 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:28:51.201350 2585 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f2a380d1-804a-4e1c-9853-2773e962a49c-istio-podinfo\") on node \"ip-10-0-136-240.ec2.internal\" DevicePath \"\"" Jul 18 12:28:51.201523 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:28:51.201358 2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7xm6h\" (UniqueName: \"kubernetes.io/projected/f2a380d1-804a-4e1c-9853-2773e962a49c-kube-api-access-7xm6h\") on node \"ip-10-0-136-240.ec2.internal\" DevicePath \"\"" Jul 18 12:28:51.476064 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:28:51.476037 2585 generic.go:356] "Generic (PLEG): container finished" podID="f2a380d1-804a-4e1c-9853-2773e962a49c" containerID="71770bc294b156f828122fea5f2582d7c7a25b830d038567ed48cc5bd9bbd111" exitCode=0 Jul 18 12:28:51.476226 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:28:51.476077 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-3be5a3-openshift-default-76dcf5b787-89wjz" Jul 18 12:28:51.476226 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:28:51.476117 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-3be5a3-openshift-default-76dcf5b787-89wjz" event={"ID":"f2a380d1-804a-4e1c-9853-2773e962a49c","Type":"ContainerDied","Data":"71770bc294b156f828122fea5f2582d7c7a25b830d038567ed48cc5bd9bbd111"} Jul 18 12:28:51.476226 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:28:51.476156 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-3be5a3-openshift-default-76dcf5b787-89wjz" event={"ID":"f2a380d1-804a-4e1c-9853-2773e962a49c","Type":"ContainerDied","Data":"dbf3c0573dd3cc29154861f05a461185cfdab8abf4d7a2342d88e5657e4ba2cb"} Jul 18 12:28:51.476226 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:28:51.476180 2585 scope.go:117] "RemoveContainer" containerID="71770bc294b156f828122fea5f2582d7c7a25b830d038567ed48cc5bd9bbd111" Jul 18 12:28:51.485799 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:28:51.485779 2585 scope.go:117] "RemoveContainer" containerID="0b1e75fa281636e01674b4f960d5db2a9a34cec889b72a83aaceac8870b5b365" Jul 18 12:28:51.494831 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:28:51.494726 2585 scope.go:117] "RemoveContainer" containerID="71770bc294b156f828122fea5f2582d7c7a25b830d038567ed48cc5bd9bbd111" Jul 18 12:28:51.495037 ip-10-0-136-240 kubenswrapper[2585]: E0718 12:28:51.495014 2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"71770bc294b156f828122fea5f2582d7c7a25b830d038567ed48cc5bd9bbd111\": container with ID starting with 71770bc294b156f828122fea5f2582d7c7a25b830d038567ed48cc5bd9bbd111 not found: ID does not exist" containerID="71770bc294b156f828122fea5f2582d7c7a25b830d038567ed48cc5bd9bbd111" Jul 18 12:28:51.495185 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:28:51.495156 2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"71770bc294b156f828122fea5f2582d7c7a25b830d038567ed48cc5bd9bbd111"} err="failed to get container status \"71770bc294b156f828122fea5f2582d7c7a25b830d038567ed48cc5bd9bbd111\": rpc error: code = NotFound desc = could not find container \"71770bc294b156f828122fea5f2582d7c7a25b830d038567ed48cc5bd9bbd111\": container with ID starting with 71770bc294b156f828122fea5f2582d7c7a25b830d038567ed48cc5bd9bbd111 not found: ID does not exist" Jul 18 12:28:51.495287 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:28:51.495188 2585 scope.go:117] "RemoveContainer" containerID="0b1e75fa281636e01674b4f960d5db2a9a34cec889b72a83aaceac8870b5b365" Jul 18 12:28:51.495473 ip-10-0-136-240 kubenswrapper[2585]: E0718 12:28:51.495440 2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0b1e75fa281636e01674b4f960d5db2a9a34cec889b72a83aaceac8870b5b365\": container with ID starting with 0b1e75fa281636e01674b4f960d5db2a9a34cec889b72a83aaceac8870b5b365 not found: ID does not exist" containerID="0b1e75fa281636e01674b4f960d5db2a9a34cec889b72a83aaceac8870b5b365" Jul 18 12:28:51.495473 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:28:51.495463 2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0b1e75fa281636e01674b4f960d5db2a9a34cec889b72a83aaceac8870b5b365"} err="failed to get container status \"0b1e75fa281636e01674b4f960d5db2a9a34cec889b72a83aaceac8870b5b365\": rpc error: code = NotFound desc = could not find container \"0b1e75fa281636e01674b4f960d5db2a9a34cec889b72a83aaceac8870b5b365\": container with ID starting with 0b1e75fa281636e01674b4f960d5db2a9a34cec889b72a83aaceac8870b5b365 not found: ID does not exist" Jul 18 12:28:51.497454 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:28:51.497388 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-3be5a3-openshift-default-76dcf5b787-89wjz"] Jul 18 12:28:51.500394 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:28:51.500374 2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-3be5a3-openshift-default-76dcf5b787-89wjz"] Jul 18 12:28:53.469403 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:28:53.469371 2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f2a380d1-804a-4e1c-9853-2773e962a49c" path="/var/lib/kubelet/pods/f2a380d1-804a-4e1c-9853-2773e962a49c/volumes" Jul 18 12:28:53.585003 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:28:53.584975 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-j9l7p_8ec47eff-5fbc-4130-baef-d8544aa4e786/console-operator/1.log" Jul 18 12:28:53.592047 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:28:53.592026 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-wq4lg_247c97c8-982b-4176-ae10-afc396745360/ovn-acl-logging/0.log" Jul 18 12:28:53.594868 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:28:53.594851 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-j9l7p_8ec47eff-5fbc-4130-baef-d8544aa4e786/console-operator/1.log" Jul 18 12:28:53.601517 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:28:53.601501 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-wq4lg_247c97c8-982b-4176-ae10-afc396745360/ovn-acl-logging/0.log" Jul 18 12:29:08.052444 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:29:08.052402 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-f133b3-openshift-default-7c787bdc9f-ccdbc"] Jul 18 12:29:08.052970 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:29:08.052677 2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-f133b3-openshift-default-7c787bdc9f-ccdbc" podUID="2eede87b-bd0d-4e01-962b-583b7c7522cd" containerName="istio-proxy" containerID="cri-o://72a191de19230a8eb7016ed473068591e013b18df50e2220b4f14e0607b5df58" gracePeriod=30 Jul 18 12:29:08.160013 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:29:08.159979 2585 patch_prober.go:28] interesting pod/e2e-inf-b-f133b3-openshift-default-7c787bdc9f-ccdbc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 18 12:29:08.160175 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:29:08.160048 2585 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-f133b3-openshift-default-7c787bdc9f-ccdbc" podUID="2eede87b-bd0d-4e01-962b-583b7c7522cd" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 18 12:29:08.990658 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:29:08.990628 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-07813a-openshift-default-55b5b54787-lqmzm"] Jul 18 12:29:08.991004 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:29:08.990993 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="45ab15c2-6649-4a37-a41b-6883fa6ba6ae" containerName="main" Jul 18 12:29:08.991050 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:29:08.991006 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="45ab15c2-6649-4a37-a41b-6883fa6ba6ae" containerName="main" Jul 18 12:29:08.991050 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:29:08.991025 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0e799b0b-cf6c-490d-a40b-b796e9f723f3" containerName="main" Jul 18 12:29:08.991050 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:29:08.991031 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="0e799b0b-cf6c-490d-a40b-b796e9f723f3" containerName="main" Jul 18 12:29:08.991050 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:29:08.991039 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f2a380d1-804a-4e1c-9853-2773e962a49c" containerName="istio-proxy" Jul 18 12:29:08.991050 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:29:08.991044 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="f2a380d1-804a-4e1c-9853-2773e962a49c" containerName="istio-proxy" Jul 18 12:29:08.991201 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:29:08.991052 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f2a380d1-804a-4e1c-9853-2773e962a49c" containerName="istio-proxy" Jul 18 12:29:08.991201 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:29:08.991058 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="f2a380d1-804a-4e1c-9853-2773e962a49c" containerName="istio-proxy" Jul 18 12:29:08.991201 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:29:08.991113 2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="f2a380d1-804a-4e1c-9853-2773e962a49c" containerName="istio-proxy" Jul 18 12:29:08.991201 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:29:08.991120 2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="0e799b0b-cf6c-490d-a40b-b796e9f723f3" containerName="main" Jul 18 12:29:08.991201 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:29:08.991127 2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="45ab15c2-6649-4a37-a41b-6883fa6ba6ae" containerName="main" Jul 18 12:29:08.994239 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:29:08.994220 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-07813a-openshift-default-55b5b54787-lqmzm" Jul 18 12:29:08.996814 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:29:08.996794 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-07813a-openshift-default-dockercfg-9576k\"" Jul 18 12:29:09.010721 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:29:09.010696 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-07813a-openshift-default-55b5b54787-lqmzm"] Jul 18 12:29:09.169749 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:29:09.169714 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cf0cb8e4-36b7-4cb5-b4c1-09f746e04491-istiod-ca-cert\") pod \"e2e-api-a-07813a-openshift-default-55b5b54787-lqmzm\" (UID: \"cf0cb8e4-36b7-4cb5-b4c1-09f746e04491\") " pod="openshift-ingress/e2e-api-a-07813a-openshift-default-55b5b54787-lqmzm" Jul 18 12:29:09.169749 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:29:09.169751 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cf0cb8e4-36b7-4cb5-b4c1-09f746e04491-istio-podinfo\") pod \"e2e-api-a-07813a-openshift-default-55b5b54787-lqmzm\" (UID: \"cf0cb8e4-36b7-4cb5-b4c1-09f746e04491\") " pod="openshift-ingress/e2e-api-a-07813a-openshift-default-55b5b54787-lqmzm" Jul 18 12:29:09.170214 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:29:09.169777 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cf0cb8e4-36b7-4cb5-b4c1-09f746e04491-istio-envoy\") pod \"e2e-api-a-07813a-openshift-default-55b5b54787-lqmzm\" (UID: \"cf0cb8e4-36b7-4cb5-b4c1-09f746e04491\") " pod="openshift-ingress/e2e-api-a-07813a-openshift-default-55b5b54787-lqmzm" Jul 18 12:29:09.170214 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:29:09.169798 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cf0cb8e4-36b7-4cb5-b4c1-09f746e04491-istio-token\") pod \"e2e-api-a-07813a-openshift-default-55b5b54787-lqmzm\" (UID: \"cf0cb8e4-36b7-4cb5-b4c1-09f746e04491\") " pod="openshift-ingress/e2e-api-a-07813a-openshift-default-55b5b54787-lqmzm" Jul 18 12:29:09.170214 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:29:09.169846 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf0cb8e4-36b7-4cb5-b4c1-09f746e04491-credential-socket\") pod \"e2e-api-a-07813a-openshift-default-55b5b54787-lqmzm\" (UID: \"cf0cb8e4-36b7-4cb5-b4c1-09f746e04491\") " pod="openshift-ingress/e2e-api-a-07813a-openshift-default-55b5b54787-lqmzm" Jul 18 12:29:09.170214 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:29:09.169901 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hpj97\" (UniqueName: \"kubernetes.io/projected/cf0cb8e4-36b7-4cb5-b4c1-09f746e04491-kube-api-access-hpj97\") pod \"e2e-api-a-07813a-openshift-default-55b5b54787-lqmzm\" (UID: \"cf0cb8e4-36b7-4cb5-b4c1-09f746e04491\") " pod="openshift-ingress/e2e-api-a-07813a-openshift-default-55b5b54787-lqmzm" Jul 18 12:29:09.170214 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:29:09.169939 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cf0cb8e4-36b7-4cb5-b4c1-09f746e04491-workload-certs\") pod \"e2e-api-a-07813a-openshift-default-55b5b54787-lqmzm\" (UID: \"cf0cb8e4-36b7-4cb5-b4c1-09f746e04491\") " pod="openshift-ingress/e2e-api-a-07813a-openshift-default-55b5b54787-lqmzm" Jul 18 12:29:09.170214 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:29:09.169959 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cf0cb8e4-36b7-4cb5-b4c1-09f746e04491-istio-data\") pod \"e2e-api-a-07813a-openshift-default-55b5b54787-lqmzm\" (UID: \"cf0cb8e4-36b7-4cb5-b4c1-09f746e04491\") " pod="openshift-ingress/e2e-api-a-07813a-openshift-default-55b5b54787-lqmzm" Jul 18 12:29:09.170214 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:29:09.170051 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf0cb8e4-36b7-4cb5-b4c1-09f746e04491-workload-socket\") pod \"e2e-api-a-07813a-openshift-default-55b5b54787-lqmzm\" (UID: \"cf0cb8e4-36b7-4cb5-b4c1-09f746e04491\") " pod="openshift-ingress/e2e-api-a-07813a-openshift-default-55b5b54787-lqmzm" Jul 18 12:29:09.277397 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:29:09.277304 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cf0cb8e4-36b7-4cb5-b4c1-09f746e04491-workload-certs\") pod \"e2e-api-a-07813a-openshift-default-55b5b54787-lqmzm\" (UID: \"cf0cb8e4-36b7-4cb5-b4c1-09f746e04491\") " pod="openshift-ingress/e2e-api-a-07813a-openshift-default-55b5b54787-lqmzm" Jul 18 12:29:09.277397 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:29:09.277354 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cf0cb8e4-36b7-4cb5-b4c1-09f746e04491-istio-data\") pod \"e2e-api-a-07813a-openshift-default-55b5b54787-lqmzm\" (UID: \"cf0cb8e4-36b7-4cb5-b4c1-09f746e04491\") " pod="openshift-ingress/e2e-api-a-07813a-openshift-default-55b5b54787-lqmzm" Jul 18 12:29:09.277663 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:29:09.277399 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf0cb8e4-36b7-4cb5-b4c1-09f746e04491-workload-socket\") pod \"e2e-api-a-07813a-openshift-default-55b5b54787-lqmzm\" (UID: \"cf0cb8e4-36b7-4cb5-b4c1-09f746e04491\") " pod="openshift-ingress/e2e-api-a-07813a-openshift-default-55b5b54787-lqmzm" Jul 18 12:29:09.277663 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:29:09.277484 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cf0cb8e4-36b7-4cb5-b4c1-09f746e04491-istiod-ca-cert\") pod \"e2e-api-a-07813a-openshift-default-55b5b54787-lqmzm\" (UID: \"cf0cb8e4-36b7-4cb5-b4c1-09f746e04491\") " pod="openshift-ingress/e2e-api-a-07813a-openshift-default-55b5b54787-lqmzm" Jul 18 12:29:09.277663 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:29:09.277516 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cf0cb8e4-36b7-4cb5-b4c1-09f746e04491-istio-podinfo\") pod \"e2e-api-a-07813a-openshift-default-55b5b54787-lqmzm\" (UID: \"cf0cb8e4-36b7-4cb5-b4c1-09f746e04491\") " pod="openshift-ingress/e2e-api-a-07813a-openshift-default-55b5b54787-lqmzm" Jul 18 12:29:09.277663 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:29:09.277579 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cf0cb8e4-36b7-4cb5-b4c1-09f746e04491-istio-envoy\") pod \"e2e-api-a-07813a-openshift-default-55b5b54787-lqmzm\" (UID: \"cf0cb8e4-36b7-4cb5-b4c1-09f746e04491\") " pod="openshift-ingress/e2e-api-a-07813a-openshift-default-55b5b54787-lqmzm" Jul 18 12:29:09.277663 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:29:09.277612 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cf0cb8e4-36b7-4cb5-b4c1-09f746e04491-istio-token\") pod \"e2e-api-a-07813a-openshift-default-55b5b54787-lqmzm\" (UID: \"cf0cb8e4-36b7-4cb5-b4c1-09f746e04491\") " pod="openshift-ingress/e2e-api-a-07813a-openshift-default-55b5b54787-lqmzm" Jul 18 12:29:09.277663 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:29:09.277652 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf0cb8e4-36b7-4cb5-b4c1-09f746e04491-credential-socket\") pod \"e2e-api-a-07813a-openshift-default-55b5b54787-lqmzm\" (UID: \"cf0cb8e4-36b7-4cb5-b4c1-09f746e04491\") " pod="openshift-ingress/e2e-api-a-07813a-openshift-default-55b5b54787-lqmzm" Jul 18 12:29:09.277980 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:29:09.277686 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hpj97\" (UniqueName: \"kubernetes.io/projected/cf0cb8e4-36b7-4cb5-b4c1-09f746e04491-kube-api-access-hpj97\") pod \"e2e-api-a-07813a-openshift-default-55b5b54787-lqmzm\" (UID: \"cf0cb8e4-36b7-4cb5-b4c1-09f746e04491\") " pod="openshift-ingress/e2e-api-a-07813a-openshift-default-55b5b54787-lqmzm" Jul 18 12:29:09.277980 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:29:09.277745 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cf0cb8e4-36b7-4cb5-b4c1-09f746e04491-workload-certs\") pod \"e2e-api-a-07813a-openshift-default-55b5b54787-lqmzm\" (UID: \"cf0cb8e4-36b7-4cb5-b4c1-09f746e04491\") " pod="openshift-ingress/e2e-api-a-07813a-openshift-default-55b5b54787-lqmzm" Jul 18 12:29:09.277980 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:29:09.277786 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cf0cb8e4-36b7-4cb5-b4c1-09f746e04491-istio-data\") pod \"e2e-api-a-07813a-openshift-default-55b5b54787-lqmzm\" (UID: \"cf0cb8e4-36b7-4cb5-b4c1-09f746e04491\") " pod="openshift-ingress/e2e-api-a-07813a-openshift-default-55b5b54787-lqmzm" Jul 18 12:29:09.277980 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:29:09.277854 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf0cb8e4-36b7-4cb5-b4c1-09f746e04491-workload-socket\") pod \"e2e-api-a-07813a-openshift-default-55b5b54787-lqmzm\" (UID: \"cf0cb8e4-36b7-4cb5-b4c1-09f746e04491\") " pod="openshift-ingress/e2e-api-a-07813a-openshift-default-55b5b54787-lqmzm" Jul 18 12:29:09.278187 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:29:09.278095 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf0cb8e4-36b7-4cb5-b4c1-09f746e04491-credential-socket\") pod \"e2e-api-a-07813a-openshift-default-55b5b54787-lqmzm\" (UID: \"cf0cb8e4-36b7-4cb5-b4c1-09f746e04491\") " pod="openshift-ingress/e2e-api-a-07813a-openshift-default-55b5b54787-lqmzm" Jul 18 12:29:09.278348 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:29:09.278320 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cf0cb8e4-36b7-4cb5-b4c1-09f746e04491-istiod-ca-cert\") pod \"e2e-api-a-07813a-openshift-default-55b5b54787-lqmzm\" (UID: \"cf0cb8e4-36b7-4cb5-b4c1-09f746e04491\") " pod="openshift-ingress/e2e-api-a-07813a-openshift-default-55b5b54787-lqmzm" Jul 18 12:29:09.280039 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:29:09.280018 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cf0cb8e4-36b7-4cb5-b4c1-09f746e04491-istio-envoy\") pod \"e2e-api-a-07813a-openshift-default-55b5b54787-lqmzm\" (UID: \"cf0cb8e4-36b7-4cb5-b4c1-09f746e04491\") " pod="openshift-ingress/e2e-api-a-07813a-openshift-default-55b5b54787-lqmzm" Jul 18 12:29:09.280250 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:29:09.280234 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cf0cb8e4-36b7-4cb5-b4c1-09f746e04491-istio-podinfo\") pod \"e2e-api-a-07813a-openshift-default-55b5b54787-lqmzm\" (UID: \"cf0cb8e4-36b7-4cb5-b4c1-09f746e04491\") " pod="openshift-ingress/e2e-api-a-07813a-openshift-default-55b5b54787-lqmzm" Jul 18 12:29:09.285923 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:29:09.285898 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hpj97\" (UniqueName: \"kubernetes.io/projected/cf0cb8e4-36b7-4cb5-b4c1-09f746e04491-kube-api-access-hpj97\") pod \"e2e-api-a-07813a-openshift-default-55b5b54787-lqmzm\" (UID: \"cf0cb8e4-36b7-4cb5-b4c1-09f746e04491\") " pod="openshift-ingress/e2e-api-a-07813a-openshift-default-55b5b54787-lqmzm" Jul 18 12:29:09.285923 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:29:09.285922 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cf0cb8e4-36b7-4cb5-b4c1-09f746e04491-istio-token\") pod \"e2e-api-a-07813a-openshift-default-55b5b54787-lqmzm\" (UID: \"cf0cb8e4-36b7-4cb5-b4c1-09f746e04491\") " pod="openshift-ingress/e2e-api-a-07813a-openshift-default-55b5b54787-lqmzm" Jul 18 12:29:09.305443 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:29:09.305421 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-07813a-openshift-default-55b5b54787-lqmzm" Jul 18 12:29:09.450314 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:29:09.450287 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-07813a-openshift-default-55b5b54787-lqmzm"] Jul 18 12:29:09.452224 ip-10-0-136-240 kubenswrapper[2585]: W0718 12:29:09.452197 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcf0cb8e4_36b7_4cb5_b4c1_09f746e04491.slice/crio-a866be9ab39b13652f602ae5e3382fc5e0982d2273ae0144ffb9c7d884eb1d20 WatchSource:0}: Error finding container a866be9ab39b13652f602ae5e3382fc5e0982d2273ae0144ffb9c7d884eb1d20: Status 404 returned error can't find the container with id a866be9ab39b13652f602ae5e3382fc5e0982d2273ae0144ffb9c7d884eb1d20 Jul 18 12:29:09.454290 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:29:09.454258 2585 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 18 12:29:09.454380 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:29:09.454323 2585 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 18 12:29:09.454380 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:29:09.454353 2585 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 18 12:29:09.576683 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:29:09.576578 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-07813a-openshift-default-55b5b54787-lqmzm" event={"ID":"cf0cb8e4-36b7-4cb5-b4c1-09f746e04491","Type":"ContainerStarted","Data":"68913856927afa3fc43589347cf16e511dc6961433d1fabb279a9e06abcba643"} Jul 18 12:29:09.576683 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:29:09.576671 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-07813a-openshift-default-55b5b54787-lqmzm" event={"ID":"cf0cb8e4-36b7-4cb5-b4c1-09f746e04491","Type":"ContainerStarted","Data":"a866be9ab39b13652f602ae5e3382fc5e0982d2273ae0144ffb9c7d884eb1d20"} Jul 18 12:29:09.605177 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:29:09.605126 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-07813a-openshift-default-55b5b54787-lqmzm" podStartSLOduration=1.605109943 podStartE2EDuration="1.605109943s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-18 12:29:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-18 12:29:09.603941389 +0000 UTC m=+2416.684147747" watchObservedRunningTime="2026-07-18 12:29:09.605109943 +0000 UTC m=+2416.685316300" Jul 18 12:29:10.306533 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:29:10.306493 2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-07813a-openshift-default-55b5b54787-lqmzm" Jul 18 12:29:10.311666 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:29:10.311643 2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-07813a-openshift-default-55b5b54787-lqmzm" Jul 18 12:29:10.582428 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:29:10.582337 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-07813a-openshift-default-55b5b54787-lqmzm" Jul 18 12:29:10.583258 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:29:10.583239 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-07813a-openshift-default-55b5b54787-lqmzm" Jul 18 12:29:13.323651 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:29:13.323592 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-f133b3-openshift-default-7c787bdc9f-ccdbc" Jul 18 12:29:13.419329 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:29:13.419295 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2eede87b-bd0d-4e01-962b-583b7c7522cd-workload-socket\") pod \"2eede87b-bd0d-4e01-962b-583b7c7522cd\" (UID: \"2eede87b-bd0d-4e01-962b-583b7c7522cd\") " Jul 18 12:29:13.419492 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:29:13.419347 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-m6hdg\" (UniqueName: \"kubernetes.io/projected/2eede87b-bd0d-4e01-962b-583b7c7522cd-kube-api-access-m6hdg\") pod \"2eede87b-bd0d-4e01-962b-583b7c7522cd\" (UID: \"2eede87b-bd0d-4e01-962b-583b7c7522cd\") " Jul 18 12:29:13.419492 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:29:13.419369 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2eede87b-bd0d-4e01-962b-583b7c7522cd-istio-token\") pod \"2eede87b-bd0d-4e01-962b-583b7c7522cd\" (UID: \"2eede87b-bd0d-4e01-962b-583b7c7522cd\") " Jul 18 12:29:13.419492 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:29:13.419394 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2eede87b-bd0d-4e01-962b-583b7c7522cd-istio-podinfo\") pod \"2eede87b-bd0d-4e01-962b-583b7c7522cd\" (UID: \"2eede87b-bd0d-4e01-962b-583b7c7522cd\") " Jul 18 12:29:13.419492 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:29:13.419450 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2eede87b-bd0d-4e01-962b-583b7c7522cd-credential-socket\") pod \"2eede87b-bd0d-4e01-962b-583b7c7522cd\" (UID: \"2eede87b-bd0d-4e01-962b-583b7c7522cd\") " Jul 18 12:29:13.419752 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:29:13.419522 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2eede87b-bd0d-4e01-962b-583b7c7522cd-istiod-ca-cert\") pod \"2eede87b-bd0d-4e01-962b-583b7c7522cd\" (UID: \"2eede87b-bd0d-4e01-962b-583b7c7522cd\") " Jul 18 12:29:13.419752 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:29:13.419585 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2eede87b-bd0d-4e01-962b-583b7c7522cd-istio-data\") pod \"2eede87b-bd0d-4e01-962b-583b7c7522cd\" (UID: \"2eede87b-bd0d-4e01-962b-583b7c7522cd\") " Jul 18 12:29:13.419752 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:29:13.419615 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2eede87b-bd0d-4e01-962b-583b7c7522cd-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "2eede87b-bd0d-4e01-962b-583b7c7522cd" (UID: "2eede87b-bd0d-4e01-962b-583b7c7522cd"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 18 12:29:13.419752 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:29:13.419669 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2eede87b-bd0d-4e01-962b-583b7c7522cd-workload-certs\") pod \"2eede87b-bd0d-4e01-962b-583b7c7522cd\" (UID: \"2eede87b-bd0d-4e01-962b-583b7c7522cd\") " Jul 18 12:29:13.419947 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:29:13.419799 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2eede87b-bd0d-4e01-962b-583b7c7522cd-istio-envoy\") pod \"2eede87b-bd0d-4e01-962b-583b7c7522cd\" (UID: \"2eede87b-bd0d-4e01-962b-583b7c7522cd\") " Jul 18 12:29:13.419947 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:29:13.419825 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2eede87b-bd0d-4e01-962b-583b7c7522cd-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "2eede87b-bd0d-4e01-962b-583b7c7522cd" (UID: "2eede87b-bd0d-4e01-962b-583b7c7522cd"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 18 12:29:13.419947 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:29:13.419917 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2eede87b-bd0d-4e01-962b-583b7c7522cd-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "2eede87b-bd0d-4e01-962b-583b7c7522cd" (UID: "2eede87b-bd0d-4e01-962b-583b7c7522cd"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 18 12:29:13.420095 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:29:13.420015 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2eede87b-bd0d-4e01-962b-583b7c7522cd-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "2eede87b-bd0d-4e01-962b-583b7c7522cd" (UID: "2eede87b-bd0d-4e01-962b-583b7c7522cd"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 18 12:29:13.420254 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:29:13.420217 2585 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2eede87b-bd0d-4e01-962b-583b7c7522cd-workload-socket\") on node \"ip-10-0-136-240.ec2.internal\" DevicePath \"\"" Jul 18 12:29:13.420254 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:29:13.420241 2585 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2eede87b-bd0d-4e01-962b-583b7c7522cd-credential-socket\") on node \"ip-10-0-136-240.ec2.internal\" DevicePath \"\"" Jul 18 12:29:13.420370 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:29:13.420257 2585 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2eede87b-bd0d-4e01-962b-583b7c7522cd-istiod-ca-cert\") on node \"ip-10-0-136-240.ec2.internal\" DevicePath \"\"" Jul 18 12:29:13.420370 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:29:13.420270 2585 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2eede87b-bd0d-4e01-962b-583b7c7522cd-workload-certs\") on node \"ip-10-0-136-240.ec2.internal\" DevicePath \"\"" Jul 18 12:29:13.420568 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:29:13.420526 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2eede87b-bd0d-4e01-962b-583b7c7522cd-istio-data" (OuterVolumeSpecName: "istio-data") pod "2eede87b-bd0d-4e01-962b-583b7c7522cd" (UID: "2eede87b-bd0d-4e01-962b-583b7c7522cd"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 18 12:29:13.421771 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:29:13.421750 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2eede87b-bd0d-4e01-962b-583b7c7522cd-istio-token" (OuterVolumeSpecName: "istio-token") pod "2eede87b-bd0d-4e01-962b-583b7c7522cd" (UID: "2eede87b-bd0d-4e01-962b-583b7c7522cd"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 18 12:29:13.421867 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:29:13.421783 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2eede87b-bd0d-4e01-962b-583b7c7522cd-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "2eede87b-bd0d-4e01-962b-583b7c7522cd" (UID: "2eede87b-bd0d-4e01-962b-583b7c7522cd"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 18 12:29:13.422406 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:29:13.422381 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2eede87b-bd0d-4e01-962b-583b7c7522cd-kube-api-access-m6hdg" (OuterVolumeSpecName: "kube-api-access-m6hdg") pod "2eede87b-bd0d-4e01-962b-583b7c7522cd" (UID: "2eede87b-bd0d-4e01-962b-583b7c7522cd"). InnerVolumeSpecName "kube-api-access-m6hdg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 18 12:29:13.422406 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:29:13.422394 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2eede87b-bd0d-4e01-962b-583b7c7522cd-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "2eede87b-bd0d-4e01-962b-583b7c7522cd" (UID: "2eede87b-bd0d-4e01-962b-583b7c7522cd"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 18 12:29:13.520724 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:29:13.520699 2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m6hdg\" (UniqueName: \"kubernetes.io/projected/2eede87b-bd0d-4e01-962b-583b7c7522cd-kube-api-access-m6hdg\") on node \"ip-10-0-136-240.ec2.internal\" DevicePath \"\"" Jul 18 12:29:13.520724 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:29:13.520722 2585 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2eede87b-bd0d-4e01-962b-583b7c7522cd-istio-token\") on node \"ip-10-0-136-240.ec2.internal\" DevicePath \"\"" Jul 18 12:29:13.520857 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:29:13.520732 2585 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2eede87b-bd0d-4e01-962b-583b7c7522cd-istio-podinfo\") on node \"ip-10-0-136-240.ec2.internal\" DevicePath \"\"" Jul 18 12:29:13.520857 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:29:13.520740 2585 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2eede87b-bd0d-4e01-962b-583b7c7522cd-istio-data\") on node \"ip-10-0-136-240.ec2.internal\" DevicePath \"\"" Jul 18 12:29:13.520857 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:29:13.520748 2585 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2eede87b-bd0d-4e01-962b-583b7c7522cd-istio-envoy\") on node \"ip-10-0-136-240.ec2.internal\" DevicePath \"\"" Jul 18 12:29:13.601129 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:29:13.601062 2585 generic.go:356] "Generic (PLEG): container finished" podID="2eede87b-bd0d-4e01-962b-583b7c7522cd" containerID="72a191de19230a8eb7016ed473068591e013b18df50e2220b4f14e0607b5df58" exitCode=0 Jul 18 12:29:13.601129 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:29:13.601113 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-f133b3-openshift-default-7c787bdc9f-ccdbc" Jul 18 12:29:13.601266 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:29:13.601106 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-f133b3-openshift-default-7c787bdc9f-ccdbc" event={"ID":"2eede87b-bd0d-4e01-962b-583b7c7522cd","Type":"ContainerDied","Data":"72a191de19230a8eb7016ed473068591e013b18df50e2220b4f14e0607b5df58"} Jul 18 12:29:13.601266 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:29:13.601230 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-f133b3-openshift-default-7c787bdc9f-ccdbc" event={"ID":"2eede87b-bd0d-4e01-962b-583b7c7522cd","Type":"ContainerDied","Data":"07c5ed0ba1252673cce0c661ae40dc5822dd74fcfd2b151ee56d177205d8059d"} Jul 18 12:29:13.601266 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:29:13.601253 2585 scope.go:117] "RemoveContainer" containerID="72a191de19230a8eb7016ed473068591e013b18df50e2220b4f14e0607b5df58" Jul 18 12:29:13.609963 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:29:13.609939 2585 scope.go:117] "RemoveContainer" containerID="e14206932686cc084d2c1a44b1ef18bc7b5f287976f0826d4530eadea0630d21" Jul 18 12:29:13.618291 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:29:13.618275 2585 scope.go:117] "RemoveContainer" containerID="72a191de19230a8eb7016ed473068591e013b18df50e2220b4f14e0607b5df58" Jul 18 12:29:13.618591 ip-10-0-136-240 kubenswrapper[2585]: E0718 12:29:13.618570 2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"72a191de19230a8eb7016ed473068591e013b18df50e2220b4f14e0607b5df58\": container with ID starting with 72a191de19230a8eb7016ed473068591e013b18df50e2220b4f14e0607b5df58 not found: ID does not exist" containerID="72a191de19230a8eb7016ed473068591e013b18df50e2220b4f14e0607b5df58" Jul 18 12:29:13.618673 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:29:13.618604 2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"72a191de19230a8eb7016ed473068591e013b18df50e2220b4f14e0607b5df58"} err="failed to get container status \"72a191de19230a8eb7016ed473068591e013b18df50e2220b4f14e0607b5df58\": rpc error: code = NotFound desc = could not find container \"72a191de19230a8eb7016ed473068591e013b18df50e2220b4f14e0607b5df58\": container with ID starting with 72a191de19230a8eb7016ed473068591e013b18df50e2220b4f14e0607b5df58 not found: ID does not exist" Jul 18 12:29:13.618673 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:29:13.618628 2585 scope.go:117] "RemoveContainer" containerID="e14206932686cc084d2c1a44b1ef18bc7b5f287976f0826d4530eadea0630d21" Jul 18 12:29:13.618673 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:29:13.618664 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-f133b3-openshift-default-7c787bdc9f-ccdbc"] Jul 18 12:29:13.618902 ip-10-0-136-240 kubenswrapper[2585]: E0718 12:29:13.618885 2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e14206932686cc084d2c1a44b1ef18bc7b5f287976f0826d4530eadea0630d21\": container with ID starting with e14206932686cc084d2c1a44b1ef18bc7b5f287976f0826d4530eadea0630d21 not found: ID does not exist" containerID="e14206932686cc084d2c1a44b1ef18bc7b5f287976f0826d4530eadea0630d21" Jul 18 12:29:13.618944 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:29:13.618908 2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e14206932686cc084d2c1a44b1ef18bc7b5f287976f0826d4530eadea0630d21"} err="failed to get container status \"e14206932686cc084d2c1a44b1ef18bc7b5f287976f0826d4530eadea0630d21\": rpc error: code = NotFound desc = could not find container \"e14206932686cc084d2c1a44b1ef18bc7b5f287976f0826d4530eadea0630d21\": container with ID starting with e14206932686cc084d2c1a44b1ef18bc7b5f287976f0826d4530eadea0630d21 not found: ID does not exist" Jul 18 12:29:13.628013 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:29:13.627994 2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-f133b3-openshift-default-7c787bdc9f-ccdbc"] Jul 18 12:29:15.469765 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:29:15.469727 2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2eede87b-bd0d-4e01-962b-583b7c7522cd" path="/var/lib/kubelet/pods/2eede87b-bd0d-4e01-962b-583b7c7522cd/volumes" Jul 18 12:29:16.738097 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:29:16.738066 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-b7127d-openshift-default-7bfbdf8dc-p6g5g"] Jul 18 12:29:16.738505 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:29:16.738492 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2eede87b-bd0d-4e01-962b-583b7c7522cd" containerName="istio-proxy" Jul 18 12:29:16.738505 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:29:16.738506 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="2eede87b-bd0d-4e01-962b-583b7c7522cd" containerName="istio-proxy" Jul 18 12:29:16.738622 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:29:16.738586 2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="2eede87b-bd0d-4e01-962b-583b7c7522cd" containerName="istio-proxy" Jul 18 12:29:16.738622 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:29:16.738598 2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="2eede87b-bd0d-4e01-962b-583b7c7522cd" containerName="istio-proxy" Jul 18 12:29:16.738622 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:29:16.738606 2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="f2a380d1-804a-4e1c-9853-2773e962a49c" containerName="istio-proxy" Jul 18 12:29:16.743257 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:29:16.743236 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-b7127d-openshift-default-7bfbdf8dc-p6g5g" Jul 18 12:29:16.745612 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:29:16.745594 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-b7127d-openshift-default-dockercfg-bczkx\"" Jul 18 12:29:16.758192 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:29:16.758169 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-b7127d-openshift-default-7bfbdf8dc-p6g5g"] Jul 18 12:29:16.850682 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:29:16.850654 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tvbln\" (UniqueName: \"kubernetes.io/projected/d44a53a4-d12a-4bf6-8601-d53d0818ea0b-kube-api-access-tvbln\") pod \"e2e-api-b-b7127d-openshift-default-7bfbdf8dc-p6g5g\" (UID: \"d44a53a4-d12a-4bf6-8601-d53d0818ea0b\") " pod="openshift-ingress/e2e-api-b-b7127d-openshift-default-7bfbdf8dc-p6g5g" Jul 18 12:29:16.850808 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:29:16.850693 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d44a53a4-d12a-4bf6-8601-d53d0818ea0b-istio-envoy\") pod \"e2e-api-b-b7127d-openshift-default-7bfbdf8dc-p6g5g\" (UID: \"d44a53a4-d12a-4bf6-8601-d53d0818ea0b\") " pod="openshift-ingress/e2e-api-b-b7127d-openshift-default-7bfbdf8dc-p6g5g" Jul 18 12:29:16.850808 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:29:16.850717 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d44a53a4-d12a-4bf6-8601-d53d0818ea0b-workload-certs\") pod \"e2e-api-b-b7127d-openshift-default-7bfbdf8dc-p6g5g\" (UID: \"d44a53a4-d12a-4bf6-8601-d53d0818ea0b\") " pod="openshift-ingress/e2e-api-b-b7127d-openshift-default-7bfbdf8dc-p6g5g" Jul 18 12:29:16.850808 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:29:16.850740 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d44a53a4-d12a-4bf6-8601-d53d0818ea0b-istio-podinfo\") pod \"e2e-api-b-b7127d-openshift-default-7bfbdf8dc-p6g5g\" (UID: \"d44a53a4-d12a-4bf6-8601-d53d0818ea0b\") " pod="openshift-ingress/e2e-api-b-b7127d-openshift-default-7bfbdf8dc-p6g5g" Jul 18 12:29:16.850808 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:29:16.850778 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d44a53a4-d12a-4bf6-8601-d53d0818ea0b-credential-socket\") pod \"e2e-api-b-b7127d-openshift-default-7bfbdf8dc-p6g5g\" (UID: \"d44a53a4-d12a-4bf6-8601-d53d0818ea0b\") " pod="openshift-ingress/e2e-api-b-b7127d-openshift-default-7bfbdf8dc-p6g5g" Jul 18 12:29:16.850946 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:29:16.850825 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d44a53a4-d12a-4bf6-8601-d53d0818ea0b-workload-socket\") pod \"e2e-api-b-b7127d-openshift-default-7bfbdf8dc-p6g5g\" (UID: \"d44a53a4-d12a-4bf6-8601-d53d0818ea0b\") " pod="openshift-ingress/e2e-api-b-b7127d-openshift-default-7bfbdf8dc-p6g5g" Jul 18 12:29:16.850946 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:29:16.850861 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d44a53a4-d12a-4bf6-8601-d53d0818ea0b-istio-data\") pod \"e2e-api-b-b7127d-openshift-default-7bfbdf8dc-p6g5g\" (UID: \"d44a53a4-d12a-4bf6-8601-d53d0818ea0b\") " pod="openshift-ingress/e2e-api-b-b7127d-openshift-default-7bfbdf8dc-p6g5g" Jul 18 12:29:16.850946 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:29:16.850880 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d44a53a4-d12a-4bf6-8601-d53d0818ea0b-istio-token\") pod \"e2e-api-b-b7127d-openshift-default-7bfbdf8dc-p6g5g\" (UID: \"d44a53a4-d12a-4bf6-8601-d53d0818ea0b\") " pod="openshift-ingress/e2e-api-b-b7127d-openshift-default-7bfbdf8dc-p6g5g" Jul 18 12:29:16.851044 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:29:16.850938 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d44a53a4-d12a-4bf6-8601-d53d0818ea0b-istiod-ca-cert\") pod \"e2e-api-b-b7127d-openshift-default-7bfbdf8dc-p6g5g\" (UID: \"d44a53a4-d12a-4bf6-8601-d53d0818ea0b\") " pod="openshift-ingress/e2e-api-b-b7127d-openshift-default-7bfbdf8dc-p6g5g" Jul 18 12:29:16.952443 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:29:16.952403 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d44a53a4-d12a-4bf6-8601-d53d0818ea0b-istiod-ca-cert\") pod \"e2e-api-b-b7127d-openshift-default-7bfbdf8dc-p6g5g\" (UID: \"d44a53a4-d12a-4bf6-8601-d53d0818ea0b\") " pod="openshift-ingress/e2e-api-b-b7127d-openshift-default-7bfbdf8dc-p6g5g" Jul 18 12:29:16.952698 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:29:16.952450 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tvbln\" (UniqueName: \"kubernetes.io/projected/d44a53a4-d12a-4bf6-8601-d53d0818ea0b-kube-api-access-tvbln\") pod \"e2e-api-b-b7127d-openshift-default-7bfbdf8dc-p6g5g\" (UID: \"d44a53a4-d12a-4bf6-8601-d53d0818ea0b\") " pod="openshift-ingress/e2e-api-b-b7127d-openshift-default-7bfbdf8dc-p6g5g" Jul 18 12:29:16.952698 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:29:16.952498 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d44a53a4-d12a-4bf6-8601-d53d0818ea0b-istio-envoy\") pod \"e2e-api-b-b7127d-openshift-default-7bfbdf8dc-p6g5g\" (UID: \"d44a53a4-d12a-4bf6-8601-d53d0818ea0b\") " pod="openshift-ingress/e2e-api-b-b7127d-openshift-default-7bfbdf8dc-p6g5g" Jul 18 12:29:16.952698 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:29:16.952529 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d44a53a4-d12a-4bf6-8601-d53d0818ea0b-workload-certs\") pod \"e2e-api-b-b7127d-openshift-default-7bfbdf8dc-p6g5g\" (UID: \"d44a53a4-d12a-4bf6-8601-d53d0818ea0b\") " pod="openshift-ingress/e2e-api-b-b7127d-openshift-default-7bfbdf8dc-p6g5g" Jul 18 12:29:16.952698 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:29:16.952581 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d44a53a4-d12a-4bf6-8601-d53d0818ea0b-istio-podinfo\") pod \"e2e-api-b-b7127d-openshift-default-7bfbdf8dc-p6g5g\" (UID: \"d44a53a4-d12a-4bf6-8601-d53d0818ea0b\") " pod="openshift-ingress/e2e-api-b-b7127d-openshift-default-7bfbdf8dc-p6g5g" Jul 18 12:29:16.952920 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:29:16.952707 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d44a53a4-d12a-4bf6-8601-d53d0818ea0b-credential-socket\") pod \"e2e-api-b-b7127d-openshift-default-7bfbdf8dc-p6g5g\" (UID: \"d44a53a4-d12a-4bf6-8601-d53d0818ea0b\") " pod="openshift-ingress/e2e-api-b-b7127d-openshift-default-7bfbdf8dc-p6g5g" Jul 18 12:29:16.952920 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:29:16.952821 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d44a53a4-d12a-4bf6-8601-d53d0818ea0b-workload-socket\") pod \"e2e-api-b-b7127d-openshift-default-7bfbdf8dc-p6g5g\" (UID: \"d44a53a4-d12a-4bf6-8601-d53d0818ea0b\") " pod="openshift-ingress/e2e-api-b-b7127d-openshift-default-7bfbdf8dc-p6g5g" Jul 18 12:29:16.952920 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:29:16.952915 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d44a53a4-d12a-4bf6-8601-d53d0818ea0b-istio-data\") pod \"e2e-api-b-b7127d-openshift-default-7bfbdf8dc-p6g5g\" (UID: \"d44a53a4-d12a-4bf6-8601-d53d0818ea0b\") " pod="openshift-ingress/e2e-api-b-b7127d-openshift-default-7bfbdf8dc-p6g5g" Jul 18 12:29:16.953074 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:29:16.952951 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d44a53a4-d12a-4bf6-8601-d53d0818ea0b-istio-token\") pod \"e2e-api-b-b7127d-openshift-default-7bfbdf8dc-p6g5g\" (UID: \"d44a53a4-d12a-4bf6-8601-d53d0818ea0b\") " pod="openshift-ingress/e2e-api-b-b7127d-openshift-default-7bfbdf8dc-p6g5g" Jul 18 12:29:16.953074 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:29:16.953041 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d44a53a4-d12a-4bf6-8601-d53d0818ea0b-credential-socket\") pod \"e2e-api-b-b7127d-openshift-default-7bfbdf8dc-p6g5g\" (UID: \"d44a53a4-d12a-4bf6-8601-d53d0818ea0b\") " pod="openshift-ingress/e2e-api-b-b7127d-openshift-default-7bfbdf8dc-p6g5g" Jul 18 12:29:16.953074 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:29:16.953067 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d44a53a4-d12a-4bf6-8601-d53d0818ea0b-workload-certs\") pod \"e2e-api-b-b7127d-openshift-default-7bfbdf8dc-p6g5g\" (UID: \"d44a53a4-d12a-4bf6-8601-d53d0818ea0b\") " pod="openshift-ingress/e2e-api-b-b7127d-openshift-default-7bfbdf8dc-p6g5g" Jul 18 12:29:16.953217 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:29:16.953196 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d44a53a4-d12a-4bf6-8601-d53d0818ea0b-workload-socket\") pod \"e2e-api-b-b7127d-openshift-default-7bfbdf8dc-p6g5g\" (UID: \"d44a53a4-d12a-4bf6-8601-d53d0818ea0b\") " pod="openshift-ingress/e2e-api-b-b7127d-openshift-default-7bfbdf8dc-p6g5g" Jul 18 12:29:16.953305 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:29:16.953278 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d44a53a4-d12a-4bf6-8601-d53d0818ea0b-istiod-ca-cert\") pod \"e2e-api-b-b7127d-openshift-default-7bfbdf8dc-p6g5g\" (UID: \"d44a53a4-d12a-4bf6-8601-d53d0818ea0b\") " pod="openshift-ingress/e2e-api-b-b7127d-openshift-default-7bfbdf8dc-p6g5g" Jul 18 12:29:16.953305 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:29:16.953288 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d44a53a4-d12a-4bf6-8601-d53d0818ea0b-istio-data\") pod \"e2e-api-b-b7127d-openshift-default-7bfbdf8dc-p6g5g\" (UID: \"d44a53a4-d12a-4bf6-8601-d53d0818ea0b\") " pod="openshift-ingress/e2e-api-b-b7127d-openshift-default-7bfbdf8dc-p6g5g" Jul 18 12:29:16.955200 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:29:16.955178 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d44a53a4-d12a-4bf6-8601-d53d0818ea0b-istio-envoy\") pod \"e2e-api-b-b7127d-openshift-default-7bfbdf8dc-p6g5g\" (UID: \"d44a53a4-d12a-4bf6-8601-d53d0818ea0b\") " pod="openshift-ingress/e2e-api-b-b7127d-openshift-default-7bfbdf8dc-p6g5g" Jul 18 12:29:16.955444 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:29:16.955425 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d44a53a4-d12a-4bf6-8601-d53d0818ea0b-istio-podinfo\") pod \"e2e-api-b-b7127d-openshift-default-7bfbdf8dc-p6g5g\" (UID: \"d44a53a4-d12a-4bf6-8601-d53d0818ea0b\") " pod="openshift-ingress/e2e-api-b-b7127d-openshift-default-7bfbdf8dc-p6g5g" Jul 18 12:29:16.961970 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:29:16.961949 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d44a53a4-d12a-4bf6-8601-d53d0818ea0b-istio-token\") pod \"e2e-api-b-b7127d-openshift-default-7bfbdf8dc-p6g5g\" (UID: \"d44a53a4-d12a-4bf6-8601-d53d0818ea0b\") " pod="openshift-ingress/e2e-api-b-b7127d-openshift-default-7bfbdf8dc-p6g5g" Jul 18 12:29:16.962051 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:29:16.962036 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tvbln\" (UniqueName: \"kubernetes.io/projected/d44a53a4-d12a-4bf6-8601-d53d0818ea0b-kube-api-access-tvbln\") pod \"e2e-api-b-b7127d-openshift-default-7bfbdf8dc-p6g5g\" (UID: \"d44a53a4-d12a-4bf6-8601-d53d0818ea0b\") " pod="openshift-ingress/e2e-api-b-b7127d-openshift-default-7bfbdf8dc-p6g5g" Jul 18 12:29:17.054073 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:29:17.053996 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-b7127d-openshift-default-7bfbdf8dc-p6g5g" Jul 18 12:29:17.250957 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:29:17.250917 2585 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 18 12:29:17.251101 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:29:17.250995 2585 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 18 12:29:17.251101 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:29:17.251039 2585 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 18 12:29:17.252057 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:29:17.252029 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-b7127d-openshift-default-7bfbdf8dc-p6g5g"] Jul 18 12:29:17.623151 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:29:17.623012 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-b7127d-openshift-default-7bfbdf8dc-p6g5g" event={"ID":"d44a53a4-d12a-4bf6-8601-d53d0818ea0b","Type":"ContainerStarted","Data":"0e896a47ccf785af9109b68628935664a99615c5af457aae0fc5d33b1c2d5e3f"} Jul 18 12:29:17.623151 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:29:17.623073 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-b7127d-openshift-default-7bfbdf8dc-p6g5g" event={"ID":"d44a53a4-d12a-4bf6-8601-d53d0818ea0b","Type":"ContainerStarted","Data":"49759bd8294666e834eb11c170c0c8d94236b03884984b6b1bd72bad70514510"} Jul 18 12:29:17.649820 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:29:17.649773 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-b7127d-openshift-default-7bfbdf8dc-p6g5g" podStartSLOduration=1.649757804 podStartE2EDuration="1.649757804s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-18 12:29:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-18 12:29:17.648187833 +0000 UTC m=+2424.728394215" watchObservedRunningTime="2026-07-18 12:29:17.649757804 +0000 UTC m=+2424.729964202" Jul 18 12:29:18.054630 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:29:18.054594 2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-b7127d-openshift-default-7bfbdf8dc-p6g5g" Jul 18 12:29:18.059911 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:29:18.059888 2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-b7127d-openshift-default-7bfbdf8dc-p6g5g" Jul 18 12:29:18.628517 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:29:18.628490 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-b7127d-openshift-default-7bfbdf8dc-p6g5g" Jul 18 12:29:18.629352 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:29:18.629335 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-b7127d-openshift-default-7bfbdf8dc-p6g5g" Jul 18 12:29:26.623696 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:29:26.623663 2585 patch_prober.go:28] interesting pod/e2e-api-a-07813a-openshift-default-55b5b54787-lqmzm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Jul 18 12:29:26.624079 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:29:26.623728 2585 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-07813a-openshift-default-55b5b54787-lqmzm" podUID="cf0cb8e4-36b7-4cb5-b4c1-09f746e04491" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Jul 18 12:29:27.679995 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:29:27.679906 2585 generic.go:356] "Generic (PLEG): container finished" podID="cf0cb8e4-36b7-4cb5-b4c1-09f746e04491" containerID="68913856927afa3fc43589347cf16e511dc6961433d1fabb279a9e06abcba643" exitCode=0 Jul 18 12:29:27.679995 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:29:27.679978 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-07813a-openshift-default-55b5b54787-lqmzm" event={"ID":"cf0cb8e4-36b7-4cb5-b4c1-09f746e04491","Type":"ContainerDied","Data":"68913856927afa3fc43589347cf16e511dc6961433d1fabb279a9e06abcba643"} Jul 18 12:29:27.680394 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:29:27.680366 2585 scope.go:117] "RemoveContainer" containerID="68913856927afa3fc43589347cf16e511dc6961433d1fabb279a9e06abcba643" Jul 18 12:29:27.681442 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:29:27.681417 2585 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 18 12:29:27.681506 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:29:27.681483 2585 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 18 12:29:27.681544 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:29:27.681513 2585 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 18 12:29:28.694242 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:29:28.693199 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-07813a-openshift-default-55b5b54787-lqmzm" event={"ID":"cf0cb8e4-36b7-4cb5-b4c1-09f746e04491","Type":"ContainerStarted","Data":"0fd1a582e34855d98f2d26bea4eee440353b7e2d4778c72e34186c24e15f9855"} Jul 18 12:29:29.306198 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:29:29.306162 2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-07813a-openshift-default-55b5b54787-lqmzm" Jul 18 12:29:29.306619 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:29:29.306593 2585 patch_prober.go:28] interesting pod/e2e-api-a-07813a-openshift-default-55b5b54787-lqmzm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" start-of-body= Jul 18 12:29:29.306810 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:29:29.306782 2585 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-07813a-openshift-default-55b5b54787-lqmzm" podUID="cf0cb8e4-36b7-4cb5-b4c1-09f746e04491" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" Jul 18 12:29:30.306405 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:29:30.306368 2585 patch_prober.go:28] interesting pod/e2e-api-a-07813a-openshift-default-55b5b54787-lqmzm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" start-of-body= Jul 18 12:29:30.307022 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:29:30.306990 2585 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-07813a-openshift-default-55b5b54787-lqmzm" podUID="cf0cb8e4-36b7-4cb5-b4c1-09f746e04491" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" Jul 18 12:29:31.306720 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:29:31.306681 2585 patch_prober.go:28] interesting pod/e2e-api-a-07813a-openshift-default-55b5b54787-lqmzm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" start-of-body= Jul 18 12:29:31.307725 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:29:31.307307 2585 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-07813a-openshift-default-55b5b54787-lqmzm" podUID="cf0cb8e4-36b7-4cb5-b4c1-09f746e04491" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" Jul 18 12:29:32.306202 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:29:32.306155 2585 patch_prober.go:28] interesting pod/e2e-api-a-07813a-openshift-default-55b5b54787-lqmzm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" start-of-body= Jul 18 12:29:32.306421 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:29:32.306229 2585 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-07813a-openshift-default-55b5b54787-lqmzm" podUID="cf0cb8e4-36b7-4cb5-b4c1-09f746e04491" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" Jul 18 12:29:33.310222 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:29:33.310185 2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-07813a-openshift-default-55b5b54787-lqmzm" Jul 18 12:29:33.310771 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:29:33.310744 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-07813a-openshift-default-55b5b54787-lqmzm" Jul 18 12:29:33.311298 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:29:33.311279 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-07813a-openshift-default-55b5b54787-lqmzm" Jul 18 12:29:34.670702 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:29:34.670660 2585 patch_prober.go:28] interesting pod/e2e-api-b-b7127d-openshift-default-7bfbdf8dc-p6g5g container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.55:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Jul 18 12:29:34.671134 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:29:34.670725 2585 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-b7127d-openshift-default-7bfbdf8dc-p6g5g" podUID="d44a53a4-d12a-4bf6-8601-d53d0818ea0b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.55:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Jul 18 12:29:35.738823 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:29:35.738761 2585 generic.go:356] "Generic (PLEG): container finished" podID="d44a53a4-d12a-4bf6-8601-d53d0818ea0b" containerID="0e896a47ccf785af9109b68628935664a99615c5af457aae0fc5d33b1c2d5e3f" exitCode=0 Jul 18 12:29:35.739144 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:29:35.738840 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-b7127d-openshift-default-7bfbdf8dc-p6g5g" event={"ID":"d44a53a4-d12a-4bf6-8601-d53d0818ea0b","Type":"ContainerDied","Data":"0e896a47ccf785af9109b68628935664a99615c5af457aae0fc5d33b1c2d5e3f"} Jul 18 12:29:35.739236 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:29:35.739222 2585 scope.go:117] "RemoveContainer" containerID="0e896a47ccf785af9109b68628935664a99615c5af457aae0fc5d33b1c2d5e3f" Jul 18 12:29:35.740273 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:29:35.740241 2585 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 18 12:29:35.740365 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:29:35.740313 2585 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 18 12:29:35.740365 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:29:35.740350 2585 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 18 12:29:36.745160 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:29:36.745126 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-b7127d-openshift-default-7bfbdf8dc-p6g5g" event={"ID":"d44a53a4-d12a-4bf6-8601-d53d0818ea0b","Type":"ContainerStarted","Data":"e7d6e4a748155512ea3134db68b6b308e9ed6d8bebda446860153b39a457732b"} Jul 18 12:29:37.055067 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:29:37.054984 2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-b7127d-openshift-default-7bfbdf8dc-p6g5g" Jul 18 12:29:37.055218 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:29:37.055192 2585 patch_prober.go:28] interesting pod/e2e-api-b-b7127d-openshift-default-7bfbdf8dc-p6g5g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" start-of-body= Jul 18 12:29:37.055259 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:29:37.055235 2585 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-b7127d-openshift-default-7bfbdf8dc-p6g5g" podUID="d44a53a4-d12a-4bf6-8601-d53d0818ea0b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" Jul 18 12:29:38.054457 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:29:38.054420 2585 patch_prober.go:28] interesting pod/e2e-api-b-b7127d-openshift-default-7bfbdf8dc-p6g5g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" start-of-body= Jul 18 12:29:38.054896 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:29:38.054477 2585 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-b7127d-openshift-default-7bfbdf8dc-p6g5g" podUID="d44a53a4-d12a-4bf6-8601-d53d0818ea0b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" Jul 18 12:29:39.055302 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:29:39.055263 2585 patch_prober.go:28] interesting pod/e2e-api-b-b7127d-openshift-default-7bfbdf8dc-p6g5g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" start-of-body= Jul 18 12:29:39.055841 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:29:39.055333 2585 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-b7127d-openshift-default-7bfbdf8dc-p6g5g" podUID="d44a53a4-d12a-4bf6-8601-d53d0818ea0b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" Jul 18 12:29:40.054708 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:29:40.054672 2585 patch_prober.go:28] interesting pod/e2e-api-b-b7127d-openshift-default-7bfbdf8dc-p6g5g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" start-of-body= Jul 18 12:29:40.054900 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:29:40.054726 2585 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-b7127d-openshift-default-7bfbdf8dc-p6g5g" podUID="d44a53a4-d12a-4bf6-8601-d53d0818ea0b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" Jul 18 12:29:41.057994 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:29:41.057965 2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-b7127d-openshift-default-7bfbdf8dc-p6g5g" Jul 18 12:29:41.058459 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:29:41.058238 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-b7127d-openshift-default-7bfbdf8dc-p6g5g" Jul 18 12:29:41.059026 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:29:41.059005 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-b7127d-openshift-default-7bfbdf8dc-p6g5g" Jul 18 12:29:58.670347 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:29:58.670314 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-07813a-openshift-default-55b5b54787-lqmzm"] Jul 18 12:29:58.670842 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:29:58.670573 2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-07813a-openshift-default-55b5b54787-lqmzm" podUID="cf0cb8e4-36b7-4cb5-b4c1-09f746e04491" containerName="istio-proxy" containerID="cri-o://0fd1a582e34855d98f2d26bea4eee440353b7e2d4778c72e34186c24e15f9855" gracePeriod=30 Jul 18 12:30:03.312721 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:30:03.312685 2585 patch_prober.go:28] interesting pod/e2e-api-a-07813a-openshift-default-55b5b54787-lqmzm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=upstream connect error or disconnect/reset before headers. reset reason: remote connection failure, transport failure reason: delayed connect error: Connection refused Jul 18 12:30:03.313260 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:30:03.312744 2585 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-07813a-openshift-default-55b5b54787-lqmzm" podUID="cf0cb8e4-36b7-4cb5-b4c1-09f746e04491" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 18 12:30:03.892796 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:30:03.892729 2585 generic.go:356] "Generic (PLEG): container finished" podID="cf0cb8e4-36b7-4cb5-b4c1-09f746e04491" containerID="0fd1a582e34855d98f2d26bea4eee440353b7e2d4778c72e34186c24e15f9855" exitCode=0 Jul 18 12:30:03.892796 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:30:03.892762 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-07813a-openshift-default-55b5b54787-lqmzm" event={"ID":"cf0cb8e4-36b7-4cb5-b4c1-09f746e04491","Type":"ContainerDied","Data":"0fd1a582e34855d98f2d26bea4eee440353b7e2d4778c72e34186c24e15f9855"} Jul 18 12:30:03.893098 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:30:03.892837 2585 scope.go:117] "RemoveContainer" containerID="68913856927afa3fc43589347cf16e511dc6961433d1fabb279a9e06abcba643" Jul 18 12:30:03.927846 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:30:03.927820 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-07813a-openshift-default-55b5b54787-lqmzm" Jul 18 12:30:04.023894 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:30:04.023861 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cf0cb8e4-36b7-4cb5-b4c1-09f746e04491-istio-data\") pod \"cf0cb8e4-36b7-4cb5-b4c1-09f746e04491\" (UID: \"cf0cb8e4-36b7-4cb5-b4c1-09f746e04491\") " Jul 18 12:30:04.024094 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:30:04.023922 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cf0cb8e4-36b7-4cb5-b4c1-09f746e04491-istio-token\") pod \"cf0cb8e4-36b7-4cb5-b4c1-09f746e04491\" (UID: \"cf0cb8e4-36b7-4cb5-b4c1-09f746e04491\") " Jul 18 12:30:04.024094 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:30:04.023940 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf0cb8e4-36b7-4cb5-b4c1-09f746e04491-credential-socket\") pod \"cf0cb8e4-36b7-4cb5-b4c1-09f746e04491\" (UID: \"cf0cb8e4-36b7-4cb5-b4c1-09f746e04491\") " Jul 18 12:30:04.024094 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:30:04.023962 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cf0cb8e4-36b7-4cb5-b4c1-09f746e04491-istio-envoy\") pod \"cf0cb8e4-36b7-4cb5-b4c1-09f746e04491\" (UID: \"cf0cb8e4-36b7-4cb5-b4c1-09f746e04491\") " Jul 18 12:30:04.024094 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:30:04.023983 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cf0cb8e4-36b7-4cb5-b4c1-09f746e04491-istiod-ca-cert\") pod \"cf0cb8e4-36b7-4cb5-b4c1-09f746e04491\" (UID: \"cf0cb8e4-36b7-4cb5-b4c1-09f746e04491\") " Jul 18 12:30:04.024094 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:30:04.024012 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hpj97\" (UniqueName: \"kubernetes.io/projected/cf0cb8e4-36b7-4cb5-b4c1-09f746e04491-kube-api-access-hpj97\") pod \"cf0cb8e4-36b7-4cb5-b4c1-09f746e04491\" (UID: \"cf0cb8e4-36b7-4cb5-b4c1-09f746e04491\") " Jul 18 12:30:04.024094 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:30:04.024028 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cf0cb8e4-36b7-4cb5-b4c1-09f746e04491-workload-certs\") pod \"cf0cb8e4-36b7-4cb5-b4c1-09f746e04491\" (UID: \"cf0cb8e4-36b7-4cb5-b4c1-09f746e04491\") " Jul 18 12:30:04.024412 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:30:04.024139 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf0cb8e4-36b7-4cb5-b4c1-09f746e04491-workload-socket\") pod \"cf0cb8e4-36b7-4cb5-b4c1-09f746e04491\" (UID: \"cf0cb8e4-36b7-4cb5-b4c1-09f746e04491\") " Jul 18 12:30:04.024412 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:30:04.024164 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cf0cb8e4-36b7-4cb5-b4c1-09f746e04491-istio-podinfo\") pod \"cf0cb8e4-36b7-4cb5-b4c1-09f746e04491\" (UID: \"cf0cb8e4-36b7-4cb5-b4c1-09f746e04491\") " Jul 18 12:30:04.024412 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:30:04.024198 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cf0cb8e4-36b7-4cb5-b4c1-09f746e04491-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "cf0cb8e4-36b7-4cb5-b4c1-09f746e04491" (UID: "cf0cb8e4-36b7-4cb5-b4c1-09f746e04491"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 18 12:30:04.024633 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:30:04.024431 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cf0cb8e4-36b7-4cb5-b4c1-09f746e04491-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "cf0cb8e4-36b7-4cb5-b4c1-09f746e04491" (UID: "cf0cb8e4-36b7-4cb5-b4c1-09f746e04491"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 18 12:30:04.024633 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:30:04.024484 2585 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf0cb8e4-36b7-4cb5-b4c1-09f746e04491-credential-socket\") on node \"ip-10-0-136-240.ec2.internal\" DevicePath \"\"" Jul 18 12:30:04.024633 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:30:04.024486 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cf0cb8e4-36b7-4cb5-b4c1-09f746e04491-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "cf0cb8e4-36b7-4cb5-b4c1-09f746e04491" (UID: "cf0cb8e4-36b7-4cb5-b4c1-09f746e04491"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 18 12:30:04.024791 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:30:04.024767 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cf0cb8e4-36b7-4cb5-b4c1-09f746e04491-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "cf0cb8e4-36b7-4cb5-b4c1-09f746e04491" (UID: "cf0cb8e4-36b7-4cb5-b4c1-09f746e04491"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 18 12:30:04.025135 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:30:04.025104 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cf0cb8e4-36b7-4cb5-b4c1-09f746e04491-istio-data" (OuterVolumeSpecName: "istio-data") pod "cf0cb8e4-36b7-4cb5-b4c1-09f746e04491" (UID: "cf0cb8e4-36b7-4cb5-b4c1-09f746e04491"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 18 12:30:04.026629 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:30:04.026598 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/cf0cb8e4-36b7-4cb5-b4c1-09f746e04491-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "cf0cb8e4-36b7-4cb5-b4c1-09f746e04491" (UID: "cf0cb8e4-36b7-4cb5-b4c1-09f746e04491"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 18 12:30:04.026629 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:30:04.026610 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cf0cb8e4-36b7-4cb5-b4c1-09f746e04491-istio-token" (OuterVolumeSpecName: "istio-token") pod "cf0cb8e4-36b7-4cb5-b4c1-09f746e04491" (UID: "cf0cb8e4-36b7-4cb5-b4c1-09f746e04491"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 18 12:30:04.026629 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:30:04.026596 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cf0cb8e4-36b7-4cb5-b4c1-09f746e04491-kube-api-access-hpj97" (OuterVolumeSpecName: "kube-api-access-hpj97") pod "cf0cb8e4-36b7-4cb5-b4c1-09f746e04491" (UID: "cf0cb8e4-36b7-4cb5-b4c1-09f746e04491"). InnerVolumeSpecName "kube-api-access-hpj97". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 18 12:30:04.026796 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:30:04.026686 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cf0cb8e4-36b7-4cb5-b4c1-09f746e04491-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "cf0cb8e4-36b7-4cb5-b4c1-09f746e04491" (UID: "cf0cb8e4-36b7-4cb5-b4c1-09f746e04491"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 18 12:30:04.125598 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:30:04.125524 2585 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf0cb8e4-36b7-4cb5-b4c1-09f746e04491-workload-socket\") on node \"ip-10-0-136-240.ec2.internal\" DevicePath \"\"" Jul 18 12:30:04.125598 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:30:04.125592 2585 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cf0cb8e4-36b7-4cb5-b4c1-09f746e04491-istio-podinfo\") on node \"ip-10-0-136-240.ec2.internal\" DevicePath \"\"" Jul 18 12:30:04.125598 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:30:04.125602 2585 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cf0cb8e4-36b7-4cb5-b4c1-09f746e04491-istio-data\") on node \"ip-10-0-136-240.ec2.internal\" DevicePath \"\"" Jul 18 12:30:04.125598 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:30:04.125611 2585 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cf0cb8e4-36b7-4cb5-b4c1-09f746e04491-istio-token\") on node \"ip-10-0-136-240.ec2.internal\" DevicePath \"\"" Jul 18 12:30:04.125878 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:30:04.125618 2585 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cf0cb8e4-36b7-4cb5-b4c1-09f746e04491-istio-envoy\") on node \"ip-10-0-136-240.ec2.internal\" DevicePath \"\"" Jul 18 12:30:04.125878 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:30:04.125626 2585 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cf0cb8e4-36b7-4cb5-b4c1-09f746e04491-istiod-ca-cert\") on node \"ip-10-0-136-240.ec2.internal\" DevicePath \"\"" Jul 18 12:30:04.125878 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:30:04.125633 2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hpj97\" (UniqueName: \"kubernetes.io/projected/cf0cb8e4-36b7-4cb5-b4c1-09f746e04491-kube-api-access-hpj97\") on node \"ip-10-0-136-240.ec2.internal\" DevicePath \"\"" Jul 18 12:30:04.125878 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:30:04.125643 2585 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cf0cb8e4-36b7-4cb5-b4c1-09f746e04491-workload-certs\") on node \"ip-10-0-136-240.ec2.internal\" DevicePath \"\"" Jul 18 12:30:04.899587 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:30:04.899542 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-07813a-openshift-default-55b5b54787-lqmzm" Jul 18 12:30:04.899587 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:30:04.899575 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-07813a-openshift-default-55b5b54787-lqmzm" event={"ID":"cf0cb8e4-36b7-4cb5-b4c1-09f746e04491","Type":"ContainerDied","Data":"a866be9ab39b13652f602ae5e3382fc5e0982d2273ae0144ffb9c7d884eb1d20"} Jul 18 12:30:04.900104 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:30:04.899630 2585 scope.go:117] "RemoveContainer" containerID="0fd1a582e34855d98f2d26bea4eee440353b7e2d4778c72e34186c24e15f9855" Jul 18 12:30:04.923722 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:30:04.923684 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-07813a-openshift-default-55b5b54787-lqmzm"] Jul 18 12:30:04.938649 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:30:04.938620 2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-07813a-openshift-default-55b5b54787-lqmzm"] Jul 18 12:30:05.470397 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:30:05.470352 2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cf0cb8e4-36b7-4cb5-b4c1-09f746e04491" path="/var/lib/kubelet/pods/cf0cb8e4-36b7-4cb5-b4c1-09f746e04491/volumes" Jul 18 12:30:25.687296 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:30:25.687259 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-b7127d-openshift-default-7bfbdf8dc-p6g5g"] Jul 18 12:30:25.687759 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:30:25.687533 2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-b7127d-openshift-default-7bfbdf8dc-p6g5g" podUID="d44a53a4-d12a-4bf6-8601-d53d0818ea0b" containerName="istio-proxy" containerID="cri-o://e7d6e4a748155512ea3134db68b6b308e9ed6d8bebda446860153b39a457732b" gracePeriod=30 Jul 18 12:30:26.060383 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:30:26.060349 2585 patch_prober.go:28] interesting pod/e2e-api-b-b7127d-openshift-default-7bfbdf8dc-p6g5g container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 18 12:30:26.060570 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:30:26.060405 2585 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-b7127d-openshift-default-7bfbdf8dc-p6g5g" podUID="d44a53a4-d12a-4bf6-8601-d53d0818ea0b" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 18 12:30:26.532926 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:30:26.532896 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-7a27b1-openshift-default-78f77667f-8l69z"] Jul 18 12:30:26.533287 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:30:26.533276 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cf0cb8e4-36b7-4cb5-b4c1-09f746e04491" containerName="istio-proxy" Jul 18 12:30:26.533331 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:30:26.533290 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="cf0cb8e4-36b7-4cb5-b4c1-09f746e04491" containerName="istio-proxy" Jul 18 12:30:26.533331 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:30:26.533301 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2eede87b-bd0d-4e01-962b-583b7c7522cd" containerName="istio-proxy" Jul 18 12:30:26.533331 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:30:26.533306 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="2eede87b-bd0d-4e01-962b-583b7c7522cd" containerName="istio-proxy" Jul 18 12:30:26.533428 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:30:26.533378 2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="cf0cb8e4-36b7-4cb5-b4c1-09f746e04491" containerName="istio-proxy" Jul 18 12:30:26.533428 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:30:26.533386 2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="cf0cb8e4-36b7-4cb5-b4c1-09f746e04491" containerName="istio-proxy" Jul 18 12:30:26.538073 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:30:26.538051 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-7a27b1-openshift-default-78f77667f-8l69z" Jul 18 12:30:26.543891 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:30:26.543871 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-7a27b1-openshift-default-dockercfg-pmh87\"" Jul 18 12:30:26.550692 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:30:26.550668 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-7a27b1-openshift-default-78f77667f-8l69z"] Jul 18 12:30:26.636173 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:30:26.636141 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3303f05-8333-424c-8824-8d4cda7ecc62-credential-socket\") pod \"e2e-shared-a-7a27b1-openshift-default-78f77667f-8l69z\" (UID: \"f3303f05-8333-424c-8824-8d4cda7ecc62\") " pod="openshift-ingress/e2e-shared-a-7a27b1-openshift-default-78f77667f-8l69z" Jul 18 12:30:26.636379 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:30:26.636202 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bhrvg\" (UniqueName: \"kubernetes.io/projected/f3303f05-8333-424c-8824-8d4cda7ecc62-kube-api-access-bhrvg\") pod \"e2e-shared-a-7a27b1-openshift-default-78f77667f-8l69z\" (UID: \"f3303f05-8333-424c-8824-8d4cda7ecc62\") " pod="openshift-ingress/e2e-shared-a-7a27b1-openshift-default-78f77667f-8l69z" Jul 18 12:30:26.636379 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:30:26.636357 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f3303f05-8333-424c-8824-8d4cda7ecc62-istio-data\") pod \"e2e-shared-a-7a27b1-openshift-default-78f77667f-8l69z\" (UID: \"f3303f05-8333-424c-8824-8d4cda7ecc62\") " pod="openshift-ingress/e2e-shared-a-7a27b1-openshift-default-78f77667f-8l69z" Jul 18 12:30:26.636517 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:30:26.636462 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f3303f05-8333-424c-8824-8d4cda7ecc62-istio-token\") pod \"e2e-shared-a-7a27b1-openshift-default-78f77667f-8l69z\" (UID: \"f3303f05-8333-424c-8824-8d4cda7ecc62\") " pod="openshift-ingress/e2e-shared-a-7a27b1-openshift-default-78f77667f-8l69z" Jul 18 12:30:26.636517 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:30:26.636489 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3303f05-8333-424c-8824-8d4cda7ecc62-workload-socket\") pod \"e2e-shared-a-7a27b1-openshift-default-78f77667f-8l69z\" (UID: \"f3303f05-8333-424c-8824-8d4cda7ecc62\") " pod="openshift-ingress/e2e-shared-a-7a27b1-openshift-default-78f77667f-8l69z" Jul 18 12:30:26.636517 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:30:26.636506 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f3303f05-8333-424c-8824-8d4cda7ecc62-workload-certs\") pod \"e2e-shared-a-7a27b1-openshift-default-78f77667f-8l69z\" (UID: \"f3303f05-8333-424c-8824-8d4cda7ecc62\") " pod="openshift-ingress/e2e-shared-a-7a27b1-openshift-default-78f77667f-8l69z" Jul 18 12:30:26.636734 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:30:26.636522 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f3303f05-8333-424c-8824-8d4cda7ecc62-istio-podinfo\") pod \"e2e-shared-a-7a27b1-openshift-default-78f77667f-8l69z\" (UID: \"f3303f05-8333-424c-8824-8d4cda7ecc62\") " pod="openshift-ingress/e2e-shared-a-7a27b1-openshift-default-78f77667f-8l69z" Jul 18 12:30:26.636734 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:30:26.636639 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f3303f05-8333-424c-8824-8d4cda7ecc62-istiod-ca-cert\") pod \"e2e-shared-a-7a27b1-openshift-default-78f77667f-8l69z\" (UID: \"f3303f05-8333-424c-8824-8d4cda7ecc62\") " pod="openshift-ingress/e2e-shared-a-7a27b1-openshift-default-78f77667f-8l69z" Jul 18 12:30:26.636734 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:30:26.636678 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f3303f05-8333-424c-8824-8d4cda7ecc62-istio-envoy\") pod \"e2e-shared-a-7a27b1-openshift-default-78f77667f-8l69z\" (UID: \"f3303f05-8333-424c-8824-8d4cda7ecc62\") " pod="openshift-ingress/e2e-shared-a-7a27b1-openshift-default-78f77667f-8l69z" Jul 18 12:30:26.737637 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:30:26.737609 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f3303f05-8333-424c-8824-8d4cda7ecc62-istio-envoy\") pod \"e2e-shared-a-7a27b1-openshift-default-78f77667f-8l69z\" (UID: \"f3303f05-8333-424c-8824-8d4cda7ecc62\") " pod="openshift-ingress/e2e-shared-a-7a27b1-openshift-default-78f77667f-8l69z" Jul 18 12:30:26.738075 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:30:26.737648 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3303f05-8333-424c-8824-8d4cda7ecc62-credential-socket\") pod \"e2e-shared-a-7a27b1-openshift-default-78f77667f-8l69z\" (UID: \"f3303f05-8333-424c-8824-8d4cda7ecc62\") " pod="openshift-ingress/e2e-shared-a-7a27b1-openshift-default-78f77667f-8l69z" Jul 18 12:30:26.738075 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:30:26.737687 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bhrvg\" (UniqueName: \"kubernetes.io/projected/f3303f05-8333-424c-8824-8d4cda7ecc62-kube-api-access-bhrvg\") pod \"e2e-shared-a-7a27b1-openshift-default-78f77667f-8l69z\" (UID: \"f3303f05-8333-424c-8824-8d4cda7ecc62\") " pod="openshift-ingress/e2e-shared-a-7a27b1-openshift-default-78f77667f-8l69z" Jul 18 12:30:26.738075 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:30:26.737720 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f3303f05-8333-424c-8824-8d4cda7ecc62-istio-data\") pod \"e2e-shared-a-7a27b1-openshift-default-78f77667f-8l69z\" (UID: \"f3303f05-8333-424c-8824-8d4cda7ecc62\") " pod="openshift-ingress/e2e-shared-a-7a27b1-openshift-default-78f77667f-8l69z" Jul 18 12:30:26.738075 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:30:26.737857 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f3303f05-8333-424c-8824-8d4cda7ecc62-istio-token\") pod \"e2e-shared-a-7a27b1-openshift-default-78f77667f-8l69z\" (UID: \"f3303f05-8333-424c-8824-8d4cda7ecc62\") " pod="openshift-ingress/e2e-shared-a-7a27b1-openshift-default-78f77667f-8l69z" Jul 18 12:30:26.738075 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:30:26.737912 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3303f05-8333-424c-8824-8d4cda7ecc62-workload-socket\") pod \"e2e-shared-a-7a27b1-openshift-default-78f77667f-8l69z\" (UID: \"f3303f05-8333-424c-8824-8d4cda7ecc62\") " pod="openshift-ingress/e2e-shared-a-7a27b1-openshift-default-78f77667f-8l69z" Jul 18 12:30:26.738075 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:30:26.737943 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f3303f05-8333-424c-8824-8d4cda7ecc62-workload-certs\") pod \"e2e-shared-a-7a27b1-openshift-default-78f77667f-8l69z\" (UID: \"f3303f05-8333-424c-8824-8d4cda7ecc62\") " pod="openshift-ingress/e2e-shared-a-7a27b1-openshift-default-78f77667f-8l69z" Jul 18 12:30:26.738075 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:30:26.737966 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f3303f05-8333-424c-8824-8d4cda7ecc62-istio-podinfo\") pod \"e2e-shared-a-7a27b1-openshift-default-78f77667f-8l69z\" (UID: \"f3303f05-8333-424c-8824-8d4cda7ecc62\") " pod="openshift-ingress/e2e-shared-a-7a27b1-openshift-default-78f77667f-8l69z" Jul 18 12:30:26.738075 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:30:26.738012 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f3303f05-8333-424c-8824-8d4cda7ecc62-istiod-ca-cert\") pod \"e2e-shared-a-7a27b1-openshift-default-78f77667f-8l69z\" (UID: \"f3303f05-8333-424c-8824-8d4cda7ecc62\") " pod="openshift-ingress/e2e-shared-a-7a27b1-openshift-default-78f77667f-8l69z" Jul 18 12:30:26.738656 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:30:26.738124 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3303f05-8333-424c-8824-8d4cda7ecc62-credential-socket\") pod \"e2e-shared-a-7a27b1-openshift-default-78f77667f-8l69z\" (UID: \"f3303f05-8333-424c-8824-8d4cda7ecc62\") " pod="openshift-ingress/e2e-shared-a-7a27b1-openshift-default-78f77667f-8l69z" Jul 18 12:30:26.738656 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:30:26.738142 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f3303f05-8333-424c-8824-8d4cda7ecc62-istio-data\") pod \"e2e-shared-a-7a27b1-openshift-default-78f77667f-8l69z\" (UID: \"f3303f05-8333-424c-8824-8d4cda7ecc62\") " pod="openshift-ingress/e2e-shared-a-7a27b1-openshift-default-78f77667f-8l69z" Jul 18 12:30:26.738656 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:30:26.738327 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3303f05-8333-424c-8824-8d4cda7ecc62-workload-socket\") pod \"e2e-shared-a-7a27b1-openshift-default-78f77667f-8l69z\" (UID: \"f3303f05-8333-424c-8824-8d4cda7ecc62\") " pod="openshift-ingress/e2e-shared-a-7a27b1-openshift-default-78f77667f-8l69z" Jul 18 12:30:26.738656 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:30:26.738369 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f3303f05-8333-424c-8824-8d4cda7ecc62-workload-certs\") pod \"e2e-shared-a-7a27b1-openshift-default-78f77667f-8l69z\" (UID: \"f3303f05-8333-424c-8824-8d4cda7ecc62\") " pod="openshift-ingress/e2e-shared-a-7a27b1-openshift-default-78f77667f-8l69z" Jul 18 12:30:26.738879 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:30:26.738819 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f3303f05-8333-424c-8824-8d4cda7ecc62-istiod-ca-cert\") pod \"e2e-shared-a-7a27b1-openshift-default-78f77667f-8l69z\" (UID: \"f3303f05-8333-424c-8824-8d4cda7ecc62\") " pod="openshift-ingress/e2e-shared-a-7a27b1-openshift-default-78f77667f-8l69z" Jul 18 12:30:26.740131 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:30:26.740108 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f3303f05-8333-424c-8824-8d4cda7ecc62-istio-envoy\") pod \"e2e-shared-a-7a27b1-openshift-default-78f77667f-8l69z\" (UID: \"f3303f05-8333-424c-8824-8d4cda7ecc62\") " pod="openshift-ingress/e2e-shared-a-7a27b1-openshift-default-78f77667f-8l69z" Jul 18 12:30:26.740596 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:30:26.740574 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f3303f05-8333-424c-8824-8d4cda7ecc62-istio-podinfo\") pod \"e2e-shared-a-7a27b1-openshift-default-78f77667f-8l69z\" (UID: \"f3303f05-8333-424c-8824-8d4cda7ecc62\") " pod="openshift-ingress/e2e-shared-a-7a27b1-openshift-default-78f77667f-8l69z" Jul 18 12:30:26.745661 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:30:26.745639 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bhrvg\" (UniqueName: \"kubernetes.io/projected/f3303f05-8333-424c-8824-8d4cda7ecc62-kube-api-access-bhrvg\") pod \"e2e-shared-a-7a27b1-openshift-default-78f77667f-8l69z\" (UID: \"f3303f05-8333-424c-8824-8d4cda7ecc62\") " pod="openshift-ingress/e2e-shared-a-7a27b1-openshift-default-78f77667f-8l69z" Jul 18 12:30:26.746113 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:30:26.746092 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f3303f05-8333-424c-8824-8d4cda7ecc62-istio-token\") pod \"e2e-shared-a-7a27b1-openshift-default-78f77667f-8l69z\" (UID: \"f3303f05-8333-424c-8824-8d4cda7ecc62\") " pod="openshift-ingress/e2e-shared-a-7a27b1-openshift-default-78f77667f-8l69z" Jul 18 12:30:26.847574 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:30:26.847490 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-7a27b1-openshift-default-78f77667f-8l69z" Jul 18 12:30:26.991042 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:30:26.990995 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-7a27b1-openshift-default-78f77667f-8l69z"] Jul 18 12:30:26.993424 ip-10-0-136-240 kubenswrapper[2585]: W0718 12:30:26.993397 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf3303f05_8333_424c_8824_8d4cda7ecc62.slice/crio-79c22c5b5ef4eb7e3d93fe68cd6a640cce7978a72eea0395c525036db5610618 WatchSource:0}: Error finding container 79c22c5b5ef4eb7e3d93fe68cd6a640cce7978a72eea0395c525036db5610618: Status 404 returned error can't find the container with id 79c22c5b5ef4eb7e3d93fe68cd6a640cce7978a72eea0395c525036db5610618 Jul 18 12:30:26.995975 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:30:26.995950 2585 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 18 12:30:26.996069 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:30:26.996016 2585 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 18 12:30:26.996069 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:30:26.996043 2585 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 18 12:30:27.012611 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:30:27.012580 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-7a27b1-openshift-default-78f77667f-8l69z" event={"ID":"f3303f05-8333-424c-8824-8d4cda7ecc62","Type":"ContainerStarted","Data":"79c22c5b5ef4eb7e3d93fe68cd6a640cce7978a72eea0395c525036db5610618"} Jul 18 12:30:28.023075 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:30:28.023036 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-7a27b1-openshift-default-78f77667f-8l69z" event={"ID":"f3303f05-8333-424c-8824-8d4cda7ecc62","Type":"ContainerStarted","Data":"f72c4fcf2b5765726bab46dbfb64d11786f50a360aede7afd870db3068cea044"} Jul 18 12:30:28.045410 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:30:28.045363 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-7a27b1-openshift-default-78f77667f-8l69z" podStartSLOduration=2.045348965 podStartE2EDuration="2.045348965s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-18 12:30:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-18 12:30:28.043749292 +0000 UTC m=+2495.123955644" watchObservedRunningTime="2026-07-18 12:30:28.045348965 +0000 UTC m=+2495.125555322" Jul 18 12:30:28.848353 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:30:28.848322 2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-7a27b1-openshift-default-78f77667f-8l69z" Jul 18 12:30:28.852973 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:30:28.852949 2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-7a27b1-openshift-default-78f77667f-8l69z" Jul 18 12:30:29.027804 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:30:29.027771 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-7a27b1-openshift-default-78f77667f-8l69z" Jul 18 12:30:29.028698 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:30:29.028675 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-7a27b1-openshift-default-78f77667f-8l69z" Jul 18 12:30:30.944794 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:30:30.944770 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-b7127d-openshift-default-7bfbdf8dc-p6g5g" Jul 18 12:30:30.979896 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:30:30.979860 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d44a53a4-d12a-4bf6-8601-d53d0818ea0b-workload-certs\") pod \"d44a53a4-d12a-4bf6-8601-d53d0818ea0b\" (UID: \"d44a53a4-d12a-4bf6-8601-d53d0818ea0b\") " Jul 18 12:30:30.980040 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:30:30.979899 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d44a53a4-d12a-4bf6-8601-d53d0818ea0b-credential-socket\") pod \"d44a53a4-d12a-4bf6-8601-d53d0818ea0b\" (UID: \"d44a53a4-d12a-4bf6-8601-d53d0818ea0b\") " Jul 18 12:30:30.980040 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:30:30.979925 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d44a53a4-d12a-4bf6-8601-d53d0818ea0b-istiod-ca-cert\") pod \"d44a53a4-d12a-4bf6-8601-d53d0818ea0b\" (UID: \"d44a53a4-d12a-4bf6-8601-d53d0818ea0b\") " Jul 18 12:30:30.980040 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:30:30.979956 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d44a53a4-d12a-4bf6-8601-d53d0818ea0b-istio-data\") pod \"d44a53a4-d12a-4bf6-8601-d53d0818ea0b\" (UID: \"d44a53a4-d12a-4bf6-8601-d53d0818ea0b\") " Jul 18 12:30:30.980040 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:30:30.979977 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d44a53a4-d12a-4bf6-8601-d53d0818ea0b-istio-podinfo\") pod \"d44a53a4-d12a-4bf6-8601-d53d0818ea0b\" (UID: \"d44a53a4-d12a-4bf6-8601-d53d0818ea0b\") " Jul 18 12:30:30.980250 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:30:30.980045 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d44a53a4-d12a-4bf6-8601-d53d0818ea0b-istio-envoy\") pod \"d44a53a4-d12a-4bf6-8601-d53d0818ea0b\" (UID: \"d44a53a4-d12a-4bf6-8601-d53d0818ea0b\") " Jul 18 12:30:30.980250 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:30:30.980101 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d44a53a4-d12a-4bf6-8601-d53d0818ea0b-istio-token\") pod \"d44a53a4-d12a-4bf6-8601-d53d0818ea0b\" (UID: \"d44a53a4-d12a-4bf6-8601-d53d0818ea0b\") " Jul 18 12:30:30.980250 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:30:30.980130 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d44a53a4-d12a-4bf6-8601-d53d0818ea0b-workload-socket\") pod \"d44a53a4-d12a-4bf6-8601-d53d0818ea0b\" (UID: \"d44a53a4-d12a-4bf6-8601-d53d0818ea0b\") " Jul 18 12:30:30.980250 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:30:30.980159 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d44a53a4-d12a-4bf6-8601-d53d0818ea0b-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "d44a53a4-d12a-4bf6-8601-d53d0818ea0b" (UID: "d44a53a4-d12a-4bf6-8601-d53d0818ea0b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 18 12:30:30.980250 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:30:30.980173 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tvbln\" (UniqueName: \"kubernetes.io/projected/d44a53a4-d12a-4bf6-8601-d53d0818ea0b-kube-api-access-tvbln\") pod \"d44a53a4-d12a-4bf6-8601-d53d0818ea0b\" (UID: \"d44a53a4-d12a-4bf6-8601-d53d0818ea0b\") " Jul 18 12:30:30.980506 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:30:30.980269 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d44a53a4-d12a-4bf6-8601-d53d0818ea0b-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "d44a53a4-d12a-4bf6-8601-d53d0818ea0b" (UID: "d44a53a4-d12a-4bf6-8601-d53d0818ea0b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 18 12:30:30.980506 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:30:30.980354 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d44a53a4-d12a-4bf6-8601-d53d0818ea0b-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "d44a53a4-d12a-4bf6-8601-d53d0818ea0b" (UID: "d44a53a4-d12a-4bf6-8601-d53d0818ea0b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 18 12:30:30.980506 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:30:30.980487 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d44a53a4-d12a-4bf6-8601-d53d0818ea0b-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "d44a53a4-d12a-4bf6-8601-d53d0818ea0b" (UID: "d44a53a4-d12a-4bf6-8601-d53d0818ea0b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 18 12:30:30.980716 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:30:30.980666 2585 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d44a53a4-d12a-4bf6-8601-d53d0818ea0b-workload-certs\") on node \"ip-10-0-136-240.ec2.internal\" DevicePath \"\"" Jul 18 12:30:30.980716 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:30:30.980688 2585 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d44a53a4-d12a-4bf6-8601-d53d0818ea0b-credential-socket\") on node \"ip-10-0-136-240.ec2.internal\" DevicePath \"\"" Jul 18 12:30:30.980716 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:30:30.980702 2585 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d44a53a4-d12a-4bf6-8601-d53d0818ea0b-istiod-ca-cert\") on node \"ip-10-0-136-240.ec2.internal\" DevicePath \"\"" Jul 18 12:30:30.980871 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:30:30.980717 2585 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d44a53a4-d12a-4bf6-8601-d53d0818ea0b-workload-socket\") on node \"ip-10-0-136-240.ec2.internal\" DevicePath \"\"" Jul 18 12:30:30.981025 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:30:30.980970 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d44a53a4-d12a-4bf6-8601-d53d0818ea0b-istio-data" (OuterVolumeSpecName: "istio-data") pod "d44a53a4-d12a-4bf6-8601-d53d0818ea0b" (UID: "d44a53a4-d12a-4bf6-8601-d53d0818ea0b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 18 12:30:30.983197 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:30:30.983173 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d44a53a4-d12a-4bf6-8601-d53d0818ea0b-kube-api-access-tvbln" (OuterVolumeSpecName: "kube-api-access-tvbln") pod "d44a53a4-d12a-4bf6-8601-d53d0818ea0b" (UID: "d44a53a4-d12a-4bf6-8601-d53d0818ea0b"). InnerVolumeSpecName "kube-api-access-tvbln". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 18 12:30:30.983287 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:30:30.983196 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d44a53a4-d12a-4bf6-8601-d53d0818ea0b-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "d44a53a4-d12a-4bf6-8601-d53d0818ea0b" (UID: "d44a53a4-d12a-4bf6-8601-d53d0818ea0b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 18 12:30:30.983287 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:30:30.983202 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d44a53a4-d12a-4bf6-8601-d53d0818ea0b-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "d44a53a4-d12a-4bf6-8601-d53d0818ea0b" (UID: "d44a53a4-d12a-4bf6-8601-d53d0818ea0b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 18 12:30:30.983287 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:30:30.983203 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d44a53a4-d12a-4bf6-8601-d53d0818ea0b-istio-token" (OuterVolumeSpecName: "istio-token") pod "d44a53a4-d12a-4bf6-8601-d53d0818ea0b" (UID: "d44a53a4-d12a-4bf6-8601-d53d0818ea0b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 18 12:30:31.040286 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:30:31.040255 2585 generic.go:356] "Generic (PLEG): container finished" podID="d44a53a4-d12a-4bf6-8601-d53d0818ea0b" containerID="e7d6e4a748155512ea3134db68b6b308e9ed6d8bebda446860153b39a457732b" exitCode=0 Jul 18 12:30:31.040383 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:30:31.040303 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-b7127d-openshift-default-7bfbdf8dc-p6g5g" Jul 18 12:30:31.040383 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:30:31.040344 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-b7127d-openshift-default-7bfbdf8dc-p6g5g" event={"ID":"d44a53a4-d12a-4bf6-8601-d53d0818ea0b","Type":"ContainerDied","Data":"e7d6e4a748155512ea3134db68b6b308e9ed6d8bebda446860153b39a457732b"} Jul 18 12:30:31.040452 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:30:31.040390 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-b7127d-openshift-default-7bfbdf8dc-p6g5g" event={"ID":"d44a53a4-d12a-4bf6-8601-d53d0818ea0b","Type":"ContainerDied","Data":"49759bd8294666e834eb11c170c0c8d94236b03884984b6b1bd72bad70514510"} Jul 18 12:30:31.040452 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:30:31.040412 2585 scope.go:117] "RemoveContainer" containerID="e7d6e4a748155512ea3134db68b6b308e9ed6d8bebda446860153b39a457732b" Jul 18 12:30:31.050126 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:30:31.050109 2585 scope.go:117] "RemoveContainer" containerID="0e896a47ccf785af9109b68628935664a99615c5af457aae0fc5d33b1c2d5e3f" Jul 18 12:30:31.058288 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:30:31.058270 2585 scope.go:117] "RemoveContainer" containerID="e7d6e4a748155512ea3134db68b6b308e9ed6d8bebda446860153b39a457732b" Jul 18 12:30:31.058624 ip-10-0-136-240 kubenswrapper[2585]: E0718 12:30:31.058525 2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e7d6e4a748155512ea3134db68b6b308e9ed6d8bebda446860153b39a457732b\": container with ID starting with e7d6e4a748155512ea3134db68b6b308e9ed6d8bebda446860153b39a457732b not found: ID does not exist" containerID="e7d6e4a748155512ea3134db68b6b308e9ed6d8bebda446860153b39a457732b" Jul 18 12:30:31.058624 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:30:31.058574 2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e7d6e4a748155512ea3134db68b6b308e9ed6d8bebda446860153b39a457732b"} err="failed to get container status \"e7d6e4a748155512ea3134db68b6b308e9ed6d8bebda446860153b39a457732b\": rpc error: code = NotFound desc = could not find container \"e7d6e4a748155512ea3134db68b6b308e9ed6d8bebda446860153b39a457732b\": container with ID starting with e7d6e4a748155512ea3134db68b6b308e9ed6d8bebda446860153b39a457732b not found: ID does not exist" Jul 18 12:30:31.058624 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:30:31.058596 2585 scope.go:117] "RemoveContainer" containerID="0e896a47ccf785af9109b68628935664a99615c5af457aae0fc5d33b1c2d5e3f" Jul 18 12:30:31.058941 ip-10-0-136-240 kubenswrapper[2585]: E0718 12:30:31.058924 2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0e896a47ccf785af9109b68628935664a99615c5af457aae0fc5d33b1c2d5e3f\": container with ID starting with 0e896a47ccf785af9109b68628935664a99615c5af457aae0fc5d33b1c2d5e3f not found: ID does not exist" containerID="0e896a47ccf785af9109b68628935664a99615c5af457aae0fc5d33b1c2d5e3f" Jul 18 12:30:31.058981 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:30:31.058948 2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0e896a47ccf785af9109b68628935664a99615c5af457aae0fc5d33b1c2d5e3f"} err="failed to get container status \"0e896a47ccf785af9109b68628935664a99615c5af457aae0fc5d33b1c2d5e3f\": rpc error: code = NotFound desc = could not find container \"0e896a47ccf785af9109b68628935664a99615c5af457aae0fc5d33b1c2d5e3f\": container with ID starting with 0e896a47ccf785af9109b68628935664a99615c5af457aae0fc5d33b1c2d5e3f not found: ID does not exist" Jul 18 12:30:31.064422 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:30:31.064385 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-b7127d-openshift-default-7bfbdf8dc-p6g5g"] Jul 18 12:30:31.066604 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:30:31.066583 2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-b7127d-openshift-default-7bfbdf8dc-p6g5g"] Jul 18 12:30:31.081677 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:30:31.081654 2585 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d44a53a4-d12a-4bf6-8601-d53d0818ea0b-istio-data\") on node \"ip-10-0-136-240.ec2.internal\" DevicePath \"\"" Jul 18 12:30:31.081781 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:30:31.081682 2585 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d44a53a4-d12a-4bf6-8601-d53d0818ea0b-istio-podinfo\") on node \"ip-10-0-136-240.ec2.internal\" DevicePath \"\"" Jul 18 12:30:31.081781 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:30:31.081695 2585 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d44a53a4-d12a-4bf6-8601-d53d0818ea0b-istio-envoy\") on node \"ip-10-0-136-240.ec2.internal\" DevicePath \"\"" Jul 18 12:30:31.081781 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:30:31.081713 2585 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d44a53a4-d12a-4bf6-8601-d53d0818ea0b-istio-token\") on node \"ip-10-0-136-240.ec2.internal\" DevicePath \"\"" Jul 18 12:30:31.081781 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:30:31.081726 2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tvbln\" (UniqueName: \"kubernetes.io/projected/d44a53a4-d12a-4bf6-8601-d53d0818ea0b-kube-api-access-tvbln\") on node \"ip-10-0-136-240.ec2.internal\" DevicePath \"\"" Jul 18 12:30:31.468979 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:30:31.468944 2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d44a53a4-d12a-4bf6-8601-d53d0818ea0b" path="/var/lib/kubelet/pods/d44a53a4-d12a-4bf6-8601-d53d0818ea0b/volumes" Jul 18 12:30:34.976741 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:30:34.976709 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-8b9f97-openshift-default-6d95c44899-969fp"] Jul 18 12:30:34.977270 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:30:34.977120 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d44a53a4-d12a-4bf6-8601-d53d0818ea0b" containerName="istio-proxy" Jul 18 12:30:34.977270 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:30:34.977132 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="d44a53a4-d12a-4bf6-8601-d53d0818ea0b" containerName="istio-proxy" Jul 18 12:30:34.977270 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:30:34.977145 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cf0cb8e4-36b7-4cb5-b4c1-09f746e04491" containerName="istio-proxy" Jul 18 12:30:34.977270 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:30:34.977153 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="cf0cb8e4-36b7-4cb5-b4c1-09f746e04491" containerName="istio-proxy" Jul 18 12:30:34.977270 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:30:34.977231 2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="d44a53a4-d12a-4bf6-8601-d53d0818ea0b" containerName="istio-proxy" Jul 18 12:30:34.977270 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:30:34.977239 2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="d44a53a4-d12a-4bf6-8601-d53d0818ea0b" containerName="istio-proxy" Jul 18 12:30:34.981586 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:30:34.981542 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-8b9f97-openshift-default-6d95c44899-969fp" Jul 18 12:30:34.985608 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:30:34.985588 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-8b9f97-openshift-default-dockercfg-v5c8t\"" Jul 18 12:30:34.992920 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:30:34.992900 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-8b9f97-openshift-default-6d95c44899-969fp"] Jul 18 12:30:35.026967 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:30:35.026943 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9be6d88e-bbb1-422c-971c-5af9cec4f1c6-workload-socket\") pod \"e2e-shared-b-8b9f97-openshift-default-6d95c44899-969fp\" (UID: \"9be6d88e-bbb1-422c-971c-5af9cec4f1c6\") " pod="openshift-ingress/e2e-shared-b-8b9f97-openshift-default-6d95c44899-969fp" Jul 18 12:30:35.027086 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:30:35.026971 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9be6d88e-bbb1-422c-971c-5af9cec4f1c6-istio-token\") pod \"e2e-shared-b-8b9f97-openshift-default-6d95c44899-969fp\" (UID: \"9be6d88e-bbb1-422c-971c-5af9cec4f1c6\") " pod="openshift-ingress/e2e-shared-b-8b9f97-openshift-default-6d95c44899-969fp" Jul 18 12:30:35.027086 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:30:35.026991 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9be6d88e-bbb1-422c-971c-5af9cec4f1c6-istio-podinfo\") pod \"e2e-shared-b-8b9f97-openshift-default-6d95c44899-969fp\" (UID: \"9be6d88e-bbb1-422c-971c-5af9cec4f1c6\") " pod="openshift-ingress/e2e-shared-b-8b9f97-openshift-default-6d95c44899-969fp" Jul 18 12:30:35.027162 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:30:35.027101 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9be6d88e-bbb1-422c-971c-5af9cec4f1c6-credential-socket\") pod \"e2e-shared-b-8b9f97-openshift-default-6d95c44899-969fp\" (UID: \"9be6d88e-bbb1-422c-971c-5af9cec4f1c6\") " pod="openshift-ingress/e2e-shared-b-8b9f97-openshift-default-6d95c44899-969fp" Jul 18 12:30:35.027162 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:30:35.027132 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9be6d88e-bbb1-422c-971c-5af9cec4f1c6-istiod-ca-cert\") pod \"e2e-shared-b-8b9f97-openshift-default-6d95c44899-969fp\" (UID: \"9be6d88e-bbb1-422c-971c-5af9cec4f1c6\") " pod="openshift-ingress/e2e-shared-b-8b9f97-openshift-default-6d95c44899-969fp" Jul 18 12:30:35.027162 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:30:35.027158 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9be6d88e-bbb1-422c-971c-5af9cec4f1c6-workload-certs\") pod \"e2e-shared-b-8b9f97-openshift-default-6d95c44899-969fp\" (UID: \"9be6d88e-bbb1-422c-971c-5af9cec4f1c6\") " pod="openshift-ingress/e2e-shared-b-8b9f97-openshift-default-6d95c44899-969fp" Jul 18 12:30:35.027258 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:30:35.027183 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-24pm7\" (UniqueName: \"kubernetes.io/projected/9be6d88e-bbb1-422c-971c-5af9cec4f1c6-kube-api-access-24pm7\") pod \"e2e-shared-b-8b9f97-openshift-default-6d95c44899-969fp\" (UID: \"9be6d88e-bbb1-422c-971c-5af9cec4f1c6\") " pod="openshift-ingress/e2e-shared-b-8b9f97-openshift-default-6d95c44899-969fp" Jul 18 12:30:35.027258 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:30:35.027228 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9be6d88e-bbb1-422c-971c-5af9cec4f1c6-istio-data\") pod \"e2e-shared-b-8b9f97-openshift-default-6d95c44899-969fp\" (UID: \"9be6d88e-bbb1-422c-971c-5af9cec4f1c6\") " pod="openshift-ingress/e2e-shared-b-8b9f97-openshift-default-6d95c44899-969fp" Jul 18 12:30:35.027321 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:30:35.027300 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9be6d88e-bbb1-422c-971c-5af9cec4f1c6-istio-envoy\") pod \"e2e-shared-b-8b9f97-openshift-default-6d95c44899-969fp\" (UID: \"9be6d88e-bbb1-422c-971c-5af9cec4f1c6\") " pod="openshift-ingress/e2e-shared-b-8b9f97-openshift-default-6d95c44899-969fp" Jul 18 12:30:35.128807 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:30:35.128762 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9be6d88e-bbb1-422c-971c-5af9cec4f1c6-workload-socket\") pod \"e2e-shared-b-8b9f97-openshift-default-6d95c44899-969fp\" (UID: \"9be6d88e-bbb1-422c-971c-5af9cec4f1c6\") " pod="openshift-ingress/e2e-shared-b-8b9f97-openshift-default-6d95c44899-969fp" Jul 18 12:30:35.128807 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:30:35.128807 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9be6d88e-bbb1-422c-971c-5af9cec4f1c6-istio-token\") pod \"e2e-shared-b-8b9f97-openshift-default-6d95c44899-969fp\" (UID: \"9be6d88e-bbb1-422c-971c-5af9cec4f1c6\") " pod="openshift-ingress/e2e-shared-b-8b9f97-openshift-default-6d95c44899-969fp" Jul 18 12:30:35.129068 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:30:35.128827 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9be6d88e-bbb1-422c-971c-5af9cec4f1c6-istio-podinfo\") pod \"e2e-shared-b-8b9f97-openshift-default-6d95c44899-969fp\" (UID: \"9be6d88e-bbb1-422c-971c-5af9cec4f1c6\") " pod="openshift-ingress/e2e-shared-b-8b9f97-openshift-default-6d95c44899-969fp" Jul 18 12:30:35.129068 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:30:35.128871 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9be6d88e-bbb1-422c-971c-5af9cec4f1c6-credential-socket\") pod \"e2e-shared-b-8b9f97-openshift-default-6d95c44899-969fp\" (UID: \"9be6d88e-bbb1-422c-971c-5af9cec4f1c6\") " pod="openshift-ingress/e2e-shared-b-8b9f97-openshift-default-6d95c44899-969fp" Jul 18 12:30:35.129068 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:30:35.128887 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9be6d88e-bbb1-422c-971c-5af9cec4f1c6-istiod-ca-cert\") pod \"e2e-shared-b-8b9f97-openshift-default-6d95c44899-969fp\" (UID: \"9be6d88e-bbb1-422c-971c-5af9cec4f1c6\") " pod="openshift-ingress/e2e-shared-b-8b9f97-openshift-default-6d95c44899-969fp" Jul 18 12:30:35.129068 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:30:35.128911 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9be6d88e-bbb1-422c-971c-5af9cec4f1c6-workload-certs\") pod \"e2e-shared-b-8b9f97-openshift-default-6d95c44899-969fp\" (UID: \"9be6d88e-bbb1-422c-971c-5af9cec4f1c6\") " pod="openshift-ingress/e2e-shared-b-8b9f97-openshift-default-6d95c44899-969fp" Jul 18 12:30:35.129068 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:30:35.128927 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-24pm7\" (UniqueName: \"kubernetes.io/projected/9be6d88e-bbb1-422c-971c-5af9cec4f1c6-kube-api-access-24pm7\") pod \"e2e-shared-b-8b9f97-openshift-default-6d95c44899-969fp\" (UID: \"9be6d88e-bbb1-422c-971c-5af9cec4f1c6\") " pod="openshift-ingress/e2e-shared-b-8b9f97-openshift-default-6d95c44899-969fp" Jul 18 12:30:35.129068 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:30:35.128949 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9be6d88e-bbb1-422c-971c-5af9cec4f1c6-istio-data\") pod \"e2e-shared-b-8b9f97-openshift-default-6d95c44899-969fp\" (UID: \"9be6d88e-bbb1-422c-971c-5af9cec4f1c6\") " pod="openshift-ingress/e2e-shared-b-8b9f97-openshift-default-6d95c44899-969fp" Jul 18 12:30:35.129068 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:30:35.129019 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9be6d88e-bbb1-422c-971c-5af9cec4f1c6-istio-envoy\") pod \"e2e-shared-b-8b9f97-openshift-default-6d95c44899-969fp\" (UID: \"9be6d88e-bbb1-422c-971c-5af9cec4f1c6\") " pod="openshift-ingress/e2e-shared-b-8b9f97-openshift-default-6d95c44899-969fp" Jul 18 12:30:35.129420 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:30:35.129194 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9be6d88e-bbb1-422c-971c-5af9cec4f1c6-workload-socket\") pod \"e2e-shared-b-8b9f97-openshift-default-6d95c44899-969fp\" (UID: \"9be6d88e-bbb1-422c-971c-5af9cec4f1c6\") " pod="openshift-ingress/e2e-shared-b-8b9f97-openshift-default-6d95c44899-969fp" Jul 18 12:30:35.129420 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:30:35.129353 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9be6d88e-bbb1-422c-971c-5af9cec4f1c6-workload-certs\") pod \"e2e-shared-b-8b9f97-openshift-default-6d95c44899-969fp\" (UID: \"9be6d88e-bbb1-422c-971c-5af9cec4f1c6\") " pod="openshift-ingress/e2e-shared-b-8b9f97-openshift-default-6d95c44899-969fp" Jul 18 12:30:35.129420 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:30:35.129399 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9be6d88e-bbb1-422c-971c-5af9cec4f1c6-credential-socket\") pod \"e2e-shared-b-8b9f97-openshift-default-6d95c44899-969fp\" (UID: \"9be6d88e-bbb1-422c-971c-5af9cec4f1c6\") " pod="openshift-ingress/e2e-shared-b-8b9f97-openshift-default-6d95c44899-969fp" Jul 18 12:30:35.129533 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:30:35.129489 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9be6d88e-bbb1-422c-971c-5af9cec4f1c6-istio-data\") pod \"e2e-shared-b-8b9f97-openshift-default-6d95c44899-969fp\" (UID: \"9be6d88e-bbb1-422c-971c-5af9cec4f1c6\") " pod="openshift-ingress/e2e-shared-b-8b9f97-openshift-default-6d95c44899-969fp" Jul 18 12:30:35.129635 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:30:35.129613 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9be6d88e-bbb1-422c-971c-5af9cec4f1c6-istiod-ca-cert\") pod \"e2e-shared-b-8b9f97-openshift-default-6d95c44899-969fp\" (UID: \"9be6d88e-bbb1-422c-971c-5af9cec4f1c6\") " pod="openshift-ingress/e2e-shared-b-8b9f97-openshift-default-6d95c44899-969fp" Jul 18 12:30:35.131544 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:30:35.131511 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9be6d88e-bbb1-422c-971c-5af9cec4f1c6-istio-envoy\") pod \"e2e-shared-b-8b9f97-openshift-default-6d95c44899-969fp\" (UID: \"9be6d88e-bbb1-422c-971c-5af9cec4f1c6\") " pod="openshift-ingress/e2e-shared-b-8b9f97-openshift-default-6d95c44899-969fp" Jul 18 12:30:35.131806 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:30:35.131785 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9be6d88e-bbb1-422c-971c-5af9cec4f1c6-istio-podinfo\") pod \"e2e-shared-b-8b9f97-openshift-default-6d95c44899-969fp\" (UID: \"9be6d88e-bbb1-422c-971c-5af9cec4f1c6\") " pod="openshift-ingress/e2e-shared-b-8b9f97-openshift-default-6d95c44899-969fp" Jul 18 12:30:35.137466 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:30:35.137442 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9be6d88e-bbb1-422c-971c-5af9cec4f1c6-istio-token\") pod \"e2e-shared-b-8b9f97-openshift-default-6d95c44899-969fp\" (UID: \"9be6d88e-bbb1-422c-971c-5af9cec4f1c6\") " pod="openshift-ingress/e2e-shared-b-8b9f97-openshift-default-6d95c44899-969fp" Jul 18 12:30:35.138214 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:30:35.138193 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-24pm7\" (UniqueName: \"kubernetes.io/projected/9be6d88e-bbb1-422c-971c-5af9cec4f1c6-kube-api-access-24pm7\") pod \"e2e-shared-b-8b9f97-openshift-default-6d95c44899-969fp\" (UID: \"9be6d88e-bbb1-422c-971c-5af9cec4f1c6\") " pod="openshift-ingress/e2e-shared-b-8b9f97-openshift-default-6d95c44899-969fp" Jul 18 12:30:35.292415 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:30:35.292320 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-8b9f97-openshift-default-6d95c44899-969fp" Jul 18 12:30:35.439471 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:30:35.439444 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-8b9f97-openshift-default-6d95c44899-969fp"] Jul 18 12:30:35.441517 ip-10-0-136-240 kubenswrapper[2585]: W0718 12:30:35.441471 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9be6d88e_bbb1_422c_971c_5af9cec4f1c6.slice/crio-74d68efa698c7b3728c1ace4118db13a6ffc2e1fccc31677573412e57df5ddaa WatchSource:0}: Error finding container 74d68efa698c7b3728c1ace4118db13a6ffc2e1fccc31677573412e57df5ddaa: Status 404 returned error can't find the container with id 74d68efa698c7b3728c1ace4118db13a6ffc2e1fccc31677573412e57df5ddaa Jul 18 12:30:35.443693 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:30:35.443662 2585 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 18 12:30:35.443805 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:30:35.443724 2585 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 18 12:30:35.443805 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:30:35.443753 2585 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 18 12:30:36.069617 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:30:36.069582 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-8b9f97-openshift-default-6d95c44899-969fp" event={"ID":"9be6d88e-bbb1-422c-971c-5af9cec4f1c6","Type":"ContainerStarted","Data":"72dc1303f28c14c5868863ff85c8c6bf1cee89cbdb8a64be169b8553940a1d24"} Jul 18 12:30:36.070028 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:30:36.069646 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-8b9f97-openshift-default-6d95c44899-969fp" event={"ID":"9be6d88e-bbb1-422c-971c-5af9cec4f1c6","Type":"ContainerStarted","Data":"74d68efa698c7b3728c1ace4118db13a6ffc2e1fccc31677573412e57df5ddaa"} Jul 18 12:30:36.094825 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:30:36.094769 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-8b9f97-openshift-default-6d95c44899-969fp" podStartSLOduration=2.094747819 podStartE2EDuration="2.094747819s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-18 12:30:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-18 12:30:36.094197674 +0000 UTC m=+2503.174404026" watchObservedRunningTime="2026-07-18 12:30:36.094747819 +0000 UTC m=+2503.174954177" Jul 18 12:30:36.293264 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:30:36.293232 2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-8b9f97-openshift-default-6d95c44899-969fp" Jul 18 12:30:36.298367 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:30:36.298343 2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-8b9f97-openshift-default-6d95c44899-969fp" Jul 18 12:30:37.075351 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:30:37.075323 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-8b9f97-openshift-default-6d95c44899-969fp" Jul 18 12:30:37.076227 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:30:37.076210 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-8b9f97-openshift-default-6d95c44899-969fp" Jul 18 12:30:45.069881 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:30:45.069780 2585 patch_prober.go:28] interesting pod/e2e-shared-a-7a27b1-openshift-default-78f77667f-8l69z container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.56:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Jul 18 12:30:45.070522 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:30:45.070477 2585 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-7a27b1-openshift-default-78f77667f-8l69z" podUID="f3303f05-8333-424c-8824-8d4cda7ecc62" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.56:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Jul 18 12:30:46.135614 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:30:46.135579 2585 generic.go:356] "Generic (PLEG): container finished" podID="f3303f05-8333-424c-8824-8d4cda7ecc62" containerID="f72c4fcf2b5765726bab46dbfb64d11786f50a360aede7afd870db3068cea044" exitCode=0 Jul 18 12:30:46.136007 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:30:46.135658 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-7a27b1-openshift-default-78f77667f-8l69z" event={"ID":"f3303f05-8333-424c-8824-8d4cda7ecc62","Type":"ContainerDied","Data":"f72c4fcf2b5765726bab46dbfb64d11786f50a360aede7afd870db3068cea044"} Jul 18 12:30:46.136121 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:30:46.136106 2585 scope.go:117] "RemoveContainer" containerID="f72c4fcf2b5765726bab46dbfb64d11786f50a360aede7afd870db3068cea044" Jul 18 12:30:46.137258 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:30:46.137233 2585 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 18 12:30:46.137334 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:30:46.137297 2585 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 18 12:30:46.137334 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:30:46.137328 2585 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 18 12:30:47.143750 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:30:47.143718 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-7a27b1-openshift-default-78f77667f-8l69z" event={"ID":"f3303f05-8333-424c-8824-8d4cda7ecc62","Type":"ContainerStarted","Data":"01fedf7826a472ec508c04079af2b639ac022cc57c94a1c29e742fb12f34c0f0"} Jul 18 12:30:47.848478 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:30:47.848442 2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-7a27b1-openshift-default-78f77667f-8l69z" Jul 18 12:30:47.848753 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:30:47.848732 2585 patch_prober.go:28] interesting pod/e2e-shared-a-7a27b1-openshift-default-78f77667f-8l69z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" start-of-body= Jul 18 12:30:47.848814 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:30:47.848779 2585 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-7a27b1-openshift-default-78f77667f-8l69z" podUID="f3303f05-8333-424c-8824-8d4cda7ecc62" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" Jul 18 12:30:48.848293 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:30:48.848255 2585 patch_prober.go:28] interesting pod/e2e-shared-a-7a27b1-openshift-default-78f77667f-8l69z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" start-of-body= Jul 18 12:30:48.848706 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:30:48.848324 2585 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-7a27b1-openshift-default-78f77667f-8l69z" podUID="f3303f05-8333-424c-8824-8d4cda7ecc62" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" Jul 18 12:30:49.848175 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:30:49.848138 2585 patch_prober.go:28] interesting pod/e2e-shared-a-7a27b1-openshift-default-78f77667f-8l69z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" start-of-body= Jul 18 12:30:49.848383 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:30:49.848196 2585 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-7a27b1-openshift-default-78f77667f-8l69z" podUID="f3303f05-8333-424c-8824-8d4cda7ecc62" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" Jul 18 12:30:50.848026 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:30:50.847986 2585 patch_prober.go:28] interesting pod/e2e-shared-a-7a27b1-openshift-default-78f77667f-8l69z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" start-of-body= Jul 18 12:30:50.848201 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:30:50.848063 2585 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-7a27b1-openshift-default-78f77667f-8l69z" podUID="f3303f05-8333-424c-8824-8d4cda7ecc62" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" Jul 18 12:30:51.852608 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:30:51.852575 2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-7a27b1-openshift-default-78f77667f-8l69z" Jul 18 12:30:51.853091 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:30:51.852979 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-7a27b1-openshift-default-78f77667f-8l69z" Jul 18 12:30:51.853846 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:30:51.853828 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-7a27b1-openshift-default-78f77667f-8l69z" Jul 18 12:30:52.130384 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:30:52.130225 2585 patch_prober.go:28] interesting pod/e2e-shared-b-8b9f97-openshift-default-6d95c44899-969fp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": read tcp 10.133.0.2:33788->10.133.0.57:15021: read: connection reset by peer" start-of-body= Jul 18 12:30:52.130384 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:30:52.130296 2585 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-8b9f97-openshift-default-6d95c44899-969fp" podUID="9be6d88e-bbb1-422c-971c-5af9cec4f1c6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": read tcp 10.133.0.2:33788->10.133.0.57:15021: read: connection reset by peer" Jul 18 12:30:53.176182 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:30:53.176148 2585 generic.go:356] "Generic (PLEG): container finished" podID="9be6d88e-bbb1-422c-971c-5af9cec4f1c6" containerID="72dc1303f28c14c5868863ff85c8c6bf1cee89cbdb8a64be169b8553940a1d24" exitCode=0 Jul 18 12:30:53.176588 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:30:53.176223 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-8b9f97-openshift-default-6d95c44899-969fp" event={"ID":"9be6d88e-bbb1-422c-971c-5af9cec4f1c6","Type":"ContainerDied","Data":"72dc1303f28c14c5868863ff85c8c6bf1cee89cbdb8a64be169b8553940a1d24"} Jul 18 12:30:53.176710 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:30:53.176692 2585 scope.go:117] "RemoveContainer" containerID="72dc1303f28c14c5868863ff85c8c6bf1cee89cbdb8a64be169b8553940a1d24" Jul 18 12:30:53.177710 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:30:53.177680 2585 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 18 12:30:53.177821 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:30:53.177748 2585 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 18 12:30:53.177821 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:30:53.177781 2585 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 18 12:30:54.194731 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:30:54.194694 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-8b9f97-openshift-default-6d95c44899-969fp" event={"ID":"9be6d88e-bbb1-422c-971c-5af9cec4f1c6","Type":"ContainerStarted","Data":"cae6f0d75038ac933649e13bbfe3d64e3fd95d1cdb97156f484532b1b6970f4e"} Jul 18 12:30:54.293231 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:30:54.293184 2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-8b9f97-openshift-default-6d95c44899-969fp" Jul 18 12:30:54.293453 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:30:54.293430 2585 patch_prober.go:28] interesting pod/e2e-shared-b-8b9f97-openshift-default-6d95c44899-969fp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" start-of-body= Jul 18 12:30:54.293532 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:30:54.293480 2585 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-8b9f97-openshift-default-6d95c44899-969fp" podUID="9be6d88e-bbb1-422c-971c-5af9cec4f1c6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" Jul 18 12:30:55.293233 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:30:55.293200 2585 patch_prober.go:28] interesting pod/e2e-shared-b-8b9f97-openshift-default-6d95c44899-969fp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" start-of-body= Jul 18 12:30:55.293642 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:30:55.293251 2585 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-8b9f97-openshift-default-6d95c44899-969fp" podUID="9be6d88e-bbb1-422c-971c-5af9cec4f1c6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" Jul 18 12:30:56.293025 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:30:56.292988 2585 patch_prober.go:28] interesting pod/e2e-shared-b-8b9f97-openshift-default-6d95c44899-969fp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" start-of-body= Jul 18 12:30:56.293197 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:30:56.293048 2585 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-8b9f97-openshift-default-6d95c44899-969fp" podUID="9be6d88e-bbb1-422c-971c-5af9cec4f1c6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" Jul 18 12:30:57.293504 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:30:57.293463 2585 patch_prober.go:28] interesting pod/e2e-shared-b-8b9f97-openshift-default-6d95c44899-969fp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" start-of-body= Jul 18 12:30:57.293913 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:30:57.293520 2585 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-8b9f97-openshift-default-6d95c44899-969fp" podUID="9be6d88e-bbb1-422c-971c-5af9cec4f1c6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" Jul 18 12:30:58.296731 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:30:58.296698 2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-8b9f97-openshift-default-6d95c44899-969fp" Jul 18 12:30:58.297122 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:30:58.296970 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-8b9f97-openshift-default-6d95c44899-969fp" Jul 18 12:30:58.297571 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:30:58.297533 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-8b9f97-openshift-default-6d95c44899-969fp" Jul 18 12:31:06.552822 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:31:06.552791 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-7a27b1/auth-test-model-7a27b1-kserve-8557d588f9-chxp7"] Jul 18 12:31:06.553455 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:31:06.553221 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d44a53a4-d12a-4bf6-8601-d53d0818ea0b" containerName="istio-proxy" Jul 18 12:31:06.553455 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:31:06.553232 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="d44a53a4-d12a-4bf6-8601-d53d0818ea0b" containerName="istio-proxy" Jul 18 12:31:06.556568 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:31:06.556537 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-7a27b1/auth-test-model-7a27b1-kserve-8557d588f9-chxp7" Jul 18 12:31:06.559827 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:31:06.559800 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-7a27b1\"/\"default-dockercfg-8lz4m\"" Jul 18 12:31:06.559982 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:31:06.559800 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-7a27b1\"/\"kube-root-ca.crt\"" Jul 18 12:31:06.559982 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:31:06.559805 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-7a27b1\"/\"openshift-service-ca.crt\"" Jul 18 12:31:06.559982 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:31:06.559806 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-7a27b1\"/\"auth-test-model-7a27b1-kserve-self-signed-certs\"" Jul 18 12:31:06.564647 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:31:06.564212 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-7a27b1/auth-test-model-7a27b1-kserve-8557d588f9-chxp7"] Jul 18 12:31:06.715895 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:31:06.715859 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e087dc9f-74c0-4f0f-903b-758efbe22c2c-tmp-dir\") pod \"auth-test-model-7a27b1-kserve-8557d588f9-chxp7\" (UID: \"e087dc9f-74c0-4f0f-903b-758efbe22c2c\") " pod="ai-tenant-e2e-shared-a-7a27b1/auth-test-model-7a27b1-kserve-8557d588f9-chxp7" Jul 18 12:31:06.716098 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:31:06.715914 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e087dc9f-74c0-4f0f-903b-758efbe22c2c-model-cache\") pod \"auth-test-model-7a27b1-kserve-8557d588f9-chxp7\" (UID: \"e087dc9f-74c0-4f0f-903b-758efbe22c2c\") " pod="ai-tenant-e2e-shared-a-7a27b1/auth-test-model-7a27b1-kserve-8557d588f9-chxp7" Jul 18 12:31:06.716098 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:31:06.716005 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e087dc9f-74c0-4f0f-903b-758efbe22c2c-home\") pod \"auth-test-model-7a27b1-kserve-8557d588f9-chxp7\" (UID: \"e087dc9f-74c0-4f0f-903b-758efbe22c2c\") " pod="ai-tenant-e2e-shared-a-7a27b1/auth-test-model-7a27b1-kserve-8557d588f9-chxp7" Jul 18 12:31:06.716098 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:31:06.716044 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e087dc9f-74c0-4f0f-903b-758efbe22c2c-dshm\") pod \"auth-test-model-7a27b1-kserve-8557d588f9-chxp7\" (UID: \"e087dc9f-74c0-4f0f-903b-758efbe22c2c\") " pod="ai-tenant-e2e-shared-a-7a27b1/auth-test-model-7a27b1-kserve-8557d588f9-chxp7" Jul 18 12:31:06.716265 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:31:06.716118 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e087dc9f-74c0-4f0f-903b-758efbe22c2c-tls-certs\") pod \"auth-test-model-7a27b1-kserve-8557d588f9-chxp7\" (UID: \"e087dc9f-74c0-4f0f-903b-758efbe22c2c\") " pod="ai-tenant-e2e-shared-a-7a27b1/auth-test-model-7a27b1-kserve-8557d588f9-chxp7" Jul 18 12:31:06.716265 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:31:06.716183 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vwv8l\" (UniqueName: \"kubernetes.io/projected/e087dc9f-74c0-4f0f-903b-758efbe22c2c-kube-api-access-vwv8l\") pod \"auth-test-model-7a27b1-kserve-8557d588f9-chxp7\" (UID: \"e087dc9f-74c0-4f0f-903b-758efbe22c2c\") " pod="ai-tenant-e2e-shared-a-7a27b1/auth-test-model-7a27b1-kserve-8557d588f9-chxp7" Jul 18 12:31:06.817424 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:31:06.817335 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vwv8l\" (UniqueName: \"kubernetes.io/projected/e087dc9f-74c0-4f0f-903b-758efbe22c2c-kube-api-access-vwv8l\") pod \"auth-test-model-7a27b1-kserve-8557d588f9-chxp7\" (UID: \"e087dc9f-74c0-4f0f-903b-758efbe22c2c\") " pod="ai-tenant-e2e-shared-a-7a27b1/auth-test-model-7a27b1-kserve-8557d588f9-chxp7" Jul 18 12:31:06.817424 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:31:06.817408 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e087dc9f-74c0-4f0f-903b-758efbe22c2c-tmp-dir\") pod \"auth-test-model-7a27b1-kserve-8557d588f9-chxp7\" (UID: \"e087dc9f-74c0-4f0f-903b-758efbe22c2c\") " pod="ai-tenant-e2e-shared-a-7a27b1/auth-test-model-7a27b1-kserve-8557d588f9-chxp7" Jul 18 12:31:06.817692 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:31:06.817446 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e087dc9f-74c0-4f0f-903b-758efbe22c2c-model-cache\") pod \"auth-test-model-7a27b1-kserve-8557d588f9-chxp7\" (UID: \"e087dc9f-74c0-4f0f-903b-758efbe22c2c\") " pod="ai-tenant-e2e-shared-a-7a27b1/auth-test-model-7a27b1-kserve-8557d588f9-chxp7" Jul 18 12:31:06.817692 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:31:06.817501 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e087dc9f-74c0-4f0f-903b-758efbe22c2c-home\") pod \"auth-test-model-7a27b1-kserve-8557d588f9-chxp7\" (UID: \"e087dc9f-74c0-4f0f-903b-758efbe22c2c\") " pod="ai-tenant-e2e-shared-a-7a27b1/auth-test-model-7a27b1-kserve-8557d588f9-chxp7" Jul 18 12:31:06.817692 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:31:06.817537 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e087dc9f-74c0-4f0f-903b-758efbe22c2c-dshm\") pod \"auth-test-model-7a27b1-kserve-8557d588f9-chxp7\" (UID: \"e087dc9f-74c0-4f0f-903b-758efbe22c2c\") " pod="ai-tenant-e2e-shared-a-7a27b1/auth-test-model-7a27b1-kserve-8557d588f9-chxp7" Jul 18 12:31:06.817854 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:31:06.817697 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e087dc9f-74c0-4f0f-903b-758efbe22c2c-tls-certs\") pod \"auth-test-model-7a27b1-kserve-8557d588f9-chxp7\" (UID: \"e087dc9f-74c0-4f0f-903b-758efbe22c2c\") " pod="ai-tenant-e2e-shared-a-7a27b1/auth-test-model-7a27b1-kserve-8557d588f9-chxp7" Jul 18 12:31:06.817907 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:31:06.817850 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e087dc9f-74c0-4f0f-903b-758efbe22c2c-tmp-dir\") pod \"auth-test-model-7a27b1-kserve-8557d588f9-chxp7\" (UID: \"e087dc9f-74c0-4f0f-903b-758efbe22c2c\") " pod="ai-tenant-e2e-shared-a-7a27b1/auth-test-model-7a27b1-kserve-8557d588f9-chxp7" Jul 18 12:31:06.817907 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:31:06.817859 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e087dc9f-74c0-4f0f-903b-758efbe22c2c-model-cache\") pod \"auth-test-model-7a27b1-kserve-8557d588f9-chxp7\" (UID: \"e087dc9f-74c0-4f0f-903b-758efbe22c2c\") " pod="ai-tenant-e2e-shared-a-7a27b1/auth-test-model-7a27b1-kserve-8557d588f9-chxp7" Jul 18 12:31:06.817907 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:31:06.817888 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e087dc9f-74c0-4f0f-903b-758efbe22c2c-home\") pod \"auth-test-model-7a27b1-kserve-8557d588f9-chxp7\" (UID: \"e087dc9f-74c0-4f0f-903b-758efbe22c2c\") " pod="ai-tenant-e2e-shared-a-7a27b1/auth-test-model-7a27b1-kserve-8557d588f9-chxp7" Jul 18 12:31:06.820046 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:31:06.820011 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e087dc9f-74c0-4f0f-903b-758efbe22c2c-dshm\") pod \"auth-test-model-7a27b1-kserve-8557d588f9-chxp7\" (UID: \"e087dc9f-74c0-4f0f-903b-758efbe22c2c\") " pod="ai-tenant-e2e-shared-a-7a27b1/auth-test-model-7a27b1-kserve-8557d588f9-chxp7" Jul 18 12:31:06.820144 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:31:06.820129 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e087dc9f-74c0-4f0f-903b-758efbe22c2c-tls-certs\") pod \"auth-test-model-7a27b1-kserve-8557d588f9-chxp7\" (UID: \"e087dc9f-74c0-4f0f-903b-758efbe22c2c\") " pod="ai-tenant-e2e-shared-a-7a27b1/auth-test-model-7a27b1-kserve-8557d588f9-chxp7" Jul 18 12:31:06.826298 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:31:06.826282 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vwv8l\" (UniqueName: \"kubernetes.io/projected/e087dc9f-74c0-4f0f-903b-758efbe22c2c-kube-api-access-vwv8l\") pod \"auth-test-model-7a27b1-kserve-8557d588f9-chxp7\" (UID: \"e087dc9f-74c0-4f0f-903b-758efbe22c2c\") " pod="ai-tenant-e2e-shared-a-7a27b1/auth-test-model-7a27b1-kserve-8557d588f9-chxp7" Jul 18 12:31:06.867666 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:31:06.867628 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-7a27b1/auth-test-model-7a27b1-kserve-8557d588f9-chxp7" Jul 18 12:31:07.002845 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:31:07.002821 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-7a27b1/auth-test-model-7a27b1-kserve-8557d588f9-chxp7"] Jul 18 12:31:07.005141 ip-10-0-136-240 kubenswrapper[2585]: W0718 12:31:07.005115 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode087dc9f_74c0_4f0f_903b_758efbe22c2c.slice/crio-523be7cae1a1adff6c8789cecf42c1ddaac0c192ecdd36cb69437bd3b713f781 WatchSource:0}: Error finding container 523be7cae1a1adff6c8789cecf42c1ddaac0c192ecdd36cb69437bd3b713f781: Status 404 returned error can't find the container with id 523be7cae1a1adff6c8789cecf42c1ddaac0c192ecdd36cb69437bd3b713f781 Jul 18 12:31:07.262677 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:31:07.262637 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-7a27b1/auth-test-model-7a27b1-kserve-8557d588f9-chxp7" event={"ID":"e087dc9f-74c0-4f0f-903b-758efbe22c2c","Type":"ContainerStarted","Data":"012aaa23df0a01085d81ea515605349f55baa40cf8ca067cb9fb60e89be35548"} Jul 18 12:31:07.262841 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:31:07.262689 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-7a27b1/auth-test-model-7a27b1-kserve-8557d588f9-chxp7" event={"ID":"e087dc9f-74c0-4f0f-903b-758efbe22c2c","Type":"ContainerStarted","Data":"523be7cae1a1adff6c8789cecf42c1ddaac0c192ecdd36cb69437bd3b713f781"} Jul 18 12:31:07.278894 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:31:07.278843 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-7a27b1/auth-test-model-7a27b1-kserve-8557d588f9-chxp7" podStartSLOduration=1.2788241870000001 podStartE2EDuration="1.278824187s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-18 12:31:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-18 12:31:07.27826709 +0000 UTC m=+2534.358473448" watchObservedRunningTime="2026-07-18 12:31:07.278824187 +0000 UTC m=+2534.359030545" Jul 18 12:31:16.868077 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:31:16.868042 2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-7a27b1/auth-test-model-7a27b1-kserve-8557d588f9-chxp7" Jul 18 12:31:16.868483 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:31:16.868096 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-7a27b1/auth-test-model-7a27b1-kserve-8557d588f9-chxp7" Jul 18 12:31:16.880339 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:31:16.880314 2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-7a27b1/auth-test-model-7a27b1-kserve-8557d588f9-chxp7" Jul 18 12:31:17.322020 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:31:17.321996 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-7a27b1/auth-test-model-7a27b1-kserve-8557d588f9-chxp7" Jul 18 12:33:28.616223 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:33:28.616190 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-7a27b1/auth-test-model-7a27b1-kserve-8557d588f9-chxp7"] Jul 18 12:33:43.637511 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:33:43.637462 2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-7a27b1/auth-test-model-7a27b1-kserve-8557d588f9-chxp7" podUID="e087dc9f-74c0-4f0f-903b-758efbe22c2c" containerName="main" containerID="cri-o://012aaa23df0a01085d81ea515605349f55baa40cf8ca067cb9fb60e89be35548" gracePeriod=45 Jul 18 12:33:43.892964 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:33:43.892906 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-7a27b1/auth-test-model-7a27b1-kserve-8557d588f9-chxp7" Jul 18 12:33:44.049313 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:33:44.049278 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e087dc9f-74c0-4f0f-903b-758efbe22c2c-tmp-dir\") pod \"e087dc9f-74c0-4f0f-903b-758efbe22c2c\" (UID: \"e087dc9f-74c0-4f0f-903b-758efbe22c2c\") " Jul 18 12:33:44.049502 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:33:44.049348 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e087dc9f-74c0-4f0f-903b-758efbe22c2c-dshm\") pod \"e087dc9f-74c0-4f0f-903b-758efbe22c2c\" (UID: \"e087dc9f-74c0-4f0f-903b-758efbe22c2c\") " Jul 18 12:33:44.049502 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:33:44.049367 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vwv8l\" (UniqueName: \"kubernetes.io/projected/e087dc9f-74c0-4f0f-903b-758efbe22c2c-kube-api-access-vwv8l\") pod \"e087dc9f-74c0-4f0f-903b-758efbe22c2c\" (UID: \"e087dc9f-74c0-4f0f-903b-758efbe22c2c\") " Jul 18 12:33:44.049502 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:33:44.049402 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e087dc9f-74c0-4f0f-903b-758efbe22c2c-tls-certs\") pod \"e087dc9f-74c0-4f0f-903b-758efbe22c2c\" (UID: \"e087dc9f-74c0-4f0f-903b-758efbe22c2c\") " Jul 18 12:33:44.049502 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:33:44.049450 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e087dc9f-74c0-4f0f-903b-758efbe22c2c-home\") pod \"e087dc9f-74c0-4f0f-903b-758efbe22c2c\" (UID: \"e087dc9f-74c0-4f0f-903b-758efbe22c2c\") " Jul 18 12:33:44.049758 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:33:44.049604 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e087dc9f-74c0-4f0f-903b-758efbe22c2c-model-cache\") pod \"e087dc9f-74c0-4f0f-903b-758efbe22c2c\" (UID: \"e087dc9f-74c0-4f0f-903b-758efbe22c2c\") " Jul 18 12:33:44.049758 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:33:44.049635 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e087dc9f-74c0-4f0f-903b-758efbe22c2c-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "e087dc9f-74c0-4f0f-903b-758efbe22c2c" (UID: "e087dc9f-74c0-4f0f-903b-758efbe22c2c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 18 12:33:44.049758 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:33:44.049730 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e087dc9f-74c0-4f0f-903b-758efbe22c2c-home" (OuterVolumeSpecName: "home") pod "e087dc9f-74c0-4f0f-903b-758efbe22c2c" (UID: "e087dc9f-74c0-4f0f-903b-758efbe22c2c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 18 12:33:44.049932 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:33:44.049910 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e087dc9f-74c0-4f0f-903b-758efbe22c2c-model-cache" (OuterVolumeSpecName: "model-cache") pod "e087dc9f-74c0-4f0f-903b-758efbe22c2c" (UID: "e087dc9f-74c0-4f0f-903b-758efbe22c2c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 18 12:33:44.050001 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:33:44.049984 2585 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e087dc9f-74c0-4f0f-903b-758efbe22c2c-home\") on node \"ip-10-0-136-240.ec2.internal\" DevicePath \"\"" Jul 18 12:33:44.050059 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:33:44.050007 2585 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e087dc9f-74c0-4f0f-903b-758efbe22c2c-model-cache\") on node \"ip-10-0-136-240.ec2.internal\" DevicePath \"\"" Jul 18 12:33:44.050059 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:33:44.050023 2585 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e087dc9f-74c0-4f0f-903b-758efbe22c2c-tmp-dir\") on node \"ip-10-0-136-240.ec2.internal\" DevicePath \"\"" Jul 18 12:33:44.051701 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:33:44.051674 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e087dc9f-74c0-4f0f-903b-758efbe22c2c-dshm" (OuterVolumeSpecName: "dshm") pod "e087dc9f-74c0-4f0f-903b-758efbe22c2c" (UID: "e087dc9f-74c0-4f0f-903b-758efbe22c2c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 18 12:33:44.051842 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:33:44.051813 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e087dc9f-74c0-4f0f-903b-758efbe22c2c-kube-api-access-vwv8l" (OuterVolumeSpecName: "kube-api-access-vwv8l") pod "e087dc9f-74c0-4f0f-903b-758efbe22c2c" (UID: "e087dc9f-74c0-4f0f-903b-758efbe22c2c"). InnerVolumeSpecName "kube-api-access-vwv8l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 18 12:33:44.051842 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:33:44.051828 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e087dc9f-74c0-4f0f-903b-758efbe22c2c-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "e087dc9f-74c0-4f0f-903b-758efbe22c2c" (UID: "e087dc9f-74c0-4f0f-903b-758efbe22c2c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 18 12:33:44.067300 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:33:44.067274 2585 generic.go:356] "Generic (PLEG): container finished" podID="e087dc9f-74c0-4f0f-903b-758efbe22c2c" containerID="012aaa23df0a01085d81ea515605349f55baa40cf8ca067cb9fb60e89be35548" exitCode=0 Jul 18 12:33:44.067399 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:33:44.067320 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-7a27b1/auth-test-model-7a27b1-kserve-8557d588f9-chxp7" Jul 18 12:33:44.067399 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:33:44.067348 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-7a27b1/auth-test-model-7a27b1-kserve-8557d588f9-chxp7" event={"ID":"e087dc9f-74c0-4f0f-903b-758efbe22c2c","Type":"ContainerDied","Data":"012aaa23df0a01085d81ea515605349f55baa40cf8ca067cb9fb60e89be35548"} Jul 18 12:33:44.067399 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:33:44.067389 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-7a27b1/auth-test-model-7a27b1-kserve-8557d588f9-chxp7" event={"ID":"e087dc9f-74c0-4f0f-903b-758efbe22c2c","Type":"ContainerDied","Data":"523be7cae1a1adff6c8789cecf42c1ddaac0c192ecdd36cb69437bd3b713f781"} Jul 18 12:33:44.067511 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:33:44.067406 2585 scope.go:117] "RemoveContainer" containerID="012aaa23df0a01085d81ea515605349f55baa40cf8ca067cb9fb60e89be35548" Jul 18 12:33:44.077056 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:33:44.077039 2585 scope.go:117] "RemoveContainer" containerID="012aaa23df0a01085d81ea515605349f55baa40cf8ca067cb9fb60e89be35548" Jul 18 12:33:44.077337 ip-10-0-136-240 kubenswrapper[2585]: E0718 12:33:44.077319 2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"012aaa23df0a01085d81ea515605349f55baa40cf8ca067cb9fb60e89be35548\": container with ID starting with 012aaa23df0a01085d81ea515605349f55baa40cf8ca067cb9fb60e89be35548 not found: ID does not exist" containerID="012aaa23df0a01085d81ea515605349f55baa40cf8ca067cb9fb60e89be35548" Jul 18 12:33:44.077386 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:33:44.077360 2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"012aaa23df0a01085d81ea515605349f55baa40cf8ca067cb9fb60e89be35548"} err="failed to get container status \"012aaa23df0a01085d81ea515605349f55baa40cf8ca067cb9fb60e89be35548\": rpc error: code = NotFound desc = could not find container \"012aaa23df0a01085d81ea515605349f55baa40cf8ca067cb9fb60e89be35548\": container with ID starting with 012aaa23df0a01085d81ea515605349f55baa40cf8ca067cb9fb60e89be35548 not found: ID does not exist" Jul 18 12:33:44.089295 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:33:44.089273 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-7a27b1/auth-test-model-7a27b1-kserve-8557d588f9-chxp7"] Jul 18 12:33:44.093286 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:33:44.093263 2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-7a27b1/auth-test-model-7a27b1-kserve-8557d588f9-chxp7"] Jul 18 12:33:44.150905 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:33:44.150836 2585 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e087dc9f-74c0-4f0f-903b-758efbe22c2c-dshm\") on node \"ip-10-0-136-240.ec2.internal\" DevicePath \"\"" Jul 18 12:33:44.150905 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:33:44.150856 2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vwv8l\" (UniqueName: \"kubernetes.io/projected/e087dc9f-74c0-4f0f-903b-758efbe22c2c-kube-api-access-vwv8l\") on node \"ip-10-0-136-240.ec2.internal\" DevicePath \"\"" Jul 18 12:33:44.150905 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:33:44.150866 2585 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e087dc9f-74c0-4f0f-903b-758efbe22c2c-tls-certs\") on node \"ip-10-0-136-240.ec2.internal\" DevicePath \"\"" Jul 18 12:33:44.850997 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:33:44.850958 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-7a27b1/sub-test-model-7a27b1-kserve-76fcf5655f-5z4bc"] Jul 18 12:33:44.851534 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:33:44.851513 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e087dc9f-74c0-4f0f-903b-758efbe22c2c" containerName="main" Jul 18 12:33:44.851672 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:33:44.851539 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="e087dc9f-74c0-4f0f-903b-758efbe22c2c" containerName="main" Jul 18 12:33:44.851732 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:33:44.851694 2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="e087dc9f-74c0-4f0f-903b-758efbe22c2c" containerName="main" Jul 18 12:33:44.863023 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:33:44.862991 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-7a27b1/sub-test-model-7a27b1-kserve-76fcf5655f-5z4bc"] Jul 18 12:33:44.863180 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:33:44.863077 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-7a27b1/sub-test-model-7a27b1-kserve-76fcf5655f-5z4bc" Jul 18 12:33:44.866797 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:33:44.866770 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-7a27b1\"/\"default-dockercfg-8lz4m\"" Jul 18 12:33:44.866938 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:33:44.866770 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-7a27b1\"/\"openshift-service-ca.crt\"" Jul 18 12:33:44.866938 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:33:44.866783 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-7a27b1\"/\"kube-root-ca.crt\"" Jul 18 12:33:44.866938 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:33:44.866830 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-7a27b1\"/\"sub-test-model-7a27b1-kserve-self-signed-certs\"" Jul 18 12:33:44.959930 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:33:44.959899 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b9a7e6f4-f660-46bf-b119-a3064ae950e1-model-cache\") pod \"sub-test-model-7a27b1-kserve-76fcf5655f-5z4bc\" (UID: \"b9a7e6f4-f660-46bf-b119-a3064ae950e1\") " pod="ai-tenant-e2e-shared-a-7a27b1/sub-test-model-7a27b1-kserve-76fcf5655f-5z4bc" Jul 18 12:33:44.960086 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:33:44.959942 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g265b\" (UniqueName: \"kubernetes.io/projected/b9a7e6f4-f660-46bf-b119-a3064ae950e1-kube-api-access-g265b\") pod \"sub-test-model-7a27b1-kserve-76fcf5655f-5z4bc\" (UID: \"b9a7e6f4-f660-46bf-b119-a3064ae950e1\") " pod="ai-tenant-e2e-shared-a-7a27b1/sub-test-model-7a27b1-kserve-76fcf5655f-5z4bc" Jul 18 12:33:44.960086 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:33:44.959999 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b9a7e6f4-f660-46bf-b119-a3064ae950e1-tmp-dir\") pod \"sub-test-model-7a27b1-kserve-76fcf5655f-5z4bc\" (UID: \"b9a7e6f4-f660-46bf-b119-a3064ae950e1\") " pod="ai-tenant-e2e-shared-a-7a27b1/sub-test-model-7a27b1-kserve-76fcf5655f-5z4bc" Jul 18 12:33:44.960086 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:33:44.960038 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b9a7e6f4-f660-46bf-b119-a3064ae950e1-tls-certs\") pod \"sub-test-model-7a27b1-kserve-76fcf5655f-5z4bc\" (UID: \"b9a7e6f4-f660-46bf-b119-a3064ae950e1\") " pod="ai-tenant-e2e-shared-a-7a27b1/sub-test-model-7a27b1-kserve-76fcf5655f-5z4bc" Jul 18 12:33:44.960086 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:33:44.960066 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b9a7e6f4-f660-46bf-b119-a3064ae950e1-home\") pod \"sub-test-model-7a27b1-kserve-76fcf5655f-5z4bc\" (UID: \"b9a7e6f4-f660-46bf-b119-a3064ae950e1\") " pod="ai-tenant-e2e-shared-a-7a27b1/sub-test-model-7a27b1-kserve-76fcf5655f-5z4bc" Jul 18 12:33:44.960231 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:33:44.960139 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b9a7e6f4-f660-46bf-b119-a3064ae950e1-dshm\") pod \"sub-test-model-7a27b1-kserve-76fcf5655f-5z4bc\" (UID: \"b9a7e6f4-f660-46bf-b119-a3064ae950e1\") " pod="ai-tenant-e2e-shared-a-7a27b1/sub-test-model-7a27b1-kserve-76fcf5655f-5z4bc" Jul 18 12:33:45.061105 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:33:45.061065 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b9a7e6f4-f660-46bf-b119-a3064ae950e1-tmp-dir\") pod \"sub-test-model-7a27b1-kserve-76fcf5655f-5z4bc\" (UID: \"b9a7e6f4-f660-46bf-b119-a3064ae950e1\") " pod="ai-tenant-e2e-shared-a-7a27b1/sub-test-model-7a27b1-kserve-76fcf5655f-5z4bc" Jul 18 12:33:45.061293 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:33:45.061123 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b9a7e6f4-f660-46bf-b119-a3064ae950e1-tls-certs\") pod \"sub-test-model-7a27b1-kserve-76fcf5655f-5z4bc\" (UID: \"b9a7e6f4-f660-46bf-b119-a3064ae950e1\") " pod="ai-tenant-e2e-shared-a-7a27b1/sub-test-model-7a27b1-kserve-76fcf5655f-5z4bc" Jul 18 12:33:45.061293 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:33:45.061160 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b9a7e6f4-f660-46bf-b119-a3064ae950e1-home\") pod \"sub-test-model-7a27b1-kserve-76fcf5655f-5z4bc\" (UID: \"b9a7e6f4-f660-46bf-b119-a3064ae950e1\") " pod="ai-tenant-e2e-shared-a-7a27b1/sub-test-model-7a27b1-kserve-76fcf5655f-5z4bc" Jul 18 12:33:45.061293 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:33:45.061215 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b9a7e6f4-f660-46bf-b119-a3064ae950e1-dshm\") pod \"sub-test-model-7a27b1-kserve-76fcf5655f-5z4bc\" (UID: \"b9a7e6f4-f660-46bf-b119-a3064ae950e1\") " pod="ai-tenant-e2e-shared-a-7a27b1/sub-test-model-7a27b1-kserve-76fcf5655f-5z4bc" Jul 18 12:33:45.061469 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:33:45.061291 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b9a7e6f4-f660-46bf-b119-a3064ae950e1-model-cache\") pod \"sub-test-model-7a27b1-kserve-76fcf5655f-5z4bc\" (UID: \"b9a7e6f4-f660-46bf-b119-a3064ae950e1\") " pod="ai-tenant-e2e-shared-a-7a27b1/sub-test-model-7a27b1-kserve-76fcf5655f-5z4bc" Jul 18 12:33:45.061469 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:33:45.061332 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g265b\" (UniqueName: \"kubernetes.io/projected/b9a7e6f4-f660-46bf-b119-a3064ae950e1-kube-api-access-g265b\") pod \"sub-test-model-7a27b1-kserve-76fcf5655f-5z4bc\" (UID: \"b9a7e6f4-f660-46bf-b119-a3064ae950e1\") " pod="ai-tenant-e2e-shared-a-7a27b1/sub-test-model-7a27b1-kserve-76fcf5655f-5z4bc" Jul 18 12:33:45.061620 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:33:45.061503 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b9a7e6f4-f660-46bf-b119-a3064ae950e1-tmp-dir\") pod \"sub-test-model-7a27b1-kserve-76fcf5655f-5z4bc\" (UID: \"b9a7e6f4-f660-46bf-b119-a3064ae950e1\") " pod="ai-tenant-e2e-shared-a-7a27b1/sub-test-model-7a27b1-kserve-76fcf5655f-5z4bc" Jul 18 12:33:45.061620 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:33:45.061523 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b9a7e6f4-f660-46bf-b119-a3064ae950e1-home\") pod \"sub-test-model-7a27b1-kserve-76fcf5655f-5z4bc\" (UID: \"b9a7e6f4-f660-46bf-b119-a3064ae950e1\") " pod="ai-tenant-e2e-shared-a-7a27b1/sub-test-model-7a27b1-kserve-76fcf5655f-5z4bc" Jul 18 12:33:45.061726 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:33:45.061640 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b9a7e6f4-f660-46bf-b119-a3064ae950e1-model-cache\") pod \"sub-test-model-7a27b1-kserve-76fcf5655f-5z4bc\" (UID: \"b9a7e6f4-f660-46bf-b119-a3064ae950e1\") " pod="ai-tenant-e2e-shared-a-7a27b1/sub-test-model-7a27b1-kserve-76fcf5655f-5z4bc" Jul 18 12:33:45.063765 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:33:45.063739 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b9a7e6f4-f660-46bf-b119-a3064ae950e1-dshm\") pod \"sub-test-model-7a27b1-kserve-76fcf5655f-5z4bc\" (UID: \"b9a7e6f4-f660-46bf-b119-a3064ae950e1\") " pod="ai-tenant-e2e-shared-a-7a27b1/sub-test-model-7a27b1-kserve-76fcf5655f-5z4bc" Jul 18 12:33:45.063956 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:33:45.063941 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b9a7e6f4-f660-46bf-b119-a3064ae950e1-tls-certs\") pod \"sub-test-model-7a27b1-kserve-76fcf5655f-5z4bc\" (UID: \"b9a7e6f4-f660-46bf-b119-a3064ae950e1\") " pod="ai-tenant-e2e-shared-a-7a27b1/sub-test-model-7a27b1-kserve-76fcf5655f-5z4bc" Jul 18 12:33:45.070285 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:33:45.070265 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g265b\" (UniqueName: \"kubernetes.io/projected/b9a7e6f4-f660-46bf-b119-a3064ae950e1-kube-api-access-g265b\") pod \"sub-test-model-7a27b1-kserve-76fcf5655f-5z4bc\" (UID: \"b9a7e6f4-f660-46bf-b119-a3064ae950e1\") " pod="ai-tenant-e2e-shared-a-7a27b1/sub-test-model-7a27b1-kserve-76fcf5655f-5z4bc" Jul 18 12:33:45.173855 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:33:45.173762 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-7a27b1/sub-test-model-7a27b1-kserve-76fcf5655f-5z4bc" Jul 18 12:33:45.313732 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:33:45.313702 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-7a27b1/sub-test-model-7a27b1-kserve-76fcf5655f-5z4bc"] Jul 18 12:33:45.314505 ip-10-0-136-240 kubenswrapper[2585]: W0718 12:33:45.314479 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb9a7e6f4_f660_46bf_b119_a3064ae950e1.slice/crio-dc93120b2bdb78c3a00faa50cb4381b85df877af8a8ff4dfcb2eb1d0ead5e809 WatchSource:0}: Error finding container dc93120b2bdb78c3a00faa50cb4381b85df877af8a8ff4dfcb2eb1d0ead5e809: Status 404 returned error can't find the container with id dc93120b2bdb78c3a00faa50cb4381b85df877af8a8ff4dfcb2eb1d0ead5e809 Jul 18 12:33:45.469429 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:33:45.469395 2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e087dc9f-74c0-4f0f-903b-758efbe22c2c" path="/var/lib/kubelet/pods/e087dc9f-74c0-4f0f-903b-758efbe22c2c/volumes" Jul 18 12:33:46.081601 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:33:46.081539 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-7a27b1/sub-test-model-7a27b1-kserve-76fcf5655f-5z4bc" event={"ID":"b9a7e6f4-f660-46bf-b119-a3064ae950e1","Type":"ContainerStarted","Data":"2cc44dbcc4c769cab541a427286a69a487c981403d1ed8a3807204f61ef76011"} Jul 18 12:33:46.081958 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:33:46.081617 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-7a27b1/sub-test-model-7a27b1-kserve-76fcf5655f-5z4bc" event={"ID":"b9a7e6f4-f660-46bf-b119-a3064ae950e1","Type":"ContainerStarted","Data":"dc93120b2bdb78c3a00faa50cb4381b85df877af8a8ff4dfcb2eb1d0ead5e809"} Jul 18 12:33:53.617087 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:33:53.617062 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-j9l7p_8ec47eff-5fbc-4130-baef-d8544aa4e786/console-operator/1.log" Jul 18 12:33:53.627138 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:33:53.627117 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-wq4lg_247c97c8-982b-4176-ae10-afc396745360/ovn-acl-logging/0.log" Jul 18 12:33:53.630433 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:33:53.630407 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-j9l7p_8ec47eff-5fbc-4130-baef-d8544aa4e786/console-operator/1.log" Jul 18 12:33:53.637162 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:33:53.637146 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-wq4lg_247c97c8-982b-4176-ae10-afc396745360/ovn-acl-logging/0.log" Jul 18 12:33:55.174648 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:33:55.174615 2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-7a27b1/sub-test-model-7a27b1-kserve-76fcf5655f-5z4bc" Jul 18 12:33:55.175023 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:33:55.174673 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-7a27b1/sub-test-model-7a27b1-kserve-76fcf5655f-5z4bc" Jul 18 12:33:55.187373 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:33:55.187350 2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-7a27b1/sub-test-model-7a27b1-kserve-76fcf5655f-5z4bc" Jul 18 12:33:55.206140 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:33:55.206073 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-7a27b1/sub-test-model-7a27b1-kserve-76fcf5655f-5z4bc" podStartSLOduration=11.206056930999999 podStartE2EDuration="11.206056931s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-18 12:33:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-18 12:33:46.099798821 +0000 UTC m=+2693.180005179" watchObservedRunningTime="2026-07-18 12:33:55.206056931 +0000 UTC m=+2702.286263291" Jul 18 12:33:56.152526 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:33:56.152494 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-7a27b1/sub-test-model-7a27b1-kserve-76fcf5655f-5z4bc" Jul 18 12:35:04.674773 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:35:04.674736 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-7a27b1/sub-test-model-7a27b1-kserve-76fcf5655f-5z4bc"] Jul 18 12:35:19.697674 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:35:19.697591 2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-7a27b1/sub-test-model-7a27b1-kserve-76fcf5655f-5z4bc" podUID="b9a7e6f4-f660-46bf-b119-a3064ae950e1" containerName="main" containerID="cri-o://2cc44dbcc4c769cab541a427286a69a487c981403d1ed8a3807204f61ef76011" gracePeriod=45 Jul 18 12:35:19.954717 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:35:19.954656 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-7a27b1/sub-test-model-7a27b1-kserve-76fcf5655f-5z4bc" Jul 18 12:35:20.092510 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:35:20.092480 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b9a7e6f4-f660-46bf-b119-a3064ae950e1-tls-certs\") pod \"b9a7e6f4-f660-46bf-b119-a3064ae950e1\" (UID: \"b9a7e6f4-f660-46bf-b119-a3064ae950e1\") " Jul 18 12:35:20.092697 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:35:20.092538 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b9a7e6f4-f660-46bf-b119-a3064ae950e1-tmp-dir\") pod \"b9a7e6f4-f660-46bf-b119-a3064ae950e1\" (UID: \"b9a7e6f4-f660-46bf-b119-a3064ae950e1\") " Jul 18 12:35:20.092697 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:35:20.092589 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b9a7e6f4-f660-46bf-b119-a3064ae950e1-dshm\") pod \"b9a7e6f4-f660-46bf-b119-a3064ae950e1\" (UID: \"b9a7e6f4-f660-46bf-b119-a3064ae950e1\") " Jul 18 12:35:20.092697 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:35:20.092649 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b9a7e6f4-f660-46bf-b119-a3064ae950e1-model-cache\") pod \"b9a7e6f4-f660-46bf-b119-a3064ae950e1\" (UID: \"b9a7e6f4-f660-46bf-b119-a3064ae950e1\") " Jul 18 12:35:20.092824 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:35:20.092703 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-g265b\" (UniqueName: \"kubernetes.io/projected/b9a7e6f4-f660-46bf-b119-a3064ae950e1-kube-api-access-g265b\") pod \"b9a7e6f4-f660-46bf-b119-a3064ae950e1\" (UID: \"b9a7e6f4-f660-46bf-b119-a3064ae950e1\") " Jul 18 12:35:20.092824 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:35:20.092725 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b9a7e6f4-f660-46bf-b119-a3064ae950e1-home\") pod \"b9a7e6f4-f660-46bf-b119-a3064ae950e1\" (UID: \"b9a7e6f4-f660-46bf-b119-a3064ae950e1\") " Jul 18 12:35:20.092916 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:35:20.092853 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b9a7e6f4-f660-46bf-b119-a3064ae950e1-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "b9a7e6f4-f660-46bf-b119-a3064ae950e1" (UID: "b9a7e6f4-f660-46bf-b119-a3064ae950e1"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 18 12:35:20.093015 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:35:20.092988 2585 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b9a7e6f4-f660-46bf-b119-a3064ae950e1-tmp-dir\") on node \"ip-10-0-136-240.ec2.internal\" DevicePath \"\"" Jul 18 12:35:20.093015 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:35:20.092995 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b9a7e6f4-f660-46bf-b119-a3064ae950e1-model-cache" (OuterVolumeSpecName: "model-cache") pod "b9a7e6f4-f660-46bf-b119-a3064ae950e1" (UID: "b9a7e6f4-f660-46bf-b119-a3064ae950e1"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 18 12:35:20.093180 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:35:20.093117 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b9a7e6f4-f660-46bf-b119-a3064ae950e1-home" (OuterVolumeSpecName: "home") pod "b9a7e6f4-f660-46bf-b119-a3064ae950e1" (UID: "b9a7e6f4-f660-46bf-b119-a3064ae950e1"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 18 12:35:20.094914 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:35:20.094894 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b9a7e6f4-f660-46bf-b119-a3064ae950e1-dshm" (OuterVolumeSpecName: "dshm") pod "b9a7e6f4-f660-46bf-b119-a3064ae950e1" (UID: "b9a7e6f4-f660-46bf-b119-a3064ae950e1"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 18 12:35:20.095020 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:35:20.094932 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b9a7e6f4-f660-46bf-b119-a3064ae950e1-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "b9a7e6f4-f660-46bf-b119-a3064ae950e1" (UID: "b9a7e6f4-f660-46bf-b119-a3064ae950e1"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 18 12:35:20.095020 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:35:20.094964 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b9a7e6f4-f660-46bf-b119-a3064ae950e1-kube-api-access-g265b" (OuterVolumeSpecName: "kube-api-access-g265b") pod "b9a7e6f4-f660-46bf-b119-a3064ae950e1" (UID: "b9a7e6f4-f660-46bf-b119-a3064ae950e1"). InnerVolumeSpecName "kube-api-access-g265b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 18 12:35:20.193582 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:35:20.193529 2585 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b9a7e6f4-f660-46bf-b119-a3064ae950e1-model-cache\") on node \"ip-10-0-136-240.ec2.internal\" DevicePath \"\"" Jul 18 12:35:20.193582 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:35:20.193580 2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g265b\" (UniqueName: \"kubernetes.io/projected/b9a7e6f4-f660-46bf-b119-a3064ae950e1-kube-api-access-g265b\") on node \"ip-10-0-136-240.ec2.internal\" DevicePath \"\"" Jul 18 12:35:20.193785 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:35:20.193592 2585 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b9a7e6f4-f660-46bf-b119-a3064ae950e1-home\") on node \"ip-10-0-136-240.ec2.internal\" DevicePath \"\"" Jul 18 12:35:20.193785 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:35:20.193600 2585 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b9a7e6f4-f660-46bf-b119-a3064ae950e1-tls-certs\") on node \"ip-10-0-136-240.ec2.internal\" DevicePath \"\"" Jul 18 12:35:20.193785 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:35:20.193609 2585 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b9a7e6f4-f660-46bf-b119-a3064ae950e1-dshm\") on node \"ip-10-0-136-240.ec2.internal\" DevicePath \"\"" Jul 18 12:35:20.575952 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:35:20.575918 2585 generic.go:356] "Generic (PLEG): container finished" podID="b9a7e6f4-f660-46bf-b119-a3064ae950e1" containerID="2cc44dbcc4c769cab541a427286a69a487c981403d1ed8a3807204f61ef76011" exitCode=0 Jul 18 12:35:20.576109 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:35:20.575966 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-7a27b1/sub-test-model-7a27b1-kserve-76fcf5655f-5z4bc" Jul 18 12:35:20.576109 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:35:20.575971 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-7a27b1/sub-test-model-7a27b1-kserve-76fcf5655f-5z4bc" event={"ID":"b9a7e6f4-f660-46bf-b119-a3064ae950e1","Type":"ContainerDied","Data":"2cc44dbcc4c769cab541a427286a69a487c981403d1ed8a3807204f61ef76011"} Jul 18 12:35:20.576109 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:35:20.576001 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-7a27b1/sub-test-model-7a27b1-kserve-76fcf5655f-5z4bc" event={"ID":"b9a7e6f4-f660-46bf-b119-a3064ae950e1","Type":"ContainerDied","Data":"dc93120b2bdb78c3a00faa50cb4381b85df877af8a8ff4dfcb2eb1d0ead5e809"} Jul 18 12:35:20.576109 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:35:20.576017 2585 scope.go:117] "RemoveContainer" containerID="2cc44dbcc4c769cab541a427286a69a487c981403d1ed8a3807204f61ef76011" Jul 18 12:35:20.585680 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:35:20.585664 2585 scope.go:117] "RemoveContainer" containerID="2cc44dbcc4c769cab541a427286a69a487c981403d1ed8a3807204f61ef76011" Jul 18 12:35:20.585929 ip-10-0-136-240 kubenswrapper[2585]: E0718 12:35:20.585910 2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2cc44dbcc4c769cab541a427286a69a487c981403d1ed8a3807204f61ef76011\": container with ID starting with 2cc44dbcc4c769cab541a427286a69a487c981403d1ed8a3807204f61ef76011 not found: ID does not exist" containerID="2cc44dbcc4c769cab541a427286a69a487c981403d1ed8a3807204f61ef76011" Jul 18 12:35:20.585983 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:35:20.585938 2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2cc44dbcc4c769cab541a427286a69a487c981403d1ed8a3807204f61ef76011"} err="failed to get container status \"2cc44dbcc4c769cab541a427286a69a487c981403d1ed8a3807204f61ef76011\": rpc error: code = NotFound desc = could not find container \"2cc44dbcc4c769cab541a427286a69a487c981403d1ed8a3807204f61ef76011\": container with ID starting with 2cc44dbcc4c769cab541a427286a69a487c981403d1ed8a3807204f61ef76011 not found: ID does not exist" Jul 18 12:35:20.601624 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:35:20.601598 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-7a27b1/sub-test-model-7a27b1-kserve-76fcf5655f-5z4bc"] Jul 18 12:35:20.605106 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:35:20.605084 2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-7a27b1/sub-test-model-7a27b1-kserve-76fcf5655f-5z4bc"] Jul 18 12:35:21.470336 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:35:21.470305 2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b9a7e6f4-f660-46bf-b119-a3064ae950e1" path="/var/lib/kubelet/pods/b9a7e6f4-f660-46bf-b119-a3064ae950e1/volumes" Jul 18 12:35:27.929312 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:35:27.929277 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-7a27b1/rate-test-model-7a27b1-kserve-5586589fc9-tmjf8"] Jul 18 12:35:27.929879 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:35:27.929832 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b9a7e6f4-f660-46bf-b119-a3064ae950e1" containerName="main" Jul 18 12:35:27.929879 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:35:27.929852 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="b9a7e6f4-f660-46bf-b119-a3064ae950e1" containerName="main" Jul 18 12:35:27.929998 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:35:27.929968 2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="b9a7e6f4-f660-46bf-b119-a3064ae950e1" containerName="main" Jul 18 12:35:27.934282 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:35:27.934265 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-7a27b1/rate-test-model-7a27b1-kserve-5586589fc9-tmjf8" Jul 18 12:35:27.937597 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:35:27.937574 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-7a27b1\"/\"rate-test-model-7a27b1-kserve-self-signed-certs\"" Jul 18 12:35:27.937718 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:35:27.937635 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-7a27b1\"/\"kube-root-ca.crt\"" Jul 18 12:35:27.937718 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:35:27.937574 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-7a27b1\"/\"default-dockercfg-8lz4m\"" Jul 18 12:35:27.937718 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:35:27.937582 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-7a27b1\"/\"openshift-service-ca.crt\"" Jul 18 12:35:27.943915 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:35:27.943893 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-7a27b1/rate-test-model-7a27b1-kserve-5586589fc9-tmjf8"] Jul 18 12:35:28.075012 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:35:28.074979 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0a77a318-9cc4-4d3b-bd0d-669e7c4a8d60-tls-certs\") pod \"rate-test-model-7a27b1-kserve-5586589fc9-tmjf8\" (UID: \"0a77a318-9cc4-4d3b-bd0d-669e7c4a8d60\") " pod="ai-tenant-e2e-shared-a-7a27b1/rate-test-model-7a27b1-kserve-5586589fc9-tmjf8" Jul 18 12:35:28.075148 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:35:28.075045 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0a77a318-9cc4-4d3b-bd0d-669e7c4a8d60-tmp-dir\") pod \"rate-test-model-7a27b1-kserve-5586589fc9-tmjf8\" (UID: \"0a77a318-9cc4-4d3b-bd0d-669e7c4a8d60\") " pod="ai-tenant-e2e-shared-a-7a27b1/rate-test-model-7a27b1-kserve-5586589fc9-tmjf8" Jul 18 12:35:28.075148 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:35:28.075079 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lmgfx\" (UniqueName: \"kubernetes.io/projected/0a77a318-9cc4-4d3b-bd0d-669e7c4a8d60-kube-api-access-lmgfx\") pod \"rate-test-model-7a27b1-kserve-5586589fc9-tmjf8\" (UID: \"0a77a318-9cc4-4d3b-bd0d-669e7c4a8d60\") " pod="ai-tenant-e2e-shared-a-7a27b1/rate-test-model-7a27b1-kserve-5586589fc9-tmjf8" Jul 18 12:35:28.075148 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:35:28.075111 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0a77a318-9cc4-4d3b-bd0d-669e7c4a8d60-dshm\") pod \"rate-test-model-7a27b1-kserve-5586589fc9-tmjf8\" (UID: \"0a77a318-9cc4-4d3b-bd0d-669e7c4a8d60\") " pod="ai-tenant-e2e-shared-a-7a27b1/rate-test-model-7a27b1-kserve-5586589fc9-tmjf8" Jul 18 12:35:28.075148 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:35:28.075135 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0a77a318-9cc4-4d3b-bd0d-669e7c4a8d60-model-cache\") pod \"rate-test-model-7a27b1-kserve-5586589fc9-tmjf8\" (UID: \"0a77a318-9cc4-4d3b-bd0d-669e7c4a8d60\") " pod="ai-tenant-e2e-shared-a-7a27b1/rate-test-model-7a27b1-kserve-5586589fc9-tmjf8" Jul 18 12:35:28.075302 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:35:28.075184 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0a77a318-9cc4-4d3b-bd0d-669e7c4a8d60-home\") pod \"rate-test-model-7a27b1-kserve-5586589fc9-tmjf8\" (UID: \"0a77a318-9cc4-4d3b-bd0d-669e7c4a8d60\") " pod="ai-tenant-e2e-shared-a-7a27b1/rate-test-model-7a27b1-kserve-5586589fc9-tmjf8" Jul 18 12:35:28.176157 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:35:28.176121 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0a77a318-9cc4-4d3b-bd0d-669e7c4a8d60-tls-certs\") pod \"rate-test-model-7a27b1-kserve-5586589fc9-tmjf8\" (UID: \"0a77a318-9cc4-4d3b-bd0d-669e7c4a8d60\") " pod="ai-tenant-e2e-shared-a-7a27b1/rate-test-model-7a27b1-kserve-5586589fc9-tmjf8" Jul 18 12:35:28.176330 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:35:28.176202 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0a77a318-9cc4-4d3b-bd0d-669e7c4a8d60-tmp-dir\") pod \"rate-test-model-7a27b1-kserve-5586589fc9-tmjf8\" (UID: \"0a77a318-9cc4-4d3b-bd0d-669e7c4a8d60\") " pod="ai-tenant-e2e-shared-a-7a27b1/rate-test-model-7a27b1-kserve-5586589fc9-tmjf8" Jul 18 12:35:28.176330 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:35:28.176226 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lmgfx\" (UniqueName: \"kubernetes.io/projected/0a77a318-9cc4-4d3b-bd0d-669e7c4a8d60-kube-api-access-lmgfx\") pod \"rate-test-model-7a27b1-kserve-5586589fc9-tmjf8\" (UID: \"0a77a318-9cc4-4d3b-bd0d-669e7c4a8d60\") " pod="ai-tenant-e2e-shared-a-7a27b1/rate-test-model-7a27b1-kserve-5586589fc9-tmjf8" Jul 18 12:35:28.176330 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:35:28.176275 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0a77a318-9cc4-4d3b-bd0d-669e7c4a8d60-dshm\") pod \"rate-test-model-7a27b1-kserve-5586589fc9-tmjf8\" (UID: \"0a77a318-9cc4-4d3b-bd0d-669e7c4a8d60\") " pod="ai-tenant-e2e-shared-a-7a27b1/rate-test-model-7a27b1-kserve-5586589fc9-tmjf8" Jul 18 12:35:28.176330 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:35:28.176294 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0a77a318-9cc4-4d3b-bd0d-669e7c4a8d60-model-cache\") pod \"rate-test-model-7a27b1-kserve-5586589fc9-tmjf8\" (UID: \"0a77a318-9cc4-4d3b-bd0d-669e7c4a8d60\") " pod="ai-tenant-e2e-shared-a-7a27b1/rate-test-model-7a27b1-kserve-5586589fc9-tmjf8" Jul 18 12:35:28.176528 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:35:28.176331 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0a77a318-9cc4-4d3b-bd0d-669e7c4a8d60-home\") pod \"rate-test-model-7a27b1-kserve-5586589fc9-tmjf8\" (UID: \"0a77a318-9cc4-4d3b-bd0d-669e7c4a8d60\") " pod="ai-tenant-e2e-shared-a-7a27b1/rate-test-model-7a27b1-kserve-5586589fc9-tmjf8" Jul 18 12:35:28.176734 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:35:28.176712 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0a77a318-9cc4-4d3b-bd0d-669e7c4a8d60-tmp-dir\") pod \"rate-test-model-7a27b1-kserve-5586589fc9-tmjf8\" (UID: \"0a77a318-9cc4-4d3b-bd0d-669e7c4a8d60\") " pod="ai-tenant-e2e-shared-a-7a27b1/rate-test-model-7a27b1-kserve-5586589fc9-tmjf8" Jul 18 12:35:28.176850 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:35:28.176766 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0a77a318-9cc4-4d3b-bd0d-669e7c4a8d60-home\") pod \"rate-test-model-7a27b1-kserve-5586589fc9-tmjf8\" (UID: \"0a77a318-9cc4-4d3b-bd0d-669e7c4a8d60\") " pod="ai-tenant-e2e-shared-a-7a27b1/rate-test-model-7a27b1-kserve-5586589fc9-tmjf8" Jul 18 12:35:28.176850 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:35:28.176808 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0a77a318-9cc4-4d3b-bd0d-669e7c4a8d60-model-cache\") pod \"rate-test-model-7a27b1-kserve-5586589fc9-tmjf8\" (UID: \"0a77a318-9cc4-4d3b-bd0d-669e7c4a8d60\") " pod="ai-tenant-e2e-shared-a-7a27b1/rate-test-model-7a27b1-kserve-5586589fc9-tmjf8" Jul 18 12:35:28.178751 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:35:28.178730 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0a77a318-9cc4-4d3b-bd0d-669e7c4a8d60-dshm\") pod \"rate-test-model-7a27b1-kserve-5586589fc9-tmjf8\" (UID: \"0a77a318-9cc4-4d3b-bd0d-669e7c4a8d60\") " pod="ai-tenant-e2e-shared-a-7a27b1/rate-test-model-7a27b1-kserve-5586589fc9-tmjf8" Jul 18 12:35:28.179037 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:35:28.179018 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0a77a318-9cc4-4d3b-bd0d-669e7c4a8d60-tls-certs\") pod \"rate-test-model-7a27b1-kserve-5586589fc9-tmjf8\" (UID: \"0a77a318-9cc4-4d3b-bd0d-669e7c4a8d60\") " pod="ai-tenant-e2e-shared-a-7a27b1/rate-test-model-7a27b1-kserve-5586589fc9-tmjf8" Jul 18 12:35:28.185216 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:35:28.185163 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lmgfx\" (UniqueName: \"kubernetes.io/projected/0a77a318-9cc4-4d3b-bd0d-669e7c4a8d60-kube-api-access-lmgfx\") pod \"rate-test-model-7a27b1-kserve-5586589fc9-tmjf8\" (UID: \"0a77a318-9cc4-4d3b-bd0d-669e7c4a8d60\") " pod="ai-tenant-e2e-shared-a-7a27b1/rate-test-model-7a27b1-kserve-5586589fc9-tmjf8" Jul 18 12:35:28.244171 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:35:28.244145 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-7a27b1/rate-test-model-7a27b1-kserve-5586589fc9-tmjf8" Jul 18 12:35:28.381127 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:35:28.381096 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-7a27b1/rate-test-model-7a27b1-kserve-5586589fc9-tmjf8"] Jul 18 12:35:28.383053 ip-10-0-136-240 kubenswrapper[2585]: W0718 12:35:28.383031 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0a77a318_9cc4_4d3b_bd0d_669e7c4a8d60.slice/crio-a29daae4c4ca6059966cd55edf49b95df36c09ef709083f40887b93ae715de88 WatchSource:0}: Error finding container a29daae4c4ca6059966cd55edf49b95df36c09ef709083f40887b93ae715de88: Status 404 returned error can't find the container with id a29daae4c4ca6059966cd55edf49b95df36c09ef709083f40887b93ae715de88 Jul 18 12:35:28.619541 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:35:28.619491 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-7a27b1/rate-test-model-7a27b1-kserve-5586589fc9-tmjf8" event={"ID":"0a77a318-9cc4-4d3b-bd0d-669e7c4a8d60","Type":"ContainerStarted","Data":"bcd53216d0d19739ad49c207e0b8b522e4d14a38d8036bb80fb3f08202469731"} Jul 18 12:35:28.619741 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:35:28.619563 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-7a27b1/rate-test-model-7a27b1-kserve-5586589fc9-tmjf8" event={"ID":"0a77a318-9cc4-4d3b-bd0d-669e7c4a8d60","Type":"ContainerStarted","Data":"a29daae4c4ca6059966cd55edf49b95df36c09ef709083f40887b93ae715de88"} Jul 18 12:35:28.635291 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:35:28.635230 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-7a27b1/rate-test-model-7a27b1-kserve-5586589fc9-tmjf8" podStartSLOduration=1.635211037 podStartE2EDuration="1.635211037s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-18 12:35:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-18 12:35:28.634480365 +0000 UTC m=+2795.714686733" watchObservedRunningTime="2026-07-18 12:35:28.635211037 +0000 UTC m=+2795.715417394" Jul 18 12:35:38.244844 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:35:38.244803 2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-7a27b1/rate-test-model-7a27b1-kserve-5586589fc9-tmjf8" Jul 18 12:35:38.244844 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:35:38.244852 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-7a27b1/rate-test-model-7a27b1-kserve-5586589fc9-tmjf8" Jul 18 12:35:38.257758 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:35:38.257724 2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-7a27b1/rate-test-model-7a27b1-kserve-5586589fc9-tmjf8" Jul 18 12:35:38.681976 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:35:38.681941 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-7a27b1/rate-test-model-7a27b1-kserve-5586589fc9-tmjf8" Jul 18 12:36:36.732866 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:36:36.732831 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-7a27b1/rate-test-model-7a27b1-kserve-5586589fc9-tmjf8"] Jul 18 12:36:51.753931 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:36:51.753834 2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-7a27b1/rate-test-model-7a27b1-kserve-5586589fc9-tmjf8" podUID="0a77a318-9cc4-4d3b-bd0d-669e7c4a8d60" containerName="main" containerID="cri-o://bcd53216d0d19739ad49c207e0b8b522e4d14a38d8036bb80fb3f08202469731" gracePeriod=45 Jul 18 12:36:52.011278 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:36:52.011214 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-7a27b1/rate-test-model-7a27b1-kserve-5586589fc9-tmjf8" Jul 18 12:36:52.040726 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:36:52.040693 2585 generic.go:356] "Generic (PLEG): container finished" podID="0a77a318-9cc4-4d3b-bd0d-669e7c4a8d60" containerID="bcd53216d0d19739ad49c207e0b8b522e4d14a38d8036bb80fb3f08202469731" exitCode=0 Jul 18 12:36:52.040870 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:36:52.040731 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-7a27b1/rate-test-model-7a27b1-kserve-5586589fc9-tmjf8" Jul 18 12:36:52.040870 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:36:52.040763 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-7a27b1/rate-test-model-7a27b1-kserve-5586589fc9-tmjf8" event={"ID":"0a77a318-9cc4-4d3b-bd0d-669e7c4a8d60","Type":"ContainerDied","Data":"bcd53216d0d19739ad49c207e0b8b522e4d14a38d8036bb80fb3f08202469731"} Jul 18 12:36:52.040870 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:36:52.040797 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-7a27b1/rate-test-model-7a27b1-kserve-5586589fc9-tmjf8" event={"ID":"0a77a318-9cc4-4d3b-bd0d-669e7c4a8d60","Type":"ContainerDied","Data":"a29daae4c4ca6059966cd55edf49b95df36c09ef709083f40887b93ae715de88"} Jul 18 12:36:52.040870 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:36:52.040815 2585 scope.go:117] "RemoveContainer" containerID="bcd53216d0d19739ad49c207e0b8b522e4d14a38d8036bb80fb3f08202469731" Jul 18 12:36:52.051209 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:36:52.051187 2585 scope.go:117] "RemoveContainer" containerID="bcd53216d0d19739ad49c207e0b8b522e4d14a38d8036bb80fb3f08202469731" Jul 18 12:36:52.051464 ip-10-0-136-240 kubenswrapper[2585]: E0718 12:36:52.051446 2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bcd53216d0d19739ad49c207e0b8b522e4d14a38d8036bb80fb3f08202469731\": container with ID starting with bcd53216d0d19739ad49c207e0b8b522e4d14a38d8036bb80fb3f08202469731 not found: ID does not exist" containerID="bcd53216d0d19739ad49c207e0b8b522e4d14a38d8036bb80fb3f08202469731" Jul 18 12:36:52.051542 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:36:52.051478 2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bcd53216d0d19739ad49c207e0b8b522e4d14a38d8036bb80fb3f08202469731"} err="failed to get container status \"bcd53216d0d19739ad49c207e0b8b522e4d14a38d8036bb80fb3f08202469731\": rpc error: code = NotFound desc = could not find container \"bcd53216d0d19739ad49c207e0b8b522e4d14a38d8036bb80fb3f08202469731\": container with ID starting with bcd53216d0d19739ad49c207e0b8b522e4d14a38d8036bb80fb3f08202469731 not found: ID does not exist" Jul 18 12:36:52.091923 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:36:52.091900 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0a77a318-9cc4-4d3b-bd0d-669e7c4a8d60-dshm\") pod \"0a77a318-9cc4-4d3b-bd0d-669e7c4a8d60\" (UID: \"0a77a318-9cc4-4d3b-bd0d-669e7c4a8d60\") " Jul 18 12:36:52.092019 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:36:52.091930 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0a77a318-9cc4-4d3b-bd0d-669e7c4a8d60-home\") pod \"0a77a318-9cc4-4d3b-bd0d-669e7c4a8d60\" (UID: \"0a77a318-9cc4-4d3b-bd0d-669e7c4a8d60\") " Jul 18 12:36:52.092019 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:36:52.091971 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0a77a318-9cc4-4d3b-bd0d-669e7c4a8d60-tmp-dir\") pod \"0a77a318-9cc4-4d3b-bd0d-669e7c4a8d60\" (UID: \"0a77a318-9cc4-4d3b-bd0d-669e7c4a8d60\") " Jul 18 12:36:52.092111 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:36:52.092020 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0a77a318-9cc4-4d3b-bd0d-669e7c4a8d60-model-cache\") pod \"0a77a318-9cc4-4d3b-bd0d-669e7c4a8d60\" (UID: \"0a77a318-9cc4-4d3b-bd0d-669e7c4a8d60\") " Jul 18 12:36:52.092111 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:36:52.092039 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lmgfx\" (UniqueName: \"kubernetes.io/projected/0a77a318-9cc4-4d3b-bd0d-669e7c4a8d60-kube-api-access-lmgfx\") pod \"0a77a318-9cc4-4d3b-bd0d-669e7c4a8d60\" (UID: \"0a77a318-9cc4-4d3b-bd0d-669e7c4a8d60\") " Jul 18 12:36:52.092111 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:36:52.092078 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0a77a318-9cc4-4d3b-bd0d-669e7c4a8d60-tls-certs\") pod \"0a77a318-9cc4-4d3b-bd0d-669e7c4a8d60\" (UID: \"0a77a318-9cc4-4d3b-bd0d-669e7c4a8d60\") " Jul 18 12:36:52.092270 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:36:52.092171 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a77a318-9cc4-4d3b-bd0d-669e7c4a8d60-home" (OuterVolumeSpecName: "home") pod "0a77a318-9cc4-4d3b-bd0d-669e7c4a8d60" (UID: "0a77a318-9cc4-4d3b-bd0d-669e7c4a8d60"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 18 12:36:52.092327 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:36:52.092274 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a77a318-9cc4-4d3b-bd0d-669e7c4a8d60-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "0a77a318-9cc4-4d3b-bd0d-669e7c4a8d60" (UID: "0a77a318-9cc4-4d3b-bd0d-669e7c4a8d60"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 18 12:36:52.092327 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:36:52.092285 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a77a318-9cc4-4d3b-bd0d-669e7c4a8d60-model-cache" (OuterVolumeSpecName: "model-cache") pod "0a77a318-9cc4-4d3b-bd0d-669e7c4a8d60" (UID: "0a77a318-9cc4-4d3b-bd0d-669e7c4a8d60"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 18 12:36:52.092435 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:36:52.092417 2585 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0a77a318-9cc4-4d3b-bd0d-669e7c4a8d60-tmp-dir\") on node \"ip-10-0-136-240.ec2.internal\" DevicePath \"\"" Jul 18 12:36:52.092486 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:36:52.092435 2585 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0a77a318-9cc4-4d3b-bd0d-669e7c4a8d60-model-cache\") on node \"ip-10-0-136-240.ec2.internal\" DevicePath \"\"" Jul 18 12:36:52.092486 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:36:52.092445 2585 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0a77a318-9cc4-4d3b-bd0d-669e7c4a8d60-home\") on node \"ip-10-0-136-240.ec2.internal\" DevicePath \"\"" Jul 18 12:36:52.094097 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:36:52.094068 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a77a318-9cc4-4d3b-bd0d-669e7c4a8d60-dshm" (OuterVolumeSpecName: "dshm") pod "0a77a318-9cc4-4d3b-bd0d-669e7c4a8d60" (UID: "0a77a318-9cc4-4d3b-bd0d-669e7c4a8d60"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 18 12:36:52.094197 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:36:52.094149 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0a77a318-9cc4-4d3b-bd0d-669e7c4a8d60-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "0a77a318-9cc4-4d3b-bd0d-669e7c4a8d60" (UID: "0a77a318-9cc4-4d3b-bd0d-669e7c4a8d60"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 18 12:36:52.094276 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:36:52.094254 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0a77a318-9cc4-4d3b-bd0d-669e7c4a8d60-kube-api-access-lmgfx" (OuterVolumeSpecName: "kube-api-access-lmgfx") pod "0a77a318-9cc4-4d3b-bd0d-669e7c4a8d60" (UID: "0a77a318-9cc4-4d3b-bd0d-669e7c4a8d60"). InnerVolumeSpecName "kube-api-access-lmgfx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 18 12:36:52.193160 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:36:52.193136 2585 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0a77a318-9cc4-4d3b-bd0d-669e7c4a8d60-dshm\") on node \"ip-10-0-136-240.ec2.internal\" DevicePath \"\"" Jul 18 12:36:52.193160 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:36:52.193158 2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lmgfx\" (UniqueName: \"kubernetes.io/projected/0a77a318-9cc4-4d3b-bd0d-669e7c4a8d60-kube-api-access-lmgfx\") on node \"ip-10-0-136-240.ec2.internal\" DevicePath \"\"" Jul 18 12:36:52.193315 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:36:52.193169 2585 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0a77a318-9cc4-4d3b-bd0d-669e7c4a8d60-tls-certs\") on node \"ip-10-0-136-240.ec2.internal\" DevicePath \"\"" Jul 18 12:36:52.362580 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:36:52.362534 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-7a27b1/rate-test-model-7a27b1-kserve-5586589fc9-tmjf8"] Jul 18 12:36:52.364593 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:36:52.364575 2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-7a27b1/rate-test-model-7a27b1-kserve-5586589fc9-tmjf8"] Jul 18 12:36:53.470395 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:36:53.470355 2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0a77a318-9cc4-4d3b-bd0d-669e7c4a8d60" path="/var/lib/kubelet/pods/0a77a318-9cc4-4d3b-bd0d-669e7c4a8d60/volumes" Jul 18 12:37:18.580862 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:37:18.580823 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-7a27b1-openshift-default-78f77667f-8l69z"] Jul 18 12:37:18.581460 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:37:18.581088 2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-7a27b1-openshift-default-78f77667f-8l69z" podUID="f3303f05-8333-424c-8824-8d4cda7ecc62" containerName="istio-proxy" containerID="cri-o://01fedf7826a472ec508c04079af2b639ac022cc57c94a1c29e742fb12f34c0f0" gracePeriod=30 Jul 18 12:37:21.855009 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:37:21.854968 2585 patch_prober.go:28] interesting pod/e2e-shared-a-7a27b1-openshift-default-78f77667f-8l69z container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 18 12:37:21.855371 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:37:21.855023 2585 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-7a27b1-openshift-default-78f77667f-8l69z" podUID="f3303f05-8333-424c-8824-8d4cda7ecc62" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 18 12:37:23.836184 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:37:23.836160 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-7a27b1-openshift-default-78f77667f-8l69z" Jul 18 12:37:24.007705 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:37:24.007678 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f3303f05-8333-424c-8824-8d4cda7ecc62-istiod-ca-cert\") pod \"f3303f05-8333-424c-8824-8d4cda7ecc62\" (UID: \"f3303f05-8333-424c-8824-8d4cda7ecc62\") " Jul 18 12:37:24.007869 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:37:24.007742 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f3303f05-8333-424c-8824-8d4cda7ecc62-istio-data\") pod \"f3303f05-8333-424c-8824-8d4cda7ecc62\" (UID: \"f3303f05-8333-424c-8824-8d4cda7ecc62\") " Jul 18 12:37:24.007869 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:37:24.007757 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f3303f05-8333-424c-8824-8d4cda7ecc62-workload-certs\") pod \"f3303f05-8333-424c-8824-8d4cda7ecc62\" (UID: \"f3303f05-8333-424c-8824-8d4cda7ecc62\") " Jul 18 12:37:24.007869 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:37:24.007775 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3303f05-8333-424c-8824-8d4cda7ecc62-workload-socket\") pod \"f3303f05-8333-424c-8824-8d4cda7ecc62\" (UID: \"f3303f05-8333-424c-8824-8d4cda7ecc62\") " Jul 18 12:37:24.008023 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:37:24.007878 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f3303f05-8333-424c-8824-8d4cda7ecc62-istio-podinfo\") pod \"f3303f05-8333-424c-8824-8d4cda7ecc62\" (UID: \"f3303f05-8333-424c-8824-8d4cda7ecc62\") " Jul 18 12:37:24.008023 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:37:24.007925 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f3303f05-8333-424c-8824-8d4cda7ecc62-istio-token\") pod \"f3303f05-8333-424c-8824-8d4cda7ecc62\" (UID: \"f3303f05-8333-424c-8824-8d4cda7ecc62\") " Jul 18 12:37:24.008023 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:37:24.007974 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3303f05-8333-424c-8824-8d4cda7ecc62-credential-socket\") pod \"f3303f05-8333-424c-8824-8d4cda7ecc62\" (UID: \"f3303f05-8333-424c-8824-8d4cda7ecc62\") " Jul 18 12:37:24.008023 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:37:24.008019 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bhrvg\" (UniqueName: \"kubernetes.io/projected/f3303f05-8333-424c-8824-8d4cda7ecc62-kube-api-access-bhrvg\") pod \"f3303f05-8333-424c-8824-8d4cda7ecc62\" (UID: \"f3303f05-8333-424c-8824-8d4cda7ecc62\") " Jul 18 12:37:24.008223 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:37:24.008023 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f3303f05-8333-424c-8824-8d4cda7ecc62-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "f3303f05-8333-424c-8824-8d4cda7ecc62" (UID: "f3303f05-8333-424c-8824-8d4cda7ecc62"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 18 12:37:24.008223 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:37:24.008044 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f3303f05-8333-424c-8824-8d4cda7ecc62-istio-envoy\") pod \"f3303f05-8333-424c-8824-8d4cda7ecc62\" (UID: \"f3303f05-8333-424c-8824-8d4cda7ecc62\") " Jul 18 12:37:24.008223 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:37:24.008094 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f3303f05-8333-424c-8824-8d4cda7ecc62-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "f3303f05-8333-424c-8824-8d4cda7ecc62" (UID: "f3303f05-8333-424c-8824-8d4cda7ecc62"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 18 12:37:24.008223 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:37:24.008143 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f3303f05-8333-424c-8824-8d4cda7ecc62-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "f3303f05-8333-424c-8824-8d4cda7ecc62" (UID: "f3303f05-8333-424c-8824-8d4cda7ecc62"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 18 12:37:24.008411 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:37:24.008304 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f3303f05-8333-424c-8824-8d4cda7ecc62-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "f3303f05-8333-424c-8824-8d4cda7ecc62" (UID: "f3303f05-8333-424c-8824-8d4cda7ecc62"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 18 12:37:24.008470 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:37:24.008448 2585 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f3303f05-8333-424c-8824-8d4cda7ecc62-istiod-ca-cert\") on node \"ip-10-0-136-240.ec2.internal\" DevicePath \"\"" Jul 18 12:37:24.008470 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:37:24.008463 2585 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f3303f05-8333-424c-8824-8d4cda7ecc62-workload-certs\") on node \"ip-10-0-136-240.ec2.internal\" DevicePath \"\"" Jul 18 12:37:24.008660 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:37:24.008476 2585 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3303f05-8333-424c-8824-8d4cda7ecc62-workload-socket\") on node \"ip-10-0-136-240.ec2.internal\" DevicePath \"\"" Jul 18 12:37:24.008660 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:37:24.008489 2585 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3303f05-8333-424c-8824-8d4cda7ecc62-credential-socket\") on node \"ip-10-0-136-240.ec2.internal\" DevicePath \"\"" Jul 18 12:37:24.008660 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:37:24.008628 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f3303f05-8333-424c-8824-8d4cda7ecc62-istio-data" (OuterVolumeSpecName: "istio-data") pod "f3303f05-8333-424c-8824-8d4cda7ecc62" (UID: "f3303f05-8333-424c-8824-8d4cda7ecc62"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 18 12:37:24.010409 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:37:24.010375 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f3303f05-8333-424c-8824-8d4cda7ecc62-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "f3303f05-8333-424c-8824-8d4cda7ecc62" (UID: "f3303f05-8333-424c-8824-8d4cda7ecc62"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 18 12:37:24.010529 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:37:24.010445 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f3303f05-8333-424c-8824-8d4cda7ecc62-kube-api-access-bhrvg" (OuterVolumeSpecName: "kube-api-access-bhrvg") pod "f3303f05-8333-424c-8824-8d4cda7ecc62" (UID: "f3303f05-8333-424c-8824-8d4cda7ecc62"). InnerVolumeSpecName "kube-api-access-bhrvg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 18 12:37:24.010818 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:37:24.010802 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f3303f05-8333-424c-8824-8d4cda7ecc62-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "f3303f05-8333-424c-8824-8d4cda7ecc62" (UID: "f3303f05-8333-424c-8824-8d4cda7ecc62"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 18 12:37:24.010866 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:37:24.010829 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f3303f05-8333-424c-8824-8d4cda7ecc62-istio-token" (OuterVolumeSpecName: "istio-token") pod "f3303f05-8333-424c-8824-8d4cda7ecc62" (UID: "f3303f05-8333-424c-8824-8d4cda7ecc62"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 18 12:37:24.109705 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:37:24.109668 2585 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f3303f05-8333-424c-8824-8d4cda7ecc62-istio-data\") on node \"ip-10-0-136-240.ec2.internal\" DevicePath \"\"" Jul 18 12:37:24.109705 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:37:24.109698 2585 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f3303f05-8333-424c-8824-8d4cda7ecc62-istio-podinfo\") on node \"ip-10-0-136-240.ec2.internal\" DevicePath \"\"" Jul 18 12:37:24.109705 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:37:24.109709 2585 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f3303f05-8333-424c-8824-8d4cda7ecc62-istio-token\") on node \"ip-10-0-136-240.ec2.internal\" DevicePath \"\"" Jul 18 12:37:24.109928 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:37:24.109717 2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bhrvg\" (UniqueName: \"kubernetes.io/projected/f3303f05-8333-424c-8824-8d4cda7ecc62-kube-api-access-bhrvg\") on node \"ip-10-0-136-240.ec2.internal\" DevicePath \"\"" Jul 18 12:37:24.109928 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:37:24.109727 2585 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f3303f05-8333-424c-8824-8d4cda7ecc62-istio-envoy\") on node \"ip-10-0-136-240.ec2.internal\" DevicePath \"\"" Jul 18 12:37:24.203589 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:37:24.203525 2585 generic.go:356] "Generic (PLEG): container finished" podID="f3303f05-8333-424c-8824-8d4cda7ecc62" containerID="01fedf7826a472ec508c04079af2b639ac022cc57c94a1c29e742fb12f34c0f0" exitCode=0 Jul 18 12:37:24.203589 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:37:24.203590 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-7a27b1-openshift-default-78f77667f-8l69z" event={"ID":"f3303f05-8333-424c-8824-8d4cda7ecc62","Type":"ContainerDied","Data":"01fedf7826a472ec508c04079af2b639ac022cc57c94a1c29e742fb12f34c0f0"} Jul 18 12:37:24.203837 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:37:24.203628 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-7a27b1-openshift-default-78f77667f-8l69z" Jul 18 12:37:24.203837 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:37:24.203641 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-7a27b1-openshift-default-78f77667f-8l69z" event={"ID":"f3303f05-8333-424c-8824-8d4cda7ecc62","Type":"ContainerDied","Data":"79c22c5b5ef4eb7e3d93fe68cd6a640cce7978a72eea0395c525036db5610618"} Jul 18 12:37:24.203837 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:37:24.203667 2585 scope.go:117] "RemoveContainer" containerID="01fedf7826a472ec508c04079af2b639ac022cc57c94a1c29e742fb12f34c0f0" Jul 18 12:37:24.215072 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:37:24.215054 2585 scope.go:117] "RemoveContainer" containerID="f72c4fcf2b5765726bab46dbfb64d11786f50a360aede7afd870db3068cea044" Jul 18 12:37:24.223326 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:37:24.223307 2585 scope.go:117] "RemoveContainer" containerID="01fedf7826a472ec508c04079af2b639ac022cc57c94a1c29e742fb12f34c0f0" Jul 18 12:37:24.223581 ip-10-0-136-240 kubenswrapper[2585]: E0718 12:37:24.223559 2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"01fedf7826a472ec508c04079af2b639ac022cc57c94a1c29e742fb12f34c0f0\": container with ID starting with 01fedf7826a472ec508c04079af2b639ac022cc57c94a1c29e742fb12f34c0f0 not found: ID does not exist" containerID="01fedf7826a472ec508c04079af2b639ac022cc57c94a1c29e742fb12f34c0f0" Jul 18 12:37:24.223686 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:37:24.223585 2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"01fedf7826a472ec508c04079af2b639ac022cc57c94a1c29e742fb12f34c0f0"} err="failed to get container status \"01fedf7826a472ec508c04079af2b639ac022cc57c94a1c29e742fb12f34c0f0\": rpc error: code = NotFound desc = could not find container \"01fedf7826a472ec508c04079af2b639ac022cc57c94a1c29e742fb12f34c0f0\": container with ID starting with 01fedf7826a472ec508c04079af2b639ac022cc57c94a1c29e742fb12f34c0f0 not found: ID does not exist" Jul 18 12:37:24.223686 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:37:24.223600 2585 scope.go:117] "RemoveContainer" containerID="f72c4fcf2b5765726bab46dbfb64d11786f50a360aede7afd870db3068cea044" Jul 18 12:37:24.223821 ip-10-0-136-240 kubenswrapper[2585]: E0718 12:37:24.223804 2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f72c4fcf2b5765726bab46dbfb64d11786f50a360aede7afd870db3068cea044\": container with ID starting with f72c4fcf2b5765726bab46dbfb64d11786f50a360aede7afd870db3068cea044 not found: ID does not exist" containerID="f72c4fcf2b5765726bab46dbfb64d11786f50a360aede7afd870db3068cea044" Jul 18 12:37:24.223865 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:37:24.223826 2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f72c4fcf2b5765726bab46dbfb64d11786f50a360aede7afd870db3068cea044"} err="failed to get container status \"f72c4fcf2b5765726bab46dbfb64d11786f50a360aede7afd870db3068cea044\": rpc error: code = NotFound desc = could not find container \"f72c4fcf2b5765726bab46dbfb64d11786f50a360aede7afd870db3068cea044\": container with ID starting with f72c4fcf2b5765726bab46dbfb64d11786f50a360aede7afd870db3068cea044 not found: ID does not exist" Jul 18 12:37:24.229622 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:37:24.229599 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-7a27b1-openshift-default-78f77667f-8l69z"] Jul 18 12:37:24.233476 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:37:24.233456 2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-7a27b1-openshift-default-78f77667f-8l69z"] Jul 18 12:37:25.469029 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:37:25.468997 2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f3303f05-8333-424c-8824-8d4cda7ecc62" path="/var/lib/kubelet/pods/f3303f05-8333-424c-8824-8d4cda7ecc62/volumes" Jul 18 12:37:40.710662 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:37:40.710626 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-8b9f97-openshift-default-6d95c44899-969fp"] Jul 18 12:37:40.711114 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:37:40.710895 2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-8b9f97-openshift-default-6d95c44899-969fp" podUID="9be6d88e-bbb1-422c-971c-5af9cec4f1c6" containerName="istio-proxy" containerID="cri-o://cae6f0d75038ac933649e13bbfe3d64e3fd95d1cdb97156f484532b1b6970f4e" gracePeriod=30 Jul 18 12:37:43.299649 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:37:43.299611 2585 patch_prober.go:28] interesting pod/e2e-shared-b-8b9f97-openshift-default-6d95c44899-969fp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 18 12:37:43.300024 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:37:43.299680 2585 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-8b9f97-openshift-default-6d95c44899-969fp" podUID="9be6d88e-bbb1-422c-971c-5af9cec4f1c6" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 18 12:37:45.966599 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:37:45.966576 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-8b9f97-openshift-default-6d95c44899-969fp" Jul 18 12:37:46.103448 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:37:46.103352 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9be6d88e-bbb1-422c-971c-5af9cec4f1c6-workload-certs\") pod \"9be6d88e-bbb1-422c-971c-5af9cec4f1c6\" (UID: \"9be6d88e-bbb1-422c-971c-5af9cec4f1c6\") " Jul 18 12:37:46.103448 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:37:46.103398 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9be6d88e-bbb1-422c-971c-5af9cec4f1c6-istio-data\") pod \"9be6d88e-bbb1-422c-971c-5af9cec4f1c6\" (UID: \"9be6d88e-bbb1-422c-971c-5af9cec4f1c6\") " Jul 18 12:37:46.103448 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:37:46.103437 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9be6d88e-bbb1-422c-971c-5af9cec4f1c6-workload-socket\") pod \"9be6d88e-bbb1-422c-971c-5af9cec4f1c6\" (UID: \"9be6d88e-bbb1-422c-971c-5af9cec4f1c6\") " Jul 18 12:37:46.103787 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:37:46.103473 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9be6d88e-bbb1-422c-971c-5af9cec4f1c6-istio-token\") pod \"9be6d88e-bbb1-422c-971c-5af9cec4f1c6\" (UID: \"9be6d88e-bbb1-422c-971c-5af9cec4f1c6\") " Jul 18 12:37:46.103787 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:37:46.103498 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9be6d88e-bbb1-422c-971c-5af9cec4f1c6-credential-socket\") pod \"9be6d88e-bbb1-422c-971c-5af9cec4f1c6\" (UID: \"9be6d88e-bbb1-422c-971c-5af9cec4f1c6\") " Jul 18 12:37:46.103787 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:37:46.103579 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9be6d88e-bbb1-422c-971c-5af9cec4f1c6-istio-envoy\") pod \"9be6d88e-bbb1-422c-971c-5af9cec4f1c6\" (UID: \"9be6d88e-bbb1-422c-971c-5af9cec4f1c6\") " Jul 18 12:37:46.103787 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:37:46.103608 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-24pm7\" (UniqueName: \"kubernetes.io/projected/9be6d88e-bbb1-422c-971c-5af9cec4f1c6-kube-api-access-24pm7\") pod \"9be6d88e-bbb1-422c-971c-5af9cec4f1c6\" (UID: \"9be6d88e-bbb1-422c-971c-5af9cec4f1c6\") " Jul 18 12:37:46.103787 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:37:46.103657 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9be6d88e-bbb1-422c-971c-5af9cec4f1c6-istio-podinfo\") pod \"9be6d88e-bbb1-422c-971c-5af9cec4f1c6\" (UID: \"9be6d88e-bbb1-422c-971c-5af9cec4f1c6\") " Jul 18 12:37:46.103787 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:37:46.103674 2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9be6d88e-bbb1-422c-971c-5af9cec4f1c6-istiod-ca-cert\") pod \"9be6d88e-bbb1-422c-971c-5af9cec4f1c6\" (UID: \"9be6d88e-bbb1-422c-971c-5af9cec4f1c6\") " Jul 18 12:37:46.103787 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:37:46.103689 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9be6d88e-bbb1-422c-971c-5af9cec4f1c6-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "9be6d88e-bbb1-422c-971c-5af9cec4f1c6" (UID: "9be6d88e-bbb1-422c-971c-5af9cec4f1c6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 18 12:37:46.104157 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:37:46.103788 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9be6d88e-bbb1-422c-971c-5af9cec4f1c6-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "9be6d88e-bbb1-422c-971c-5af9cec4f1c6" (UID: "9be6d88e-bbb1-422c-971c-5af9cec4f1c6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 18 12:37:46.104157 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:37:46.103970 2585 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9be6d88e-bbb1-422c-971c-5af9cec4f1c6-workload-certs\") on node \"ip-10-0-136-240.ec2.internal\" DevicePath \"\"" Jul 18 12:37:46.104157 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:37:46.103988 2585 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9be6d88e-bbb1-422c-971c-5af9cec4f1c6-workload-socket\") on node \"ip-10-0-136-240.ec2.internal\" DevicePath \"\"" Jul 18 12:37:46.104157 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:37:46.104117 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9be6d88e-bbb1-422c-971c-5af9cec4f1c6-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "9be6d88e-bbb1-422c-971c-5af9cec4f1c6" (UID: "9be6d88e-bbb1-422c-971c-5af9cec4f1c6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 18 12:37:46.104157 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:37:46.104145 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9be6d88e-bbb1-422c-971c-5af9cec4f1c6-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "9be6d88e-bbb1-422c-971c-5af9cec4f1c6" (UID: "9be6d88e-bbb1-422c-971c-5af9cec4f1c6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 18 12:37:46.104543 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:37:46.104244 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9be6d88e-bbb1-422c-971c-5af9cec4f1c6-istio-data" (OuterVolumeSpecName: "istio-data") pod "9be6d88e-bbb1-422c-971c-5af9cec4f1c6" (UID: "9be6d88e-bbb1-422c-971c-5af9cec4f1c6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 18 12:37:46.106036 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:37:46.106005 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9be6d88e-bbb1-422c-971c-5af9cec4f1c6-kube-api-access-24pm7" (OuterVolumeSpecName: "kube-api-access-24pm7") pod "9be6d88e-bbb1-422c-971c-5af9cec4f1c6" (UID: "9be6d88e-bbb1-422c-971c-5af9cec4f1c6"). InnerVolumeSpecName "kube-api-access-24pm7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 18 12:37:46.106150 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:37:46.106037 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9be6d88e-bbb1-422c-971c-5af9cec4f1c6-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "9be6d88e-bbb1-422c-971c-5af9cec4f1c6" (UID: "9be6d88e-bbb1-422c-971c-5af9cec4f1c6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 18 12:37:46.106150 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:37:46.106039 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9be6d88e-bbb1-422c-971c-5af9cec4f1c6-istio-token" (OuterVolumeSpecName: "istio-token") pod "9be6d88e-bbb1-422c-971c-5af9cec4f1c6" (UID: "9be6d88e-bbb1-422c-971c-5af9cec4f1c6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 18 12:37:46.106150 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:37:46.106104 2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9be6d88e-bbb1-422c-971c-5af9cec4f1c6-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "9be6d88e-bbb1-422c-971c-5af9cec4f1c6" (UID: "9be6d88e-bbb1-422c-971c-5af9cec4f1c6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 18 12:37:46.204747 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:37:46.204720 2585 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9be6d88e-bbb1-422c-971c-5af9cec4f1c6-istio-data\") on node \"ip-10-0-136-240.ec2.internal\" DevicePath \"\"" Jul 18 12:37:46.204747 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:37:46.204744 2585 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9be6d88e-bbb1-422c-971c-5af9cec4f1c6-istio-token\") on node \"ip-10-0-136-240.ec2.internal\" DevicePath \"\"" Jul 18 12:37:46.204900 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:37:46.204762 2585 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9be6d88e-bbb1-422c-971c-5af9cec4f1c6-credential-socket\") on node \"ip-10-0-136-240.ec2.internal\" DevicePath \"\"" Jul 18 12:37:46.204900 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:37:46.204772 2585 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9be6d88e-bbb1-422c-971c-5af9cec4f1c6-istio-envoy\") on node \"ip-10-0-136-240.ec2.internal\" DevicePath \"\"" Jul 18 12:37:46.204900 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:37:46.204781 2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-24pm7\" (UniqueName: \"kubernetes.io/projected/9be6d88e-bbb1-422c-971c-5af9cec4f1c6-kube-api-access-24pm7\") on node \"ip-10-0-136-240.ec2.internal\" DevicePath \"\"" Jul 18 12:37:46.204900 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:37:46.204790 2585 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9be6d88e-bbb1-422c-971c-5af9cec4f1c6-istio-podinfo\") on node \"ip-10-0-136-240.ec2.internal\" DevicePath \"\"" Jul 18 12:37:46.204900 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:37:46.204799 2585 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9be6d88e-bbb1-422c-971c-5af9cec4f1c6-istiod-ca-cert\") on node \"ip-10-0-136-240.ec2.internal\" DevicePath \"\"" Jul 18 12:37:46.322263 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:37:46.322224 2585 generic.go:356] "Generic (PLEG): container finished" podID="9be6d88e-bbb1-422c-971c-5af9cec4f1c6" containerID="cae6f0d75038ac933649e13bbfe3d64e3fd95d1cdb97156f484532b1b6970f4e" exitCode=0 Jul 18 12:37:46.322435 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:37:46.322280 2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-8b9f97-openshift-default-6d95c44899-969fp" Jul 18 12:37:46.322435 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:37:46.322317 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-8b9f97-openshift-default-6d95c44899-969fp" event={"ID":"9be6d88e-bbb1-422c-971c-5af9cec4f1c6","Type":"ContainerDied","Data":"cae6f0d75038ac933649e13bbfe3d64e3fd95d1cdb97156f484532b1b6970f4e"} Jul 18 12:37:46.322435 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:37:46.322386 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-8b9f97-openshift-default-6d95c44899-969fp" event={"ID":"9be6d88e-bbb1-422c-971c-5af9cec4f1c6","Type":"ContainerDied","Data":"74d68efa698c7b3728c1ace4118db13a6ffc2e1fccc31677573412e57df5ddaa"} Jul 18 12:37:46.322435 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:37:46.322415 2585 scope.go:117] "RemoveContainer" containerID="cae6f0d75038ac933649e13bbfe3d64e3fd95d1cdb97156f484532b1b6970f4e" Jul 18 12:37:46.338143 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:37:46.338123 2585 scope.go:117] "RemoveContainer" containerID="72dc1303f28c14c5868863ff85c8c6bf1cee89cbdb8a64be169b8553940a1d24" Jul 18 12:37:46.349786 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:37:46.349768 2585 scope.go:117] "RemoveContainer" containerID="cae6f0d75038ac933649e13bbfe3d64e3fd95d1cdb97156f484532b1b6970f4e" Jul 18 12:37:46.350006 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:37:46.349972 2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-8b9f97-openshift-default-6d95c44899-969fp"] Jul 18 12:37:46.350101 ip-10-0-136-240 kubenswrapper[2585]: E0718 12:37:46.350028 2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cae6f0d75038ac933649e13bbfe3d64e3fd95d1cdb97156f484532b1b6970f4e\": container with ID starting with cae6f0d75038ac933649e13bbfe3d64e3fd95d1cdb97156f484532b1b6970f4e not found: ID does not exist" containerID="cae6f0d75038ac933649e13bbfe3d64e3fd95d1cdb97156f484532b1b6970f4e" Jul 18 12:37:46.350101 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:37:46.350048 2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cae6f0d75038ac933649e13bbfe3d64e3fd95d1cdb97156f484532b1b6970f4e"} err="failed to get container status \"cae6f0d75038ac933649e13bbfe3d64e3fd95d1cdb97156f484532b1b6970f4e\": rpc error: code = NotFound desc = could not find container \"cae6f0d75038ac933649e13bbfe3d64e3fd95d1cdb97156f484532b1b6970f4e\": container with ID starting with cae6f0d75038ac933649e13bbfe3d64e3fd95d1cdb97156f484532b1b6970f4e not found: ID does not exist" Jul 18 12:37:46.350101 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:37:46.350065 2585 scope.go:117] "RemoveContainer" containerID="72dc1303f28c14c5868863ff85c8c6bf1cee89cbdb8a64be169b8553940a1d24" Jul 18 12:37:46.350350 ip-10-0-136-240 kubenswrapper[2585]: E0718 12:37:46.350333 2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"72dc1303f28c14c5868863ff85c8c6bf1cee89cbdb8a64be169b8553940a1d24\": container with ID starting with 72dc1303f28c14c5868863ff85c8c6bf1cee89cbdb8a64be169b8553940a1d24 not found: ID does not exist" containerID="72dc1303f28c14c5868863ff85c8c6bf1cee89cbdb8a64be169b8553940a1d24" Jul 18 12:37:46.350396 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:37:46.350356 2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"72dc1303f28c14c5868863ff85c8c6bf1cee89cbdb8a64be169b8553940a1d24"} err="failed to get container status \"72dc1303f28c14c5868863ff85c8c6bf1cee89cbdb8a64be169b8553940a1d24\": rpc error: code = NotFound desc = could not find container \"72dc1303f28c14c5868863ff85c8c6bf1cee89cbdb8a64be169b8553940a1d24\": container with ID starting with 72dc1303f28c14c5868863ff85c8c6bf1cee89cbdb8a64be169b8553940a1d24 not found: ID does not exist" Jul 18 12:37:46.355673 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:37:46.355626 2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-8b9f97-openshift-default-6d95c44899-969fp"] Jul 18 12:37:47.125717 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:37:47.125688 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-fbb59dc99-9rkcw_d46e8b83-a5c7-44b0-aa32-7c68ac538c32/copy-manifests/0.log" Jul 18 12:37:47.131749 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:37:47.131729 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-fbb59dc99-9rkcw_d46e8b83-a5c7-44b0-aa32-7c68ac538c32/manager/0.log" Jul 18 12:37:47.469979 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:37:47.469949 2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9be6d88e-bbb1-422c-971c-5af9cec4f1c6" path="/var/lib/kubelet/pods/9be6d88e-bbb1-422c-971c-5af9cec4f1c6/volumes" Jul 18 12:37:49.528763 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:37:49.528728 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-75cc8d5cb7-vbvt6_3a1aa8e5-bd4a-4cf2-b4d4-25c0dae5c37e/manager/0.log" Jul 18 12:37:49.637608 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:37:49.637583 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-864dd78c4c-vq5ct_1f66be24-7dfd-4669-9f68-464837b42d1b/manager/0.log" Jul 18 12:37:49.848695 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:37:49.848628 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-catalog-75bjg_72eecc76-acef-47ea-b4f2-d0fa11afb272/registry-server/0.log" Jul 18 12:37:50.180478 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:37:50.180406 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-859d7b78bd-5kx76_2f6f85c7-2112-4511-b178-6d602a8ce6fe/manager/0.log" Jul 18 12:37:50.534648 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:37:50.534618 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-6576859cf49clx6_d999136c-b948-4ed1-a61a-ad0cdf3e339d/istio-proxy/0.log" Jul 18 12:37:50.981408 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:37:50.981377 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-2gcb2_faf0508a-7db7-4cbe-915a-499c404d0bd0/istio-proxy/1.log" Jul 18 12:37:51.318772 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:37:51.318697 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-696b95fb6d-hx9nd_9477954a-e6a6-4bba-ad8f-3a8788b2b2ec/router/0.log" Jul 18 12:37:51.753829 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:37:51.753805 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-59975d69f9-tjmmg_67ebd32b-5627-48c6-ae45-2db77bd3f9ed/main/0.log" Jul 18 12:37:52.070374 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:37:52.070275 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-74b68447df-6lbk7_3daa0fca-c6f0-4020-9eb2-cb5db4393ac0/main/0.log" Jul 18 12:37:52.179657 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:37:52.179631 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-57fc66f697-mjjj4_136cde5b-95d7-480c-b712-3a6233f43e5d/main/0.log" Jul 18 12:37:57.865338 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:37:57.865290 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-6lsxg/must-gather-bf2rx"] Jul 18 12:37:57.865741 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:37:57.865703 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9be6d88e-bbb1-422c-971c-5af9cec4f1c6" containerName="istio-proxy" Jul 18 12:37:57.865741 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:37:57.865716 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="9be6d88e-bbb1-422c-971c-5af9cec4f1c6" containerName="istio-proxy" Jul 18 12:37:57.865741 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:37:57.865729 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f3303f05-8333-424c-8824-8d4cda7ecc62" containerName="istio-proxy" Jul 18 12:37:57.865741 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:37:57.865735 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="f3303f05-8333-424c-8824-8d4cda7ecc62" containerName="istio-proxy" Jul 18 12:37:57.865741 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:37:57.865744 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9be6d88e-bbb1-422c-971c-5af9cec4f1c6" containerName="istio-proxy" Jul 18 12:37:57.866054 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:37:57.865750 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="9be6d88e-bbb1-422c-971c-5af9cec4f1c6" containerName="istio-proxy" Jul 18 12:37:57.866054 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:37:57.865760 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0a77a318-9cc4-4d3b-bd0d-669e7c4a8d60" containerName="main" Jul 18 12:37:57.866054 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:37:57.865766 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="0a77a318-9cc4-4d3b-bd0d-669e7c4a8d60" containerName="main" Jul 18 12:37:57.866054 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:37:57.865828 2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="9be6d88e-bbb1-422c-971c-5af9cec4f1c6" containerName="istio-proxy" Jul 18 12:37:57.866054 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:37:57.865834 2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="f3303f05-8333-424c-8824-8d4cda7ecc62" containerName="istio-proxy" Jul 18 12:37:57.866054 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:37:57.865840 2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="0a77a318-9cc4-4d3b-bd0d-669e7c4a8d60" containerName="main" Jul 18 12:37:57.866054 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:37:57.865846 2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="9be6d88e-bbb1-422c-971c-5af9cec4f1c6" containerName="istio-proxy" Jul 18 12:37:57.866054 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:37:57.865853 2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="f3303f05-8333-424c-8824-8d4cda7ecc62" containerName="istio-proxy" Jul 18 12:37:57.866054 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:37:57.865914 2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f3303f05-8333-424c-8824-8d4cda7ecc62" containerName="istio-proxy" Jul 18 12:37:57.866054 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:37:57.865923 2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="f3303f05-8333-424c-8824-8d4cda7ecc62" containerName="istio-proxy" Jul 18 12:37:57.870727 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:37:57.870705 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-6lsxg/must-gather-bf2rx" Jul 18 12:37:57.873174 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:37:57.873143 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-6lsxg\"/\"openshift-service-ca.crt\"" Jul 18 12:37:57.873295 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:37:57.873195 2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-6lsxg\"/\"kube-root-ca.crt\"" Jul 18 12:37:57.873956 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:37:57.873937 2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-6lsxg\"/\"default-dockercfg-fk6km\"" Jul 18 12:37:57.877492 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:37:57.877466 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-6lsxg/must-gather-bf2rx"] Jul 18 12:37:58.007696 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:37:58.007660 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/6cd8aa0b-a832-4d70-aa2d-76edf4e79908-must-gather-output\") pod \"must-gather-bf2rx\" (UID: \"6cd8aa0b-a832-4d70-aa2d-76edf4e79908\") " pod="openshift-must-gather-6lsxg/must-gather-bf2rx" Jul 18 12:37:58.007848 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:37:58.007753 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kkvqg\" (UniqueName: \"kubernetes.io/projected/6cd8aa0b-a832-4d70-aa2d-76edf4e79908-kube-api-access-kkvqg\") pod \"must-gather-bf2rx\" (UID: \"6cd8aa0b-a832-4d70-aa2d-76edf4e79908\") " pod="openshift-must-gather-6lsxg/must-gather-bf2rx" Jul 18 12:37:58.108899 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:37:58.108870 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kkvqg\" (UniqueName: \"kubernetes.io/projected/6cd8aa0b-a832-4d70-aa2d-76edf4e79908-kube-api-access-kkvqg\") pod \"must-gather-bf2rx\" (UID: \"6cd8aa0b-a832-4d70-aa2d-76edf4e79908\") " pod="openshift-must-gather-6lsxg/must-gather-bf2rx" Jul 18 12:37:58.109081 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:37:58.108935 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/6cd8aa0b-a832-4d70-aa2d-76edf4e79908-must-gather-output\") pod \"must-gather-bf2rx\" (UID: \"6cd8aa0b-a832-4d70-aa2d-76edf4e79908\") " pod="openshift-must-gather-6lsxg/must-gather-bf2rx" Jul 18 12:37:58.109236 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:37:58.109216 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/6cd8aa0b-a832-4d70-aa2d-76edf4e79908-must-gather-output\") pod \"must-gather-bf2rx\" (UID: \"6cd8aa0b-a832-4d70-aa2d-76edf4e79908\") " pod="openshift-must-gather-6lsxg/must-gather-bf2rx" Jul 18 12:37:58.121521 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:37:58.121462 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kkvqg\" (UniqueName: \"kubernetes.io/projected/6cd8aa0b-a832-4d70-aa2d-76edf4e79908-kube-api-access-kkvqg\") pod \"must-gather-bf2rx\" (UID: \"6cd8aa0b-a832-4d70-aa2d-76edf4e79908\") " pod="openshift-must-gather-6lsxg/must-gather-bf2rx" Jul 18 12:37:58.181044 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:37:58.181019 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-6lsxg/must-gather-bf2rx" Jul 18 12:37:58.317849 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:37:58.317820 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-6lsxg/must-gather-bf2rx"] Jul 18 12:37:58.319975 ip-10-0-136-240 kubenswrapper[2585]: W0718 12:37:58.319941 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6cd8aa0b_a832_4d70_aa2d_76edf4e79908.slice/crio-c47f3c86e8f9cadeb891a4e199e2460a584326920fa320570c58297ca820cd35 WatchSource:0}: Error finding container c47f3c86e8f9cadeb891a4e199e2460a584326920fa320570c58297ca820cd35: Status 404 returned error can't find the container with id c47f3c86e8f9cadeb891a4e199e2460a584326920fa320570c58297ca820cd35 Jul 18 12:37:58.322100 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:37:58.322081 2585 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 18 12:37:58.385460 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:37:58.385393 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-6lsxg/must-gather-bf2rx" event={"ID":"6cd8aa0b-a832-4d70-aa2d-76edf4e79908","Type":"ContainerStarted","Data":"c47f3c86e8f9cadeb891a4e199e2460a584326920fa320570c58297ca820cd35"} Jul 18 12:37:59.394892 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:37:59.394793 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-6lsxg/must-gather-bf2rx" event={"ID":"6cd8aa0b-a832-4d70-aa2d-76edf4e79908","Type":"ContainerStarted","Data":"c1f5ec0d89da18fc3aef981e33ce8e73217fc4317e5c201c9994bb63469260cf"} Jul 18 12:37:59.394892 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:37:59.394841 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-6lsxg/must-gather-bf2rx" event={"ID":"6cd8aa0b-a832-4d70-aa2d-76edf4e79908","Type":"ContainerStarted","Data":"08d61e3af96b42877153599dfe83be4c1bde98c9d6fd235853fc79131e73d2ab"} Jul 18 12:37:59.414221 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:37:59.414173 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-6lsxg/must-gather-bf2rx" podStartSLOduration=1.63745383 podStartE2EDuration="2.414156992s" totalImagesPullingTime="776.703162ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-18 12:37:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-18 12:37:59.41036449 +0000 UTC m=+2946.490570847" watchObservedRunningTime="2026-07-18 12:37:59.414156992 +0000 UTC m=+2946.494363350" Jul 18 12:38:00.682143 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:38:00.682111 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-sp74l_62e02fd7-1e2e-4f92-9989-08bb7b819cfa/global-pull-secret-syncer/0.log" Jul 18 12:38:00.837824 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:38:00.837774 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-844xq_a2e064ca-7d18-4e4a-952c-c00aebf5deb4/konnectivity-agent/0.log" Jul 18 12:38:00.877232 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:38:00.877186 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-136-240.ec2.internal_a5cb0d1fcbe931b511b49dbd31969429/haproxy/0.log" Jul 18 12:38:06.246398 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:38:06.246327 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-75cc8d5cb7-vbvt6_3a1aa8e5-bd4a-4cf2-b4d4-25c0dae5c37e/manager/0.log" Jul 18 12:38:06.278613 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:38:06.278582 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-864dd78c4c-vq5ct_1f66be24-7dfd-4669-9f68-464837b42d1b/manager/0.log" Jul 18 12:38:06.355822 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:38:06.355779 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-catalog-75bjg_72eecc76-acef-47ea-b4f2-d0fa11afb272/registry-server/0.log" Jul 18 12:38:06.591834 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:38:06.591725 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-859d7b78bd-5kx76_2f6f85c7-2112-4511-b178-6d602a8ce6fe/manager/0.log" Jul 18 12:38:08.149856 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:38:08.149767 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_c6ca8f97-d51f-425d-9a79-58d7de301827/alertmanager/0.log" Jul 18 12:38:08.174756 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:38:08.174726 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_c6ca8f97-d51f-425d-9a79-58d7de301827/config-reloader/0.log" Jul 18 12:38:08.197416 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:38:08.197389 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_c6ca8f97-d51f-425d-9a79-58d7de301827/kube-rbac-proxy-web/0.log" Jul 18 12:38:08.217404 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:38:08.217378 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_c6ca8f97-d51f-425d-9a79-58d7de301827/kube-rbac-proxy/0.log" Jul 18 12:38:08.238084 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:38:08.238008 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_c6ca8f97-d51f-425d-9a79-58d7de301827/kube-rbac-proxy-metric/0.log" Jul 18 12:38:08.258958 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:38:08.258877 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_c6ca8f97-d51f-425d-9a79-58d7de301827/prom-label-proxy/0.log" Jul 18 12:38:08.279560 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:38:08.279515 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_c6ca8f97-d51f-425d-9a79-58d7de301827/init-config-reloader/0.log" Jul 18 12:38:08.341458 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:38:08.341428 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-56c5594674-z2dzf_2c36ca40-cd3b-4b4d-a0d0-b725909d65ca/kube-state-metrics/0.log" Jul 18 12:38:08.359347 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:38:08.359300 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-56c5594674-z2dzf_2c36ca40-cd3b-4b4d-a0d0-b725909d65ca/kube-rbac-proxy-main/0.log" Jul 18 12:38:08.389457 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:38:08.389404 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-56c5594674-z2dzf_2c36ca40-cd3b-4b4d-a0d0-b725909d65ca/kube-rbac-proxy-self/0.log" Jul 18 12:38:08.415442 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:38:08.415243 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-5f695cc87f-9rppb_456885f6-d25a-4ae8-8ce2-d80c621900af/metrics-server/0.log" Jul 18 12:38:08.538735 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:38:08.538685 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-5qrx7_60e1cfe5-abff-405b-9ed9-083ec6ab7b73/node-exporter/0.log" Jul 18 12:38:08.565680 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:38:08.565626 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-5qrx7_60e1cfe5-abff-405b-9ed9-083ec6ab7b73/kube-rbac-proxy/0.log" Jul 18 12:38:08.587317 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:38:08.587281 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-5qrx7_60e1cfe5-abff-405b-9ed9-083ec6ab7b73/init-textfile/0.log" Jul 18 12:38:08.704932 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:38:08.704421 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-79f6958cdf-dfvf2_02675456-e24b-4ca5-aa20-55083a8495f5/kube-rbac-proxy-main/0.log" Jul 18 12:38:08.729297 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:38:08.729258 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-79f6958cdf-dfvf2_02675456-e24b-4ca5-aa20-55083a8495f5/kube-rbac-proxy-self/0.log" Jul 18 12:38:08.749679 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:38:08.749654 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-79f6958cdf-dfvf2_02675456-e24b-4ca5-aa20-55083a8495f5/openshift-state-metrics/0.log" Jul 18 12:38:09.020269 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:38:09.020234 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-7f6cccbddb-lztkh_8d8d0932-1f5f-4fdb-aaae-7bd83340950f/telemeter-client/0.log" Jul 18 12:38:09.050387 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:38:09.050285 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-7f6cccbddb-lztkh_8d8d0932-1f5f-4fdb-aaae-7bd83340950f/reload/0.log" Jul 18 12:38:09.072723 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:38:09.072692 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-7f6cccbddb-lztkh_8d8d0932-1f5f-4fdb-aaae-7bd83340950f/kube-rbac-proxy/0.log" Jul 18 12:38:11.016927 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:38:11.016888 2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-6lsxg/perf-node-gather-daemonset-sq5c2"] Jul 18 12:38:11.025117 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:38:11.025074 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-6lsxg/perf-node-gather-daemonset-sq5c2" Jul 18 12:38:11.031281 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:38:11.031247 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-6lsxg/perf-node-gather-daemonset-sq5c2"] Jul 18 12:38:11.058573 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:38:11.058151 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/52f9a553-853e-40c6-9043-b8d19a64f2e3-proc\") pod \"perf-node-gather-daemonset-sq5c2\" (UID: \"52f9a553-853e-40c6-9043-b8d19a64f2e3\") " pod="openshift-must-gather-6lsxg/perf-node-gather-daemonset-sq5c2" Jul 18 12:38:11.058573 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:38:11.058209 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/52f9a553-853e-40c6-9043-b8d19a64f2e3-sys\") pod \"perf-node-gather-daemonset-sq5c2\" (UID: \"52f9a553-853e-40c6-9043-b8d19a64f2e3\") " pod="openshift-must-gather-6lsxg/perf-node-gather-daemonset-sq5c2" Jul 18 12:38:11.058573 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:38:11.058259 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/52f9a553-853e-40c6-9043-b8d19a64f2e3-lib-modules\") pod \"perf-node-gather-daemonset-sq5c2\" (UID: \"52f9a553-853e-40c6-9043-b8d19a64f2e3\") " pod="openshift-must-gather-6lsxg/perf-node-gather-daemonset-sq5c2" Jul 18 12:38:11.058573 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:38:11.058294 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mntzk\" (UniqueName: \"kubernetes.io/projected/52f9a553-853e-40c6-9043-b8d19a64f2e3-kube-api-access-mntzk\") pod \"perf-node-gather-daemonset-sq5c2\" (UID: \"52f9a553-853e-40c6-9043-b8d19a64f2e3\") " pod="openshift-must-gather-6lsxg/perf-node-gather-daemonset-sq5c2" Jul 18 12:38:11.058573 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:38:11.058333 2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/52f9a553-853e-40c6-9043-b8d19a64f2e3-podres\") pod \"perf-node-gather-daemonset-sq5c2\" (UID: \"52f9a553-853e-40c6-9043-b8d19a64f2e3\") " pod="openshift-must-gather-6lsxg/perf-node-gather-daemonset-sq5c2" Jul 18 12:38:11.062135 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:38:11.061958 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-j9l7p_8ec47eff-5fbc-4130-baef-d8544aa4e786/console-operator/1.log" Jul 18 12:38:11.067934 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:38:11.067908 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-j9l7p_8ec47eff-5fbc-4130-baef-d8544aa4e786/console-operator/2.log" Jul 18 12:38:11.159636 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:38:11.159596 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/52f9a553-853e-40c6-9043-b8d19a64f2e3-proc\") pod \"perf-node-gather-daemonset-sq5c2\" (UID: \"52f9a553-853e-40c6-9043-b8d19a64f2e3\") " pod="openshift-must-gather-6lsxg/perf-node-gather-daemonset-sq5c2" Jul 18 12:38:11.159818 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:38:11.159659 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/52f9a553-853e-40c6-9043-b8d19a64f2e3-sys\") pod \"perf-node-gather-daemonset-sq5c2\" (UID: \"52f9a553-853e-40c6-9043-b8d19a64f2e3\") " pod="openshift-must-gather-6lsxg/perf-node-gather-daemonset-sq5c2" Jul 18 12:38:11.159818 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:38:11.159707 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/52f9a553-853e-40c6-9043-b8d19a64f2e3-lib-modules\") pod \"perf-node-gather-daemonset-sq5c2\" (UID: \"52f9a553-853e-40c6-9043-b8d19a64f2e3\") " pod="openshift-must-gather-6lsxg/perf-node-gather-daemonset-sq5c2" Jul 18 12:38:11.159818 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:38:11.159738 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mntzk\" (UniqueName: \"kubernetes.io/projected/52f9a553-853e-40c6-9043-b8d19a64f2e3-kube-api-access-mntzk\") pod \"perf-node-gather-daemonset-sq5c2\" (UID: \"52f9a553-853e-40c6-9043-b8d19a64f2e3\") " pod="openshift-must-gather-6lsxg/perf-node-gather-daemonset-sq5c2" Jul 18 12:38:11.159818 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:38:11.159773 2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/52f9a553-853e-40c6-9043-b8d19a64f2e3-podres\") pod \"perf-node-gather-daemonset-sq5c2\" (UID: \"52f9a553-853e-40c6-9043-b8d19a64f2e3\") " pod="openshift-must-gather-6lsxg/perf-node-gather-daemonset-sq5c2" Jul 18 12:38:11.163767 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:38:11.163734 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/52f9a553-853e-40c6-9043-b8d19a64f2e3-lib-modules\") pod \"perf-node-gather-daemonset-sq5c2\" (UID: \"52f9a553-853e-40c6-9043-b8d19a64f2e3\") " pod="openshift-must-gather-6lsxg/perf-node-gather-daemonset-sq5c2" Jul 18 12:38:11.163911 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:38:11.163837 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/52f9a553-853e-40c6-9043-b8d19a64f2e3-proc\") pod \"perf-node-gather-daemonset-sq5c2\" (UID: \"52f9a553-853e-40c6-9043-b8d19a64f2e3\") " pod="openshift-must-gather-6lsxg/perf-node-gather-daemonset-sq5c2" Jul 18 12:38:11.163911 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:38:11.163895 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/52f9a553-853e-40c6-9043-b8d19a64f2e3-sys\") pod \"perf-node-gather-daemonset-sq5c2\" (UID: \"52f9a553-853e-40c6-9043-b8d19a64f2e3\") " pod="openshift-must-gather-6lsxg/perf-node-gather-daemonset-sq5c2" Jul 18 12:38:11.164374 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:38:11.164345 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/52f9a553-853e-40c6-9043-b8d19a64f2e3-podres\") pod \"perf-node-gather-daemonset-sq5c2\" (UID: \"52f9a553-853e-40c6-9043-b8d19a64f2e3\") " pod="openshift-must-gather-6lsxg/perf-node-gather-daemonset-sq5c2" Jul 18 12:38:11.180332 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:38:11.180297 2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mntzk\" (UniqueName: \"kubernetes.io/projected/52f9a553-853e-40c6-9043-b8d19a64f2e3-kube-api-access-mntzk\") pod \"perf-node-gather-daemonset-sq5c2\" (UID: \"52f9a553-853e-40c6-9043-b8d19a64f2e3\") " pod="openshift-must-gather-6lsxg/perf-node-gather-daemonset-sq5c2" Jul 18 12:38:11.343149 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:38:11.343061 2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-6lsxg/perf-node-gather-daemonset-sq5c2" Jul 18 12:38:11.526376 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:38:11.526343 2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-6lsxg/perf-node-gather-daemonset-sq5c2"] Jul 18 12:38:11.531598 ip-10-0-136-240 kubenswrapper[2585]: W0718 12:38:11.528232 2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod52f9a553_853e_40c6_9043_b8d19a64f2e3.slice/crio-b75102e645aa81b1fed172d1aeb2915ac7c84e1a1de1df0e691d744b6f21de88 WatchSource:0}: Error finding container b75102e645aa81b1fed172d1aeb2915ac7c84e1a1de1df0e691d744b6f21de88: Status 404 returned error can't find the container with id b75102e645aa81b1fed172d1aeb2915ac7c84e1a1de1df0e691d744b6f21de88 Jul 18 12:38:11.714234 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:38:11.714013 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-5bc67dcd46-rpxdk_cd834b16-b441-4739-a530-75ca9dacba75/download-server/0.log" Jul 18 12:38:12.331220 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:38:12.331192 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-78cf4c6f6-gftqx_79fb452b-b19a-48d8-930d-679ea447ee0a/volume-data-source-validator/0.log" Jul 18 12:38:12.491274 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:38:12.491230 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-6lsxg/perf-node-gather-daemonset-sq5c2" event={"ID":"52f9a553-853e-40c6-9043-b8d19a64f2e3","Type":"ContainerStarted","Data":"cd557d287dee026366424fe7dca0b072813db8afa56ac0af747e7a76d79a1a0a"} Jul 18 12:38:12.491471 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:38:12.491345 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-6lsxg/perf-node-gather-daemonset-sq5c2" Jul 18 12:38:12.491471 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:38:12.491371 2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-6lsxg/perf-node-gather-daemonset-sq5c2" event={"ID":"52f9a553-853e-40c6-9043-b8d19a64f2e3","Type":"ContainerStarted","Data":"b75102e645aa81b1fed172d1aeb2915ac7c84e1a1de1df0e691d744b6f21de88"} Jul 18 12:38:12.508092 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:38:12.508036 2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-6lsxg/perf-node-gather-daemonset-sq5c2" podStartSLOduration=2.508014669 podStartE2EDuration="2.508014669s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-18 12:38:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-18 12:38:12.507018804 +0000 UTC m=+2959.587225163" watchObservedRunningTime="2026-07-18 12:38:12.508014669 +0000 UTC m=+2959.588221029" Jul 18 12:38:13.375240 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:38:13.375212 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-mrh2r_42cb6807-1262-4919-856b-80c7b6d08ae2/dns/0.log" Jul 18 12:38:13.395455 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:38:13.395410 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-mrh2r_42cb6807-1262-4919-856b-80c7b6d08ae2/kube-rbac-proxy/0.log" Jul 18 12:38:13.460033 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:38:13.459999 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-hv6g8_97228923-3172-4691-927f-723ef88035cf/dns-node-resolver/0.log" Jul 18 12:38:14.027847 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:38:14.027812 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-6d5c77756-pkbzb_b9ae0f78-c6f7-441c-a614-00c96d69dd39/registry/0.log" Jul 18 12:38:14.087095 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:38:14.087058 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-nzpfs_d12e3a2e-a05d-4659-b74e-c30e765448bf/node-ca/0.log" Jul 18 12:38:15.022402 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:38:15.022371 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-6576859cf49clx6_d999136c-b948-4ed1-a61a-ad0cdf3e339d/istio-proxy/0.log" Jul 18 12:38:15.313903 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:38:15.313825 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-2gcb2_faf0508a-7db7-4cbe-915a-499c404d0bd0/istio-proxy/0.log" Jul 18 12:38:15.353443 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:38:15.353409 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-2gcb2_faf0508a-7db7-4cbe-915a-499c404d0bd0/istio-proxy/1.log" Jul 18 12:38:15.435803 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:38:15.435776 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-696b95fb6d-hx9nd_9477954a-e6a6-4bba-ad8f-3a8788b2b2ec/router/0.log" Jul 18 12:38:16.173302 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:38:16.173274 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-88757_056ad49c-5e1c-4331-8a8f-65c7202be995/serve-healthcheck-canary/0.log" Jul 18 12:38:16.630449 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:38:16.630413 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-555d9d7885-44h2s_43f38029-3002-493c-a718-2fe047c3cd99/insights-operator/0.log" Jul 18 12:38:16.630988 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:38:16.630965 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-555d9d7885-44h2s_43f38029-3002-493c-a718-2fe047c3cd99/insights-operator/1.log" Jul 18 12:38:16.712038 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:38:16.712005 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-hhwkn_f44aedd4-16f8-4adb-a7b7-51719a187834/kube-rbac-proxy/0.log" Jul 18 12:38:16.731006 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:38:16.730980 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-hhwkn_f44aedd4-16f8-4adb-a7b7-51719a187834/exporter/0.log" Jul 18 12:38:16.749534 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:38:16.749504 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-hhwkn_f44aedd4-16f8-4adb-a7b7-51719a187834/extractor/0.log" Jul 18 12:38:18.509854 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:38:18.509136 2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-6lsxg/perf-node-gather-daemonset-sq5c2" Jul 18 12:38:19.567288 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:38:19.567242 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-fbb59dc99-9rkcw_d46e8b83-a5c7-44b0-aa32-7c68ac538c32/manager/0.log" Jul 18 12:38:19.585176 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:38:19.585151 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-fbb59dc99-9rkcw_d46e8b83-a5c7-44b0-aa32-7c68ac538c32/copy-manifests/0.log" Jul 18 12:38:21.121733 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:38:21.121644 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-77c8bd676b-lsmbc_98206ec9-6aca-4f15-8186-3b6b2c4aec08/manager/0.log" Jul 18 12:38:21.170075 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:38:21.170031 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_openshift-lws-operator-fd8ccff4c-c7gc6_cb99d1b5-ebd0-4feb-8aca-0c6b827c0c4a/openshift-lws-operator/0.log" Jul 18 12:38:21.689213 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:38:21.689176 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-6576859cf49clx6_d999136c-b948-4ed1-a61a-ad0cdf3e339d/istio-proxy/0.log" Jul 18 12:38:21.954650 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:38:21.954540 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-2gcb2_faf0508a-7db7-4cbe-915a-499c404d0bd0/istio-proxy/0.log" Jul 18 12:38:21.995153 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:38:21.995106 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-2gcb2_faf0508a-7db7-4cbe-915a-499c404d0bd0/istio-proxy/1.log" Jul 18 12:38:22.059989 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:38:22.059960 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-696b95fb6d-hx9nd_9477954a-e6a6-4bba-ad8f-3a8788b2b2ec/router/0.log" Jul 18 12:38:23.255122 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:38:23.255094 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-6576859cf49clx6_d999136c-b948-4ed1-a61a-ad0cdf3e339d/istio-proxy/0.log" Jul 18 12:38:23.524946 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:38:23.524917 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-2gcb2_faf0508a-7db7-4cbe-915a-499c404d0bd0/istio-proxy/0.log" Jul 18 12:38:23.562491 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:38:23.562462 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-2gcb2_faf0508a-7db7-4cbe-915a-499c404d0bd0/istio-proxy/1.log" Jul 18 12:38:23.622178 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:38:23.622151 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-696b95fb6d-hx9nd_9477954a-e6a6-4bba-ad8f-3a8788b2b2ec/router/0.log" Jul 18 12:38:26.512826 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:38:26.512795 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-7658d9b749-29ngm_45cc3bd5-b19f-4ac1-9257-6c39802b8e2f/kube-storage-version-migrator-operator/1.log" Jul 18 12:38:26.513818 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:38:26.513799 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-7658d9b749-29ngm_45cc3bd5-b19f-4ac1-9257-6c39802b8e2f/kube-storage-version-migrator-operator/0.log" Jul 18 12:38:27.606355 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:38:27.606331 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-9xp94_760609c0-c9c8-44d2-87f4-7fd81b6eb7da/kube-multus-additional-cni-plugins/0.log" Jul 18 12:38:27.625826 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:38:27.625799 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-9xp94_760609c0-c9c8-44d2-87f4-7fd81b6eb7da/egress-router-binary-copy/0.log" Jul 18 12:38:27.644284 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:38:27.644259 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-9xp94_760609c0-c9c8-44d2-87f4-7fd81b6eb7da/cni-plugins/0.log" Jul 18 12:38:27.663458 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:38:27.663436 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-9xp94_760609c0-c9c8-44d2-87f4-7fd81b6eb7da/bond-cni-plugin/0.log" Jul 18 12:38:27.681439 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:38:27.681418 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-9xp94_760609c0-c9c8-44d2-87f4-7fd81b6eb7da/routeoverride-cni/0.log" Jul 18 12:38:27.701597 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:38:27.701542 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-9xp94_760609c0-c9c8-44d2-87f4-7fd81b6eb7da/whereabouts-cni-bincopy/0.log" Jul 18 12:38:27.720514 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:38:27.720488 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-9xp94_760609c0-c9c8-44d2-87f4-7fd81b6eb7da/whereabouts-cni/0.log" Jul 18 12:38:28.020762 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:38:28.020726 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-xlwqp_3072709c-b577-46c4-9616-f0ca460c4196/kube-multus/0.log" Jul 18 12:38:28.040067 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:38:28.040041 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-hppk2_4731170e-9d96-4544-a641-5fc8e1659d8e/network-metrics-daemon/0.log" Jul 18 12:38:28.055868 ip-10-0-136-240 kubenswrapper[2585]: I0718 12:38:28.055840 2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-hppk2_4731170e-9d96-4544-a641-5fc8e1659d8e/kube-rbac-proxy/0.log"