Jul 09 22:08:20.144393 ip-10-0-136-251 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory Jul 09 22:08:20.144406 ip-10-0-136-251 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory Jul 09 22:08:20.144416 ip-10-0-136-251 systemd[1]: kubelet.service: Failed with result 'resources'. Jul 09 22:08:20.144900 ip-10-0-136-251 systemd[1]: Failed to start Kubernetes Kubelet. Jul 09 22:08:30.194308 ip-10-0-136-251 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found. Jul 09 22:08:30.194324 ip-10-0-136-251 systemd[1]: kubelet.service: Failed with result 'resources'. -- Boot c05a7e41391746489854b1f6aa21196f -- Jul 09 22:10:06.741759 ip-10-0-136-251 systemd[1]: Starting Kubernetes Kubelet... Jul 09 22:10:07.155818 ip-10-0-136-251 kubenswrapper[2579]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 09 22:10:07.155818 ip-10-0-136-251 kubenswrapper[2579]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Jul 09 22:10:07.155818 ip-10-0-136-251 kubenswrapper[2579]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 09 22:10:07.155818 ip-10-0-136-251 kubenswrapper[2579]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Jul 09 22:10:07.155818 ip-10-0-136-251 kubenswrapper[2579]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 09 22:10:07.155818 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:07.140462 2579 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Jul 09 22:10:07.155818 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:10:07.144991 2579 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 09 22:10:07.155818 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:10:07.145003 2579 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 09 22:10:07.155818 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:10:07.145006 2579 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 09 22:10:07.155818 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:10:07.145011 2579 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 09 22:10:07.155818 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:10:07.145015 2579 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 09 22:10:07.155818 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:10:07.145018 2579 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 09 22:10:07.155818 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:10:07.145021 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 09 22:10:07.155818 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:10:07.145024 2579 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 09 22:10:07.316366 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:10:07.145027 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 09 22:10:07.316366 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:10:07.145031 2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 09 22:10:07.316366 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:10:07.145036 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 09 22:10:07.316366 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:10:07.145040 2579 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 09 22:10:07.316366 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:10:07.145057 2579 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 09 22:10:07.316366 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:10:07.145061 2579 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 09 22:10:07.316366 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:10:07.145064 2579 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 09 22:10:07.316366 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:10:07.145067 2579 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 09 22:10:07.316366 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:10:07.145069 2579 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 09 22:10:07.316366 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:10:07.145072 2579 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 09 22:10:07.316366 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:10:07.145076 2579 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 09 22:10:07.316366 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:10:07.145078 2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 09 22:10:07.316366 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:10:07.145081 2579 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 09 22:10:07.316366 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:10:07.145085 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 09 22:10:07.316366 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:10:07.145089 2579 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 09 22:10:07.316366 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:10:07.145094 2579 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 09 22:10:07.316366 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:10:07.145097 2579 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 09 22:10:07.316366 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:10:07.145100 2579 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 09 22:10:07.316366 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:10:07.145103 2579 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 09 22:10:07.212115 ip-10-0-136-251 systemd[1]: Started Kubernetes Kubelet. Jul 09 22:10:07.330122 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:10:07.145106 2579 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 09 22:10:07.330122 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:10:07.145109 2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 09 22:10:07.330122 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:10:07.145112 2579 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 09 22:10:07.330122 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:10:07.145114 2579 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 09 22:10:07.330122 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:10:07.145117 2579 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 09 22:10:07.330122 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:10:07.145121 2579 feature_gate.go:328] unrecognized feature gate: Example2 Jul 09 22:10:07.330122 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:10:07.145123 2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 09 22:10:07.330122 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:10:07.145126 2579 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 09 22:10:07.330122 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:10:07.145130 2579 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 09 22:10:07.330122 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:10:07.145141 2579 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 09 22:10:07.330122 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:10:07.145144 2579 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 09 22:10:07.330122 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:10:07.145147 2579 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 09 22:10:07.330122 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:10:07.145149 2579 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 09 22:10:07.330122 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:10:07.145153 2579 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 09 22:10:07.330122 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:10:07.145155 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 09 22:10:07.330122 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:10:07.145158 2579 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 09 22:10:07.330122 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:10:07.145160 2579 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 09 22:10:07.330122 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:10:07.145163 2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 09 22:10:07.330122 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:10:07.145165 2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 09 22:10:07.330122 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:10:07.145168 2579 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 09 22:10:07.332744 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:10:07.145170 2579 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 09 22:10:07.332744 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:10:07.145173 2579 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 09 22:10:07.332744 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:10:07.145177 2579 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 09 22:10:07.332744 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:10:07.145179 2579 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 09 22:10:07.332744 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:10:07.145182 2579 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 09 22:10:07.332744 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:10:07.145185 2579 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 09 22:10:07.332744 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:10:07.145188 2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 09 22:10:07.332744 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:10:07.145191 2579 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 09 22:10:07.332744 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:10:07.145194 2579 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 09 22:10:07.332744 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:10:07.145196 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 09 22:10:07.332744 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:10:07.145199 2579 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 09 22:10:07.332744 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:10:07.145202 2579 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 09 22:10:07.332744 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:10:07.145204 2579 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 09 22:10:07.332744 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:10:07.145207 2579 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 09 22:10:07.332744 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:10:07.145209 2579 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 09 22:10:07.332744 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:10:07.145212 2579 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 09 22:10:07.332744 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:10:07.145215 2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 09 22:10:07.332744 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:10:07.145218 2579 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 09 22:10:07.332744 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:10:07.145220 2579 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 09 22:10:07.332744 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:10:07.145223 2579 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 09 22:10:07.336243 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:10:07.145227 2579 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 09 22:10:07.336243 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:10:07.145230 2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 09 22:10:07.336243 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:10:07.145234 2579 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 09 22:10:07.336243 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:10:07.145237 2579 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 09 22:10:07.336243 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:10:07.145240 2579 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 09 22:10:07.336243 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:10:07.145244 2579 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 09 22:10:07.336243 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:10:07.145246 2579 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 09 22:10:07.336243 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:10:07.145249 2579 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 09 22:10:07.336243 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:10:07.145252 2579 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 09 22:10:07.336243 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:10:07.145254 2579 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 09 22:10:07.336243 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:10:07.145257 2579 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 09 22:10:07.336243 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:10:07.145260 2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 09 22:10:07.336243 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:10:07.145262 2579 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 09 22:10:07.336243 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:10:07.145265 2579 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 09 22:10:07.336243 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:10:07.145268 2579 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 09 22:10:07.336243 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:10:07.145271 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 09 22:10:07.336243 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:10:07.145274 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 09 22:10:07.336243 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:10:07.145276 2579 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 09 22:10:07.336243 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:10:07.145279 2579 feature_gate.go:328] unrecognized feature gate: Example Jul 09 22:10:07.336243 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:10:07.145282 2579 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 09 22:10:07.338029 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:10:07.145285 2579 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 09 22:10:07.338029 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:10:07.145287 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 09 22:10:07.338029 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:10:07.145290 2579 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 09 22:10:07.338029 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:10:07.145293 2579 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 09 22:10:07.338029 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:10:07.145295 2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 09 22:10:07.338029 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:10:07.145298 2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 09 22:10:07.338029 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:10:07.145300 2579 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 09 22:10:07.338029 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:10:07.145303 2579 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 09 22:10:07.338029 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:10:07.145305 2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 09 22:10:07.338029 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:10:07.145308 2579 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 09 22:10:07.338029 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:10:07.145312 2579 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 09 22:10:07.338029 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:10:07.145316 2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 09 22:10:07.338029 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:10:07.146868 2579 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 09 22:10:07.338029 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:10:07.146881 2579 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 09 22:10:07.338029 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:10:07.146885 2579 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 09 22:10:07.338029 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:10:07.146888 2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 09 22:10:07.338029 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:10:07.146891 2579 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 09 22:10:07.338029 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:10:07.146894 2579 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 09 22:10:07.338029 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:10:07.146897 2579 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 09 22:10:07.340773 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:10:07.146899 2579 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 09 22:10:07.340773 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:10:07.146902 2579 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 09 22:10:07.340773 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:10:07.146905 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 09 22:10:07.340773 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:10:07.146908 2579 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 09 22:10:07.340773 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:10:07.146910 2579 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 09 22:10:07.340773 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:10:07.146913 2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 09 22:10:07.340773 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:10:07.146916 2579 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 09 22:10:07.340773 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:10:07.146918 2579 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 09 22:10:07.340773 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:10:07.146921 2579 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 09 22:10:07.340773 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:10:07.146924 2579 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 09 22:10:07.340773 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:10:07.146928 2579 feature_gate.go:328] unrecognized feature gate: Example2 Jul 09 22:10:07.340773 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:10:07.146930 2579 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 09 22:10:07.340773 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:10:07.146933 2579 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 09 22:10:07.340773 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:10:07.146936 2579 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 09 22:10:07.340773 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:10:07.146938 2579 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 09 22:10:07.340773 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:10:07.146941 2579 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 09 22:10:07.340773 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:10:07.146944 2579 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 09 22:10:07.340773 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:10:07.146947 2579 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 09 22:10:07.340773 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:10:07.146949 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 09 22:10:07.340773 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:10:07.146952 2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 09 22:10:07.344751 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:10:07.146954 2579 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 09 22:10:07.344751 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:10:07.146957 2579 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 09 22:10:07.344751 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:10:07.146961 2579 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 09 22:10:07.344751 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:10:07.146965 2579 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 09 22:10:07.344751 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:10:07.146968 2579 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 09 22:10:07.344751 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:10:07.146970 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 09 22:10:07.344751 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:10:07.146973 2579 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 09 22:10:07.344751 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:10:07.146975 2579 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 09 22:10:07.344751 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:10:07.146978 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 09 22:10:07.344751 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:10:07.146981 2579 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 09 22:10:07.344751 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:10:07.146984 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 09 22:10:07.344751 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:10:07.146986 2579 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 09 22:10:07.344751 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:10:07.146990 2579 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 09 22:10:07.344751 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:10:07.146993 2579 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 09 22:10:07.344751 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:10:07.146995 2579 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 09 22:10:07.344751 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:10:07.146998 2579 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 09 22:10:07.344751 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:10:07.147000 2579 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 09 22:10:07.344751 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:10:07.147003 2579 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 09 22:10:07.344751 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:10:07.147006 2579 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 09 22:10:07.344751 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:10:07.147008 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 09 22:10:07.345341 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:10:07.147010 2579 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 09 22:10:07.345341 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:10:07.147013 2579 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 09 22:10:07.345341 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:10:07.147016 2579 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 09 22:10:07.345341 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:10:07.147019 2579 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 09 22:10:07.345341 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:10:07.147022 2579 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 09 22:10:07.345341 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:10:07.147024 2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 09 22:10:07.345341 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:10:07.147027 2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 09 22:10:07.345341 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:10:07.147029 2579 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 09 22:10:07.345341 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:10:07.147032 2579 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 09 22:10:07.345341 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:10:07.147035 2579 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 09 22:10:07.345341 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:10:07.147037 2579 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 09 22:10:07.345341 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:10:07.147040 2579 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 09 22:10:07.345341 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:10:07.147043 2579 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 09 22:10:07.345341 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:10:07.147046 2579 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 09 22:10:07.345341 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:10:07.147048 2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 09 22:10:07.345341 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:10:07.147051 2579 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 09 22:10:07.345341 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:10:07.147053 2579 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 09 22:10:07.345341 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:10:07.147056 2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 09 22:10:07.345341 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:10:07.147059 2579 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 09 22:10:07.345341 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:10:07.147061 2579 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 09 22:10:07.345904 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:10:07.147064 2579 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 09 22:10:07.345904 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:10:07.147066 2579 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 09 22:10:07.345904 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:10:07.147069 2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 09 22:10:07.345904 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:10:07.147072 2579 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 09 22:10:07.345904 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:10:07.147074 2579 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 09 22:10:07.345904 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:10:07.147076 2579 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 09 22:10:07.345904 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:10:07.147079 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 09 22:10:07.345904 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:10:07.147082 2579 feature_gate.go:328] unrecognized feature gate: Example Jul 09 22:10:07.345904 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:10:07.147084 2579 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 09 22:10:07.345904 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:10:07.147087 2579 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 09 22:10:07.345904 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:10:07.147089 2579 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 09 22:10:07.345904 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:10:07.147092 2579 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 09 22:10:07.345904 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:10:07.147094 2579 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 09 22:10:07.345904 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:10:07.147096 2579 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 09 22:10:07.345904 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:10:07.147099 2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 09 22:10:07.345904 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:10:07.147103 2579 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 09 22:10:07.345904 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:10:07.147109 2579 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 09 22:10:07.345904 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:10:07.147115 2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 09 22:10:07.345904 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:10:07.147118 2579 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 09 22:10:07.346419 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:10:07.147121 2579 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 09 22:10:07.346419 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:10:07.147124 2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 09 22:10:07.346419 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:10:07.147127 2579 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 09 22:10:07.346419 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:10:07.147130 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 09 22:10:07.346419 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:10:07.147133 2579 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 09 22:10:07.346419 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:10:07.147135 2579 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 09 22:10:07.346419 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:10:07.147138 2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 09 22:10:07.346419 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:10:07.147141 2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 09 22:10:07.346419 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:10:07.147144 2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 09 22:10:07.346419 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:10:07.147146 2579 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 09 22:10:07.346419 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:10:07.147149 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 09 22:10:07.346419 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:10:07.147152 2579 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 09 22:10:07.346419 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:10:07.147154 2579 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 09 22:10:07.346419 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:10:07.147563 2579 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 09 22:10:07.346419 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:10:07.147568 2579 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 09 22:10:07.346419 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:10:07.147571 2579 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 09 22:10:07.346419 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:10:07.147574 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 09 22:10:07.346419 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:10:07.147577 2579 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 09 22:10:07.346419 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:10:07.147580 2579 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 09 22:10:07.346987 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:10:07.147584 2579 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 09 22:10:07.346987 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:10:07.147588 2579 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 09 22:10:07.346987 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:10:07.147591 2579 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 09 22:10:07.346987 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:10:07.147594 2579 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 09 22:10:07.346987 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:10:07.147596 2579 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 09 22:10:07.346987 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:10:07.147599 2579 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 09 22:10:07.346987 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:10:07.147601 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 09 22:10:07.346987 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:10:07.147604 2579 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 09 22:10:07.346987 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:10:07.147607 2579 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 09 22:10:07.346987 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:10:07.147611 2579 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 09 22:10:07.346987 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:10:07.147613 2579 feature_gate.go:328] unrecognized feature gate: Example Jul 09 22:10:07.346987 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:10:07.147618 2579 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 09 22:10:07.346987 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:10:07.147620 2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 09 22:10:07.346987 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:10:07.147623 2579 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 09 22:10:07.346987 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:10:07.147626 2579 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 09 22:10:07.346987 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:10:07.147628 2579 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 09 22:10:07.346987 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:10:07.147632 2579 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 09 22:10:07.346987 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:10:07.147635 2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 09 22:10:07.346987 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:10:07.147638 2579 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 09 22:10:07.347566 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:10:07.147641 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 09 22:10:07.347566 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:10:07.147644 2579 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 09 22:10:07.347566 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:10:07.147646 2579 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 09 22:10:07.347566 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:10:07.147649 2579 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 09 22:10:07.347566 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:10:07.147651 2579 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 09 22:10:07.347566 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:10:07.147654 2579 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 09 22:10:07.347566 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:10:07.147656 2579 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 09 22:10:07.347566 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:10:07.147659 2579 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 09 22:10:07.347566 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:10:07.147662 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 09 22:10:07.347566 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:10:07.147665 2579 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 09 22:10:07.347566 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:10:07.147667 2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 09 22:10:07.347566 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:10:07.147670 2579 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 09 22:10:07.347566 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:10:07.147673 2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 09 22:10:07.347566 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:10:07.147675 2579 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 09 22:10:07.347566 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:10:07.147678 2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 09 22:10:07.347566 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:10:07.147680 2579 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 09 22:10:07.347566 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:10:07.147683 2579 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 09 22:10:07.347566 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:10:07.147685 2579 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 09 22:10:07.347566 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:10:07.147687 2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 09 22:10:07.347566 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:10:07.147690 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 09 22:10:07.348082 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:10:07.147693 2579 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 09 22:10:07.348082 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:10:07.147697 2579 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 09 22:10:07.348082 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:10:07.147700 2579 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 09 22:10:07.348082 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:10:07.147703 2579 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 09 22:10:07.348082 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:10:07.147706 2579 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 09 22:10:07.348082 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:10:07.147709 2579 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 09 22:10:07.348082 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:10:07.147712 2579 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 09 22:10:07.348082 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:10:07.147714 2579 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 09 22:10:07.348082 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:10:07.147717 2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 09 22:10:07.348082 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:10:07.147719 2579 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 09 22:10:07.348082 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:10:07.147722 2579 feature_gate.go:328] unrecognized feature gate: Example2 Jul 09 22:10:07.348082 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:10:07.147725 2579 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 09 22:10:07.348082 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:10:07.147727 2579 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 09 22:10:07.348082 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:10:07.147730 2579 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 09 22:10:07.348082 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:10:07.147732 2579 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 09 22:10:07.348082 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:10:07.147735 2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 09 22:10:07.348082 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:10:07.147737 2579 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 09 22:10:07.348082 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:10:07.147740 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 09 22:10:07.348082 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:10:07.147742 2579 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 09 22:10:07.348082 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:10:07.147745 2579 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 09 22:10:07.348082 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:10:07.147747 2579 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 09 22:10:07.348709 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:10:07.147750 2579 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 09 22:10:07.348709 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:10:07.147752 2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 09 22:10:07.348709 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:10:07.147755 2579 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 09 22:10:07.348709 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:10:07.147757 2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 09 22:10:07.348709 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:10:07.147760 2579 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 09 22:10:07.348709 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:10:07.147762 2579 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 09 22:10:07.348709 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:10:07.147765 2579 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 09 22:10:07.348709 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:10:07.147767 2579 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 09 22:10:07.348709 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:10:07.147771 2579 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 09 22:10:07.348709 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:10:07.147775 2579 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 09 22:10:07.348709 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:10:07.147777 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 09 22:10:07.348709 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:10:07.147780 2579 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 09 22:10:07.348709 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:10:07.147786 2579 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 09 22:10:07.348709 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:10:07.147789 2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 09 22:10:07.348709 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:10:07.147791 2579 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 09 22:10:07.348709 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:10:07.147796 2579 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 09 22:10:07.348709 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:10:07.147798 2579 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 09 22:10:07.348709 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:10:07.147801 2579 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 09 22:10:07.348709 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:10:07.147804 2579 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 09 22:10:07.348709 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:10:07.147806 2579 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 09 22:10:07.349187 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:10:07.147809 2579 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 09 22:10:07.349187 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:10:07.147812 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 09 22:10:07.349187 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:10:07.147814 2579 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 09 22:10:07.349187 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:10:07.147817 2579 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 09 22:10:07.349187 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:10:07.147820 2579 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 09 22:10:07.349187 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:10:07.147823 2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 09 22:10:07.349187 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:10:07.147826 2579 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 09 22:10:07.349187 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:10:07.147828 2579 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 09 22:10:07.349187 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:10:07.147831 2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 09 22:10:07.349187 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:10:07.147833 2579 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 09 22:10:07.349187 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:10:07.147836 2579 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 09 22:10:07.349187 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:10:07.147838 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 09 22:10:07.349187 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:10:07.147841 2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 09 22:10:07.349187 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:07.147912 2579 flags.go:64] FLAG: --address="0.0.0.0" Jul 09 22:10:07.349187 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:07.147919 2579 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Jul 09 22:10:07.349187 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:07.147926 2579 flags.go:64] FLAG: --anonymous-auth="true" Jul 09 22:10:07.349187 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:07.147930 2579 flags.go:64] FLAG: --application-metrics-count-limit="100" Jul 09 22:10:07.349187 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:07.147935 2579 flags.go:64] FLAG: --authentication-token-webhook="false" Jul 09 22:10:07.349187 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:07.147938 2579 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Jul 09 22:10:07.349187 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:07.147943 2579 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Jul 09 22:10:07.349187 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:07.147947 2579 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Jul 09 22:10:07.349751 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:07.147951 2579 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Jul 09 22:10:07.349751 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:07.147954 2579 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Jul 09 22:10:07.349751 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:07.147957 2579 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Jul 09 22:10:07.349751 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:07.147963 2579 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Jul 09 22:10:07.349751 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:07.147966 2579 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Jul 09 22:10:07.349751 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:07.147969 2579 flags.go:64] FLAG: --cgroup-root="" Jul 09 22:10:07.349751 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:07.147973 2579 flags.go:64] FLAG: --cgroups-per-qos="true" Jul 09 22:10:07.349751 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:07.147977 2579 flags.go:64] FLAG: --client-ca-file="" Jul 09 22:10:07.349751 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:07.147980 2579 flags.go:64] FLAG: --cloud-provider="external" Jul 09 22:10:07.349751 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:07.147984 2579 flags.go:64] FLAG: --cluster-dns="[]" Jul 09 22:10:07.349751 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:07.147987 2579 flags.go:64] FLAG: --cluster-domain="" Jul 09 22:10:07.349751 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:07.147990 2579 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Jul 09 22:10:07.349751 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:07.147993 2579 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Jul 09 22:10:07.349751 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:07.147996 2579 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Jul 09 22:10:07.349751 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:07.148000 2579 flags.go:64] FLAG: --container-log-max-files="5" Jul 09 22:10:07.349751 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:07.148004 2579 flags.go:64] FLAG: --container-log-max-size="10Mi" Jul 09 22:10:07.349751 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:07.148007 2579 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Jul 09 22:10:07.349751 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:07.148010 2579 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Jul 09 22:10:07.349751 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:07.148013 2579 flags.go:64] FLAG: --containerd-namespace="k8s.io" Jul 09 22:10:07.349751 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:07.148017 2579 flags.go:64] FLAG: --contention-profiling="false" Jul 09 22:10:07.349751 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:07.148020 2579 flags.go:64] FLAG: --cpu-cfs-quota="true" Jul 09 22:10:07.349751 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:07.148023 2579 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Jul 09 22:10:07.349751 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:07.148027 2579 flags.go:64] FLAG: --cpu-manager-policy="none" Jul 09 22:10:07.349751 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:07.148030 2579 flags.go:64] FLAG: --cpu-manager-policy-options="" Jul 09 22:10:07.350325 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:07.148034 2579 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Jul 09 22:10:07.350325 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:07.148037 2579 flags.go:64] FLAG: --enable-controller-attach-detach="true" Jul 09 22:10:07.350325 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:07.148040 2579 flags.go:64] FLAG: --enable-debugging-handlers="true" Jul 09 22:10:07.350325 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:07.148043 2579 flags.go:64] FLAG: --enable-load-reader="false" Jul 09 22:10:07.350325 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:07.148046 2579 flags.go:64] FLAG: --enable-server="true" Jul 09 22:10:07.350325 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:07.148049 2579 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Jul 09 22:10:07.350325 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:07.148053 2579 flags.go:64] FLAG: --event-burst="100" Jul 09 22:10:07.350325 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:07.148056 2579 flags.go:64] FLAG: --event-qps="50" Jul 09 22:10:07.350325 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:07.148059 2579 flags.go:64] FLAG: --event-storage-age-limit="default=0" Jul 09 22:10:07.350325 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:07.148062 2579 flags.go:64] FLAG: --event-storage-event-limit="default=0" Jul 09 22:10:07.350325 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:07.148065 2579 flags.go:64] FLAG: --eviction-hard="" Jul 09 22:10:07.350325 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:07.148070 2579 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Jul 09 22:10:07.350325 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:07.148073 2579 flags.go:64] FLAG: --eviction-minimum-reclaim="" Jul 09 22:10:07.350325 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:07.148077 2579 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Jul 09 22:10:07.350325 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:07.148081 2579 flags.go:64] FLAG: --eviction-soft="" Jul 09 22:10:07.350325 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:07.148084 2579 flags.go:64] FLAG: --eviction-soft-grace-period="" Jul 09 22:10:07.350325 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:07.148089 2579 flags.go:64] FLAG: --exit-on-lock-contention="false" Jul 09 22:10:07.350325 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:07.148093 2579 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Jul 09 22:10:07.350325 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:07.148109 2579 flags.go:64] FLAG: --experimental-mounter-path="" Jul 09 22:10:07.350325 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:07.148114 2579 flags.go:64] FLAG: --fail-cgroupv1="false" Jul 09 22:10:07.350325 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:07.148117 2579 flags.go:64] FLAG: --fail-swap-on="true" Jul 09 22:10:07.350325 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:07.148121 2579 flags.go:64] FLAG: --feature-gates="" Jul 09 22:10:07.350325 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:07.148132 2579 flags.go:64] FLAG: --file-check-frequency="20s" Jul 09 22:10:07.350325 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:07.148135 2579 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Jul 09 22:10:07.350325 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:07.148139 2579 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Jul 09 22:10:07.350968 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:07.148142 2579 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Jul 09 22:10:07.350968 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:07.148145 2579 flags.go:64] FLAG: --healthz-port="10248" Jul 09 22:10:07.350968 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:07.148149 2579 flags.go:64] FLAG: --help="false" Jul 09 22:10:07.350968 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:07.148152 2579 flags.go:64] FLAG: --hostname-override="ip-10-0-136-251.ec2.internal" Jul 09 22:10:07.350968 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:07.148155 2579 flags.go:64] FLAG: --housekeeping-interval="10s" Jul 09 22:10:07.350968 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:07.148158 2579 flags.go:64] FLAG: --http-check-frequency="20s" Jul 09 22:10:07.350968 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:07.148161 2579 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Jul 09 22:10:07.350968 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:07.148164 2579 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Jul 09 22:10:07.350968 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:07.148168 2579 flags.go:64] FLAG: --image-gc-high-threshold="85" Jul 09 22:10:07.350968 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:07.148171 2579 flags.go:64] FLAG: --image-gc-low-threshold="80" Jul 09 22:10:07.350968 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:07.148174 2579 flags.go:64] FLAG: --image-service-endpoint="" Jul 09 22:10:07.350968 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:07.148177 2579 flags.go:64] FLAG: --kernel-memcg-notification="false" Jul 09 22:10:07.350968 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:07.148180 2579 flags.go:64] FLAG: --kube-api-burst="100" Jul 09 22:10:07.350968 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:07.148182 2579 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Jul 09 22:10:07.350968 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:07.148186 2579 flags.go:64] FLAG: --kube-api-qps="50" Jul 09 22:10:07.350968 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:07.148188 2579 flags.go:64] FLAG: --kube-reserved="" Jul 09 22:10:07.350968 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:07.148191 2579 flags.go:64] FLAG: --kube-reserved-cgroup="" Jul 09 22:10:07.350968 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:07.148194 2579 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Jul 09 22:10:07.350968 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:07.148199 2579 flags.go:64] FLAG: --kubelet-cgroups="" Jul 09 22:10:07.350968 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:07.148202 2579 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Jul 09 22:10:07.350968 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:07.148205 2579 flags.go:64] FLAG: --lock-file="" Jul 09 22:10:07.350968 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:07.148209 2579 flags.go:64] FLAG: --log-cadvisor-usage="false" Jul 09 22:10:07.350968 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:07.148213 2579 flags.go:64] FLAG: --log-flush-frequency="5s" Jul 09 22:10:07.350968 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:07.148216 2579 flags.go:64] FLAG: --log-json-info-buffer-size="0" Jul 09 22:10:07.351580 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:07.148222 2579 flags.go:64] FLAG: --log-json-split-stream="false" Jul 09 22:10:07.351580 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:07.148225 2579 flags.go:64] FLAG: --log-text-info-buffer-size="0" Jul 09 22:10:07.351580 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:07.148228 2579 flags.go:64] FLAG: --log-text-split-stream="false" Jul 09 22:10:07.351580 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:07.148231 2579 flags.go:64] FLAG: --logging-format="text" Jul 09 22:10:07.351580 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:07.148234 2579 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Jul 09 22:10:07.351580 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:07.148238 2579 flags.go:64] FLAG: --make-iptables-util-chains="true" Jul 09 22:10:07.351580 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:07.148241 2579 flags.go:64] FLAG: --manifest-url="" Jul 09 22:10:07.351580 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:07.148243 2579 flags.go:64] FLAG: --manifest-url-header="" Jul 09 22:10:07.351580 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:07.148248 2579 flags.go:64] FLAG: --max-housekeeping-interval="15s" Jul 09 22:10:07.351580 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:07.148251 2579 flags.go:64] FLAG: --max-open-files="1000000" Jul 09 22:10:07.351580 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:07.148256 2579 flags.go:64] FLAG: --max-pods="110" Jul 09 22:10:07.351580 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:07.148259 2579 flags.go:64] FLAG: --maximum-dead-containers="-1" Jul 09 22:10:07.351580 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:07.148262 2579 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Jul 09 22:10:07.351580 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:07.148265 2579 flags.go:64] FLAG: --memory-manager-policy="None" Jul 09 22:10:07.351580 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:07.148268 2579 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Jul 09 22:10:07.351580 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:07.148271 2579 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Jul 09 22:10:07.351580 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:07.148274 2579 flags.go:64] FLAG: --node-ip="0.0.0.0" Jul 09 22:10:07.351580 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:07.148277 2579 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Jul 09 22:10:07.351580 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:07.148285 2579 flags.go:64] FLAG: --node-status-max-images="50" Jul 09 22:10:07.351580 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:07.148288 2579 flags.go:64] FLAG: --node-status-update-frequency="10s" Jul 09 22:10:07.351580 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:07.148292 2579 flags.go:64] FLAG: --oom-score-adj="-999" Jul 09 22:10:07.351580 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:07.148295 2579 flags.go:64] FLAG: --pod-cidr="" Jul 09 22:10:07.351580 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:07.148298 2579 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ce93f6b4197b9c62a26a3131f8057ffe9c9379c5079a4f62e5919b75151394e5" Jul 09 22:10:07.352127 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:07.148303 2579 flags.go:64] FLAG: --pod-manifest-path="" Jul 09 22:10:07.352127 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:07.148306 2579 flags.go:64] FLAG: --pod-max-pids="-1" Jul 09 22:10:07.352127 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:07.148309 2579 flags.go:64] FLAG: --pods-per-core="0" Jul 09 22:10:07.352127 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:07.148312 2579 flags.go:64] FLAG: --port="10250" Jul 09 22:10:07.352127 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:07.148317 2579 flags.go:64] FLAG: --protect-kernel-defaults="false" Jul 09 22:10:07.352127 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:07.148320 2579 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0d4678496b40dc74f" Jul 09 22:10:07.352127 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:07.148325 2579 flags.go:64] FLAG: --qos-reserved="" Jul 09 22:10:07.352127 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:07.148328 2579 flags.go:64] FLAG: --read-only-port="10255" Jul 09 22:10:07.352127 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:07.148331 2579 flags.go:64] FLAG: --register-node="true" Jul 09 22:10:07.352127 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:07.148334 2579 flags.go:64] FLAG: --register-with-taints="" Jul 09 22:10:07.352127 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:07.148345 2579 flags.go:64] FLAG: --registry-burst="10" Jul 09 22:10:07.352127 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:07.148348 2579 flags.go:64] FLAG: --registry-qps="5" Jul 09 22:10:07.352127 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:07.148351 2579 flags.go:64] FLAG: --reserved-cpus="" Jul 09 22:10:07.352127 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:07.148354 2579 flags.go:64] FLAG: --reserved-memory="" Jul 09 22:10:07.352127 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:07.148362 2579 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Jul 09 22:10:07.352127 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:07.148365 2579 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Jul 09 22:10:07.352127 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:07.148368 2579 flags.go:64] FLAG: --rotate-certificates="false" Jul 09 22:10:07.352127 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:07.148372 2579 flags.go:64] FLAG: --rotate-server-certificates="false" Jul 09 22:10:07.352127 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:07.148375 2579 flags.go:64] FLAG: --runonce="false" Jul 09 22:10:07.352127 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:07.148378 2579 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Jul 09 22:10:07.352127 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:07.148381 2579 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Jul 09 22:10:07.352127 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:07.148384 2579 flags.go:64] FLAG: --seccomp-default="false" Jul 09 22:10:07.352127 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:07.148387 2579 flags.go:64] FLAG: --serialize-image-pulls="true" Jul 09 22:10:07.352127 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:07.148389 2579 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Jul 09 22:10:07.352127 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:07.148392 2579 flags.go:64] FLAG: --storage-driver-db="cadvisor" Jul 09 22:10:07.352127 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:07.148395 2579 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Jul 09 22:10:07.352787 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:07.148399 2579 flags.go:64] FLAG: --storage-driver-password="root" Jul 09 22:10:07.352787 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:07.148402 2579 flags.go:64] FLAG: --storage-driver-secure="false" Jul 09 22:10:07.352787 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:07.148405 2579 flags.go:64] FLAG: --storage-driver-table="stats" Jul 09 22:10:07.352787 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:07.148408 2579 flags.go:64] FLAG: --storage-driver-user="root" Jul 09 22:10:07.352787 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:07.148411 2579 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Jul 09 22:10:07.352787 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:07.148414 2579 flags.go:64] FLAG: --sync-frequency="1m0s" Jul 09 22:10:07.352787 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:07.148417 2579 flags.go:64] FLAG: --system-cgroups="" Jul 09 22:10:07.352787 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:07.148420 2579 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Jul 09 22:10:07.352787 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:07.148425 2579 flags.go:64] FLAG: --system-reserved-cgroup="" Jul 09 22:10:07.352787 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:07.148443 2579 flags.go:64] FLAG: --tls-cert-file="" Jul 09 22:10:07.352787 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:07.148449 2579 flags.go:64] FLAG: --tls-cipher-suites="[]" Jul 09 22:10:07.352787 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:07.148453 2579 flags.go:64] FLAG: --tls-min-version="" Jul 09 22:10:07.352787 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:07.148457 2579 flags.go:64] FLAG: --tls-private-key-file="" Jul 09 22:10:07.352787 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:07.148461 2579 flags.go:64] FLAG: --topology-manager-policy="none" Jul 09 22:10:07.352787 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:07.148464 2579 flags.go:64] FLAG: --topology-manager-policy-options="" Jul 09 22:10:07.352787 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:07.148467 2579 flags.go:64] FLAG: --topology-manager-scope="container" Jul 09 22:10:07.352787 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:07.148470 2579 flags.go:64] FLAG: --v="2" Jul 09 22:10:07.352787 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:07.148475 2579 flags.go:64] FLAG: --version="false" Jul 09 22:10:07.352787 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:07.148479 2579 flags.go:64] FLAG: --vmodule="" Jul 09 22:10:07.352787 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:07.148483 2579 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Jul 09 22:10:07.352787 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:07.148487 2579 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Jul 09 22:10:07.352787 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:10:07.148609 2579 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 09 22:10:07.352787 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:10:07.148614 2579 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 09 22:10:07.352787 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:10:07.148617 2579 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 09 22:10:07.353508 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:10:07.148620 2579 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 09 22:10:07.353508 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:10:07.148623 2579 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 09 22:10:07.353508 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:10:07.148625 2579 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 09 22:10:07.353508 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:10:07.148628 2579 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 09 22:10:07.353508 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:10:07.148631 2579 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 09 22:10:07.353508 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:10:07.148633 2579 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 09 22:10:07.353508 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:10:07.148636 2579 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 09 22:10:07.353508 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:10:07.148638 2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 09 22:10:07.353508 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:10:07.148641 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 09 22:10:07.353508 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:10:07.148643 2579 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 09 22:10:07.353508 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:10:07.148646 2579 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 09 22:10:07.353508 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:10:07.148648 2579 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 09 22:10:07.353508 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:10:07.148651 2579 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 09 22:10:07.353508 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:10:07.148653 2579 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 09 22:10:07.353508 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:10:07.148656 2579 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 09 22:10:07.353508 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:10:07.148658 2579 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 09 22:10:07.353508 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:10:07.148661 2579 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 09 22:10:07.353508 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:10:07.148663 2579 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 09 22:10:07.353508 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:10:07.148667 2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 09 22:10:07.354004 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:10:07.148670 2579 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 09 22:10:07.354004 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:10:07.148675 2579 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 09 22:10:07.354004 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:10:07.148678 2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 09 22:10:07.354004 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:10:07.148680 2579 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 09 22:10:07.354004 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:10:07.148683 2579 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 09 22:10:07.354004 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:10:07.148685 2579 feature_gate.go:328] unrecognized feature gate: Example2 Jul 09 22:10:07.354004 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:10:07.148688 2579 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 09 22:10:07.354004 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:10:07.148691 2579 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 09 22:10:07.354004 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:10:07.148694 2579 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 09 22:10:07.354004 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:10:07.148697 2579 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 09 22:10:07.354004 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:10:07.148699 2579 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 09 22:10:07.354004 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:10:07.148702 2579 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 09 22:10:07.354004 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:10:07.148704 2579 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 09 22:10:07.354004 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:10:07.148707 2579 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 09 22:10:07.354004 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:10:07.148710 2579 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 09 22:10:07.354004 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:10:07.148712 2579 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 09 22:10:07.354004 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:10:07.148715 2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 09 22:10:07.354004 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:10:07.148717 2579 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 09 22:10:07.354004 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:10:07.148720 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 09 22:10:07.354004 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:10:07.148722 2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 09 22:10:07.354514 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:10:07.148725 2579 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 09 22:10:07.354514 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:10:07.148727 2579 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 09 22:10:07.354514 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:10:07.148730 2579 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 09 22:10:07.354514 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:10:07.148732 2579 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 09 22:10:07.354514 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:10:07.148735 2579 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 09 22:10:07.354514 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:10:07.148738 2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 09 22:10:07.354514 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:10:07.148740 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 09 22:10:07.354514 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:10:07.148742 2579 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 09 22:10:07.354514 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:10:07.148745 2579 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 09 22:10:07.354514 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:10:07.148747 2579 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 09 22:10:07.354514 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:10:07.148750 2579 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 09 22:10:07.354514 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:10:07.148753 2579 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 09 22:10:07.354514 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:10:07.148756 2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 09 22:10:07.354514 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:10:07.148760 2579 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 09 22:10:07.354514 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:10:07.148763 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 09 22:10:07.354514 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:10:07.148765 2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 09 22:10:07.354514 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:10:07.148768 2579 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 09 22:10:07.354514 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:10:07.148770 2579 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 09 22:10:07.354514 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:10:07.148773 2579 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 09 22:10:07.354514 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:10:07.148775 2579 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 09 22:10:07.355011 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:10:07.149135 2579 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 09 22:10:07.355011 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:10:07.149138 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 09 22:10:07.355011 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:10:07.149141 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 09 22:10:07.355011 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:10:07.149143 2579 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 09 22:10:07.355011 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:10:07.149146 2579 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 09 22:10:07.355011 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:10:07.149148 2579 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 09 22:10:07.355011 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:10:07.149151 2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 09 22:10:07.355011 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:10:07.149153 2579 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 09 22:10:07.355011 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:10:07.149156 2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 09 22:10:07.355011 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:10:07.149158 2579 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 09 22:10:07.355011 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:10:07.149161 2579 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 09 22:10:07.355011 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:10:07.149163 2579 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 09 22:10:07.355011 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:10:07.149166 2579 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 09 22:10:07.355011 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:10:07.149169 2579 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 09 22:10:07.355011 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:10:07.149171 2579 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 09 22:10:07.355011 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:10:07.149174 2579 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 09 22:10:07.355011 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:10:07.149176 2579 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 09 22:10:07.355011 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:10:07.149179 2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 09 22:10:07.355011 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:10:07.149181 2579 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 09 22:10:07.355011 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:10:07.149184 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 09 22:10:07.355618 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:10:07.149186 2579 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 09 22:10:07.355618 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:10:07.149189 2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 09 22:10:07.355618 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:10:07.149192 2579 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 09 22:10:07.355618 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:10:07.149195 2579 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 09 22:10:07.355618 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:10:07.149198 2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 09 22:10:07.355618 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:10:07.149203 2579 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 09 22:10:07.355618 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:10:07.149205 2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 09 22:10:07.355618 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:10:07.149208 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 09 22:10:07.355618 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:10:07.149211 2579 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 09 22:10:07.355618 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:10:07.149214 2579 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 09 22:10:07.355618 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:10:07.149217 2579 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 09 22:10:07.355618 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:10:07.149219 2579 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 09 22:10:07.355618 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:10:07.149222 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 09 22:10:07.355618 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:10:07.149225 2579 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 09 22:10:07.355618 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:10:07.149227 2579 feature_gate.go:328] unrecognized feature gate: Example Jul 09 22:10:07.355618 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:10:07.149231 2579 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 09 22:10:07.355618 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:10:07.149236 2579 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 09 22:10:07.356033 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:07.149242 2579 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 09 22:10:07.356033 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:07.157551 2579 server.go:529] "Kubelet version" kubeletVersion="v1.34.8" Jul 09 22:10:07.356033 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:07.157569 2579 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Jul 09 22:10:07.356033 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:07.158272 2579 watchdog_linux.go:95] "Systemd watchdog is not enabled" Jul 09 22:10:07.356033 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:07.158280 2579 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Jul 09 22:10:07.356033 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:10:07.158343 2579 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 09 22:10:07.356033 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:10:07.158348 2579 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 09 22:10:07.356033 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:10:07.158352 2579 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 09 22:10:07.356033 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:10:07.158355 2579 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 09 22:10:07.356033 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:10:07.158358 2579 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 09 22:10:07.356033 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:10:07.158361 2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 09 22:10:07.356033 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:10:07.158364 2579 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 09 22:10:07.356033 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:10:07.158368 2579 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 09 22:10:07.356033 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:10:07.158370 2579 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 09 22:10:07.356388 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:10:07.158374 2579 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 09 22:10:07.356388 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:10:07.158376 2579 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 09 22:10:07.356388 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:10:07.158379 2579 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 09 22:10:07.356388 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:10:07.158382 2579 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 09 22:10:07.356388 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:10:07.158385 2579 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 09 22:10:07.356388 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:10:07.158387 2579 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 09 22:10:07.356388 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:10:07.158390 2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 09 22:10:07.356388 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:10:07.158393 2579 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 09 22:10:07.356388 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:10:07.158396 2579 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 09 22:10:07.356388 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:10:07.158398 2579 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 09 22:10:07.356388 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:10:07.158401 2579 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 09 22:10:07.356388 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:10:07.158404 2579 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 09 22:10:07.356388 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:10:07.158406 2579 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 09 22:10:07.356388 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:10:07.158409 2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 09 22:10:07.356388 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:10:07.158411 2579 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 09 22:10:07.356388 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:10:07.158414 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 09 22:10:07.356388 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:10:07.158416 2579 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 09 22:10:07.356388 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:10:07.158420 2579 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 09 22:10:07.356388 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:10:07.158423 2579 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 09 22:10:07.356388 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:10:07.158425 2579 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 09 22:10:07.356888 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:10:07.158428 2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 09 22:10:07.356888 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:10:07.158445 2579 feature_gate.go:328] unrecognized feature gate: Example2 Jul 09 22:10:07.356888 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:10:07.158448 2579 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 09 22:10:07.356888 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:10:07.158451 2579 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 09 22:10:07.356888 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:10:07.158454 2579 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 09 22:10:07.356888 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:10:07.158457 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 09 22:10:07.356888 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:10:07.158460 2579 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 09 22:10:07.356888 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:10:07.158463 2579 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 09 22:10:07.356888 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:10:07.158465 2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 09 22:10:07.356888 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:10:07.158468 2579 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 09 22:10:07.356888 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:10:07.158470 2579 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 09 22:10:07.356888 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:10:07.158473 2579 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 09 22:10:07.356888 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:10:07.158476 2579 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 09 22:10:07.356888 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:10:07.158479 2579 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 09 22:10:07.356888 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:10:07.158481 2579 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 09 22:10:07.356888 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:10:07.158483 2579 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 09 22:10:07.356888 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:10:07.158486 2579 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 09 22:10:07.356888 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:10:07.158489 2579 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 09 22:10:07.356888 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:10:07.158491 2579 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 09 22:10:07.356888 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:10:07.158496 2579 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 09 22:10:07.357377 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:10:07.158500 2579 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 09 22:10:07.357377 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:10:07.158503 2579 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 09 22:10:07.357377 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:10:07.158505 2579 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 09 22:10:07.357377 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:10:07.158508 2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 09 22:10:07.357377 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:10:07.158511 2579 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 09 22:10:07.357377 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:10:07.158513 2579 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 09 22:10:07.357377 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:10:07.158516 2579 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 09 22:10:07.357377 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:10:07.158518 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 09 22:10:07.357377 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:10:07.158521 2579 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 09 22:10:07.357377 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:10:07.158527 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 09 22:10:07.357377 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:10:07.158530 2579 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 09 22:10:07.357377 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:10:07.158533 2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 09 22:10:07.357377 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:10:07.158535 2579 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 09 22:10:07.357377 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:10:07.158538 2579 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 09 22:10:07.357377 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:10:07.158541 2579 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 09 22:10:07.357377 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:10:07.158544 2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 09 22:10:07.357377 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:10:07.158546 2579 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 09 22:10:07.357377 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:10:07.158550 2579 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 09 22:10:07.357377 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:10:07.158552 2579 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 09 22:10:07.357377 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:10:07.158555 2579 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 09 22:10:07.357874 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:10:07.158558 2579 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 09 22:10:07.357874 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:10:07.158561 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 09 22:10:07.357874 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:10:07.158563 2579 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 09 22:10:07.357874 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:10:07.158566 2579 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 09 22:10:07.357874 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:10:07.158570 2579 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 09 22:10:07.357874 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:10:07.158574 2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 09 22:10:07.357874 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:10:07.158577 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 09 22:10:07.357874 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:10:07.158580 2579 feature_gate.go:328] unrecognized feature gate: Example Jul 09 22:10:07.357874 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:10:07.158590 2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 09 22:10:07.357874 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:10:07.158593 2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 09 22:10:07.357874 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:10:07.158597 2579 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 09 22:10:07.357874 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:10:07.158600 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 09 22:10:07.357874 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:10:07.158603 2579 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 09 22:10:07.357874 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:10:07.158606 2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 09 22:10:07.357874 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:10:07.158608 2579 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 09 22:10:07.357874 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:10:07.158611 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 09 22:10:07.357874 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:10:07.158614 2579 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 09 22:10:07.357874 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:10:07.158616 2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 09 22:10:07.357874 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:10:07.158619 2579 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 09 22:10:07.357874 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:10:07.158621 2579 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 09 22:10:07.358359 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:10:07.158624 2579 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 09 22:10:07.358359 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:10:07.158628 2579 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 09 22:10:07.358359 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:10:07.158631 2579 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 09 22:10:07.358359 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:10:07.158634 2579 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 09 22:10:07.358359 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:10:07.158636 2579 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 09 22:10:07.358359 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:10:07.158639 2579 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 09 22:10:07.358359 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:10:07.158642 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 09 22:10:07.358359 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:10:07.158645 2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 09 22:10:07.358359 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:10:07.158648 2579 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 09 22:10:07.358359 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:10:07.158651 2579 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 09 22:10:07.358359 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:07.158657 2579 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 09 22:10:07.358359 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:10:07.158805 2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 09 22:10:07.358359 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:10:07.158811 2579 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 09 22:10:07.358359 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:10:07.158814 2579 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 09 22:10:07.358733 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:10:07.158817 2579 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 09 22:10:07.358733 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:10:07.158820 2579 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 09 22:10:07.358733 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:10:07.158822 2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 09 22:10:07.358733 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:10:07.158825 2579 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 09 22:10:07.358733 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:10:07.158828 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 09 22:10:07.358733 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:10:07.158830 2579 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 09 22:10:07.358733 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:10:07.158833 2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 09 22:10:07.358733 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:10:07.158835 2579 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 09 22:10:07.358733 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:10:07.158838 2579 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 09 22:10:07.358733 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:10:07.158841 2579 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 09 22:10:07.358733 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:10:07.158844 2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 09 22:10:07.358733 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:10:07.158846 2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 09 22:10:07.358733 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:10:07.158849 2579 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 09 22:10:07.358733 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:10:07.158851 2579 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 09 22:10:07.358733 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:10:07.158854 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 09 22:10:07.358733 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:10:07.158857 2579 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 09 22:10:07.358733 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:10:07.158859 2579 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 09 22:10:07.358733 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:10:07.158862 2579 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 09 22:10:07.358733 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:10:07.158864 2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 09 22:10:07.358733 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:10:07.158868 2579 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 09 22:10:07.359226 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:10:07.158871 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 09 22:10:07.359226 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:10:07.158874 2579 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 09 22:10:07.359226 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:10:07.158877 2579 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 09 22:10:07.359226 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:10:07.158879 2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 09 22:10:07.359226 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:10:07.158882 2579 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 09 22:10:07.359226 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:10:07.158884 2579 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 09 22:10:07.359226 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:10:07.158887 2579 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 09 22:10:07.359226 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:10:07.158890 2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 09 22:10:07.359226 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:10:07.158892 2579 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 09 22:10:07.359226 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:10:07.158895 2579 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 09 22:10:07.359226 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:10:07.158898 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 09 22:10:07.359226 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:10:07.158900 2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 09 22:10:07.359226 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:10:07.158903 2579 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 09 22:10:07.359226 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:10:07.158906 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 09 22:10:07.359226 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:10:07.158908 2579 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 09 22:10:07.359226 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:10:07.158913 2579 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 09 22:10:07.359226 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:10:07.158916 2579 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 09 22:10:07.359226 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:10:07.158920 2579 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 09 22:10:07.359226 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:10:07.158922 2579 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 09 22:10:07.359786 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:10:07.158925 2579 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 09 22:10:07.359786 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:10:07.158928 2579 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 09 22:10:07.359786 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:10:07.158931 2579 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 09 22:10:07.359786 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:10:07.158935 2579 feature_gate.go:328] unrecognized feature gate: Example2 Jul 09 22:10:07.359786 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:10:07.158937 2579 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 09 22:10:07.359786 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:10:07.158940 2579 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 09 22:10:07.359786 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:10:07.158942 2579 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 09 22:10:07.359786 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:10:07.158945 2579 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 09 22:10:07.359786 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:10:07.158947 2579 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 09 22:10:07.359786 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:10:07.158950 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 09 22:10:07.359786 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:10:07.158952 2579 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 09 22:10:07.359786 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:10:07.158955 2579 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 09 22:10:07.359786 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:10:07.158957 2579 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 09 22:10:07.359786 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:10:07.158961 2579 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 09 22:10:07.359786 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:10:07.158964 2579 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 09 22:10:07.359786 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:10:07.158966 2579 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 09 22:10:07.359786 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:10:07.158969 2579 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 09 22:10:07.359786 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:10:07.158972 2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 09 22:10:07.359786 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:10:07.158975 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 09 22:10:07.359786 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:10:07.158978 2579 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 09 22:10:07.360282 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:10:07.158980 2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 09 22:10:07.360282 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:10:07.158983 2579 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 09 22:10:07.360282 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:10:07.158986 2579 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 09 22:10:07.360282 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:10:07.158989 2579 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 09 22:10:07.360282 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:10:07.158991 2579 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 09 22:10:07.360282 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:10:07.158994 2579 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 09 22:10:07.360282 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:10:07.158996 2579 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 09 22:10:07.360282 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:10:07.158999 2579 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 09 22:10:07.360282 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:10:07.159001 2579 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 09 22:10:07.360282 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:10:07.159004 2579 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 09 22:10:07.360282 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:10:07.159006 2579 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 09 22:10:07.360282 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:10:07.159009 2579 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 09 22:10:07.360282 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:10:07.159011 2579 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 09 22:10:07.360282 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:10:07.159014 2579 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 09 22:10:07.360282 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:10:07.159017 2579 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 09 22:10:07.360282 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:10:07.159019 2579 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 09 22:10:07.360282 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:10:07.159023 2579 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 09 22:10:07.360282 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:10:07.159025 2579 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 09 22:10:07.360282 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:10:07.159028 2579 feature_gate.go:328] unrecognized feature gate: Example Jul 09 22:10:07.360282 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:10:07.159030 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 09 22:10:07.360778 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:10:07.159033 2579 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 09 22:10:07.360778 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:10:07.159035 2579 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 09 22:10:07.360778 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:10:07.159037 2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 09 22:10:07.360778 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:10:07.159040 2579 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 09 22:10:07.360778 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:10:07.159044 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 09 22:10:07.360778 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:10:07.159048 2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 09 22:10:07.360778 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:10:07.159051 2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 09 22:10:07.360778 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:10:07.159053 2579 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 09 22:10:07.360778 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:10:07.159056 2579 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 09 22:10:07.360778 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:10:07.159059 2579 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 09 22:10:07.360778 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:10:07.159063 2579 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 09 22:10:07.360778 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:10:07.159066 2579 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 09 22:10:07.360778 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:10:07.159068 2579 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 09 22:10:07.360778 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:10:07.159071 2579 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 09 22:10:07.360778 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:10:07.159074 2579 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 09 22:10:07.360778 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:10:07.159078 2579 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 09 22:10:07.360778 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:10:07.159081 2579 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 09 22:10:07.361187 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:07.159086 2579 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 09 22:10:07.361187 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:07.159218 2579 server.go:962] "Client rotation is on, will bootstrap in background" Jul 09 22:10:07.361187 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:07.161522 2579 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Jul 09 22:10:07.361187 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:07.162842 2579 server.go:1019] "Starting client certificate rotation" Jul 09 22:10:07.361187 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:07.162944 2579 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 09 22:10:07.361187 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:07.162976 2579 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 09 22:10:07.361187 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:07.165101 2579 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 09 22:10:07.361187 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:07.166309 2579 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 09 22:10:07.361187 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:07.176336 2579 log.go:25] "Validated CRI v1 runtime API" Jul 09 22:10:07.361187 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:07.180004 2579 log.go:25] "Validated CRI v1 image API" Jul 09 22:10:07.361187 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:07.182166 2579 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Jul 09 22:10:07.361187 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:07.184833 2579 fs.go:135] Filesystem UUIDs: map[607d667f-7c62-4936-8398-b63378d6d6d8:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 cc12ec0e-d88f-4ad2-a98c-4f450403afd7:/dev/nvme0n1p3] Jul 09 22:10:07.361554 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:07.184852 2579 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Jul 09 22:10:07.361554 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:07.191131 2579 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 09 22:10:07.361618 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:07.192457 2579 manager.go:216] Machine: {Timestamp:2026-07-09 22:10:07.190998621 +0000 UTC m=+0.325936950 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3101074 MemoryCapacity:33164496896 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec20d8921c7ac8fdd85be2dda2d061fe SystemUUID:ec20d892-1c7a-c8fd-d85b-e2dda2d061fe BootID:c05a7e41-3917-4648-9854-b1f6aa21196f Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582246400 Type:vfs Inodes:4048400 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632902656 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6107136 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582250496 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:cc:93:5b:46:2b Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:cc:93:5b:46:2b Speed:0 Mtu:9001} {Name:ovs-system MacAddress:4e:7b:d7:37:0d:41 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164496896 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Jul 09 22:10:07.361618 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:07.192556 2579 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Jul 09 22:10:07.361618 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:07.192650 2579 manager.go:232] Version: {KernelVersion:5.14.0-570.125.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260630-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Jul 09 22:10:07.361618 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:07.195371 2579 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Jul 09 22:10:07.361618 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:07.195397 2579 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-136-251.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Jul 09 22:10:07.361618 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:07.196110 2579 topology_manager.go:138] "Creating topology manager with none policy" Jul 09 22:10:07.361618 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:07.196121 2579 container_manager_linux.go:309] "Creating device plugin manager" Jul 09 22:10:07.361618 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:07.196128 2579 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 09 22:10:07.361618 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:07.196699 2579 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 09 22:10:07.361618 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:07.197908 2579 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Jul 09 22:10:07.361618 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:07.200616 2579 state_mem.go:36] "Initialized new in-memory state store" Jul 09 22:10:07.361618 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:07.200742 2579 server.go:1240] "Using root directory" path="/var/lib/kubelet" Jul 09 22:10:07.361618 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:07.200770 2579 kubelet.go:486] "Attempting to sync node with API server" Jul 09 22:10:07.361618 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:07.200780 2579 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Jul 09 22:10:07.361618 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:07.200795 2579 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Jul 09 22:10:07.361618 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:07.200805 2579 kubelet.go:398] "Adding apiserver pod source" Jul 09 22:10:07.361618 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:07.200814 2579 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Jul 09 22:10:07.361618 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:07.201204 2579 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 09 22:10:07.362096 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:07.201228 2579 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 09 22:10:07.362096 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:07.203100 2579 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.8-3.rhaos4.21.git91b4b0b.el9" apiVersion="v1" Jul 09 22:10:07.362096 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:07.205308 2579 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Jul 09 22:10:07.362096 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:07.205336 2579 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Jul 09 22:10:07.362096 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:07.206509 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Jul 09 22:10:07.362096 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:07.206523 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Jul 09 22:10:07.362096 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:07.206529 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Jul 09 22:10:07.362096 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:07.206535 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Jul 09 22:10:07.362096 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:07.206544 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Jul 09 22:10:07.362096 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:07.206550 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Jul 09 22:10:07.362096 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:07.206558 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Jul 09 22:10:07.362096 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:07.206563 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Jul 09 22:10:07.362096 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:07.206570 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Jul 09 22:10:07.362096 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:07.206576 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Jul 09 22:10:07.362096 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:07.206593 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Jul 09 22:10:07.362096 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:07.206600 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Jul 09 22:10:07.362096 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:07.207948 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Jul 09 22:10:07.362096 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:07.207964 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Jul 09 22:10:07.362096 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:07.211147 2579 server.go:1268] "Started kubelet" Jul 09 22:10:07.362096 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:07.211252 2579 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Jul 09 22:10:07.362096 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:07.211381 2579 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Jul 09 22:10:07.362096 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:07.211343 2579 server_v1.go:49] "podresources" method="list" useActivePods=true Jul 09 22:10:07.362096 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:07.212379 2579 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-136-251.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Jul 09 22:10:07.362096 ip-10-0-136-251 kubenswrapper[2579]: E0709 22:10:07.212401 2579 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Jul 09 22:10:07.362096 ip-10-0-136-251 kubenswrapper[2579]: E0709 22:10:07.212495 2579 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-136-251.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Jul 09 22:10:07.362096 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:07.213137 2579 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Jul 09 22:10:07.362096 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:07.213319 2579 server.go:310] "Adding debug handlers to kubelet server" Jul 09 22:10:07.362096 ip-10-0-136-251 kubenswrapper[2579]: E0709 22:10:07.216590 2579 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Jul 09 22:10:07.362096 ip-10-0-136-251 kubenswrapper[2579]: E0709 22:10:07.217045 2579 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-136-251.ec2.internal.18c0be96c5c523af default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-136-251.ec2.internal,UID:ip-10-0-136-251.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-136-251.ec2.internal,},FirstTimestamp:2026-07-09 22:10:07.211119535 +0000 UTC m=+0.346057844,LastTimestamp:2026-07-09 22:10:07.211119535 +0000 UTC m=+0.346057844,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-136-251.ec2.internal,}" Jul 09 22:10:07.362096 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:07.221216 2579 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Jul 09 22:10:07.362096 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:07.222078 2579 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Jul 09 22:10:07.362096 ip-10-0-136-251 kubenswrapper[2579]: E0709 22:10:07.222691 2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-251.ec2.internal\" not found" Jul 09 22:10:07.362096 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:07.222769 2579 volume_manager.go:311] "The desired_state_of_world populator starts" Jul 09 22:10:07.362096 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:07.222781 2579 volume_manager.go:313] "Starting Kubelet Volume Manager" Jul 09 22:10:07.362096 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:07.222761 2579 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Jul 09 22:10:07.362096 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:07.222876 2579 reconstruct.go:97] "Volume reconstruction finished" Jul 09 22:10:07.362096 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:07.222884 2579 reconciler.go:29] "Reconciler: start to sync state" Jul 09 22:10:07.362096 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:07.223016 2579 factory.go:55] Registering systemd factory Jul 09 22:10:07.362096 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:07.223027 2579 factory.go:223] Registration of the systemd container factory successfully Jul 09 22:10:07.362096 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:07.223222 2579 factory.go:153] Registering CRI-O factory Jul 09 22:10:07.362096 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:07.223236 2579 factory.go:223] Registration of the crio container factory successfully Jul 09 22:10:07.363155 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:07.223289 2579 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Jul 09 22:10:07.363155 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:07.223315 2579 factory.go:103] Registering Raw factory Jul 09 22:10:07.363155 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:07.223331 2579 manager.go:1196] Started watching for new ooms in manager Jul 09 22:10:07.363155 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:07.224476 2579 manager.go:349] Starting recovery of all containers Jul 09 22:10:07.363155 ip-10-0-136-251 kubenswrapper[2579]: E0709 22:10:07.235265 2579 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-136-251.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Jul 09 22:10:07.363155 ip-10-0-136-251 kubenswrapper[2579]: E0709 22:10:07.235479 2579 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" Jul 09 22:10:07.363155 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:07.236789 2579 manager.go:354] Recovery completed Jul 09 22:10:07.363155 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:07.240894 2579 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 09 22:10:07.363155 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:07.241170 2579 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-bwb4j" Jul 09 22:10:07.363155 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:07.242956 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-251.ec2.internal" event="NodeHasSufficientMemory" Jul 09 22:10:07.363155 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:07.242985 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-251.ec2.internal" event="NodeHasNoDiskPressure" Jul 09 22:10:07.363155 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:07.243007 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-251.ec2.internal" event="NodeHasSufficientPID" Jul 09 22:10:07.363155 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:07.243690 2579 cpu_manager.go:222] "Starting CPU manager" policy="none" Jul 09 22:10:07.363155 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:07.243701 2579 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Jul 09 22:10:07.363155 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:07.243704 2579 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 09 22:10:07.363155 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:07.243717 2579 state_mem.go:36] "Initialized new in-memory state store" Jul 09 22:10:07.363155 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:07.244427 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-251.ec2.internal" event="NodeHasSufficientMemory" Jul 09 22:10:07.363155 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:07.244466 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-251.ec2.internal" event="NodeHasNoDiskPressure" Jul 09 22:10:07.363155 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:07.244477 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-251.ec2.internal" event="NodeHasSufficientPID" Jul 09 22:10:07.363155 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:07.244863 2579 policy_none.go:49] "None policy: Start" Jul 09 22:10:07.363155 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:07.244877 2579 memory_manager.go:187] "Starting memorymanager" policy="None" Jul 09 22:10:07.363155 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:07.244886 2579 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Jul 09 22:10:07.363155 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:07.245709 2579 policy_none.go:47] "Start" Jul 09 22:10:07.363155 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:07.246550 2579 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-bwb4j" Jul 09 22:10:07.363155 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:07.286055 2579 manager.go:337] "Starting Device Plugin manager" Jul 09 22:10:07.363155 ip-10-0-136-251 kubenswrapper[2579]: E0709 22:10:07.286101 2579 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Jul 09 22:10:07.363155 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:07.286113 2579 server.go:87] "Starting device plugin registration server" Jul 09 22:10:07.363155 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:07.286393 2579 eviction_manager.go:189] "Eviction manager: starting control loop" Jul 09 22:10:07.363155 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:07.286405 2579 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Jul 09 22:10:07.363155 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:07.286505 2579 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Jul 09 22:10:07.363155 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:07.286613 2579 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Jul 09 22:10:07.363155 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:07.286623 2579 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Jul 09 22:10:07.363155 ip-10-0-136-251 kubenswrapper[2579]: E0709 22:10:07.287119 2579 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Jul 09 22:10:07.363155 ip-10-0-136-251 kubenswrapper[2579]: E0709 22:10:07.287176 2579 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-136-251.ec2.internal\" not found" Jul 09 22:10:07.363155 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:07.333024 2579 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Jul 09 22:10:07.363155 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:07.334343 2579 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Jul 09 22:10:07.363155 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:07.334385 2579 status_manager.go:244] "Starting to sync pod status with apiserver" Jul 09 22:10:07.363155 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:07.334412 2579 kubelet.go:2448] "Starting kubelet main sync loop" Jul 09 22:10:07.363155 ip-10-0-136-251 kubenswrapper[2579]: E0709 22:10:07.334471 2579 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Jul 09 22:10:07.363155 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:07.336597 2579 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 09 22:10:07.387388 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:07.387356 2579 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 09 22:10:07.388338 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:07.388322 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-251.ec2.internal" event="NodeHasSufficientMemory" Jul 09 22:10:07.388397 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:07.388352 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-251.ec2.internal" event="NodeHasNoDiskPressure" Jul 09 22:10:07.388397 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:07.388361 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-251.ec2.internal" event="NodeHasSufficientPID" Jul 09 22:10:07.388397 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:07.388385 2579 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-136-251.ec2.internal" Jul 09 22:10:07.397242 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:07.397224 2579 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-136-251.ec2.internal" Jul 09 22:10:07.397289 ip-10-0-136-251 kubenswrapper[2579]: E0709 22:10:07.397249 2579 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-136-251.ec2.internal\": node \"ip-10-0-136-251.ec2.internal\" not found" Jul 09 22:10:07.419713 ip-10-0-136-251 kubenswrapper[2579]: E0709 22:10:07.419640 2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-251.ec2.internal\" not found" Jul 09 22:10:07.435036 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:07.434994 2579 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-251.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-136-251.ec2.internal"] Jul 09 22:10:07.435099 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:07.435078 2579 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 09 22:10:07.436113 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:07.436097 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-251.ec2.internal" event="NodeHasSufficientMemory" Jul 09 22:10:07.436165 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:07.436128 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-251.ec2.internal" event="NodeHasNoDiskPressure" Jul 09 22:10:07.436165 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:07.436141 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-251.ec2.internal" event="NodeHasSufficientPID" Jul 09 22:10:07.437286 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:07.437272 2579 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 09 22:10:07.437400 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:07.437385 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-251.ec2.internal" Jul 09 22:10:07.437472 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:07.437422 2579 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 09 22:10:07.438209 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:07.438194 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-251.ec2.internal" event="NodeHasSufficientMemory" Jul 09 22:10:07.438281 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:07.438220 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-251.ec2.internal" event="NodeHasNoDiskPressure" Jul 09 22:10:07.438281 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:07.438231 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-251.ec2.internal" event="NodeHasSufficientPID" Jul 09 22:10:07.438281 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:07.438238 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-251.ec2.internal" event="NodeHasSufficientMemory" Jul 09 22:10:07.438281 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:07.438260 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-251.ec2.internal" event="NodeHasNoDiskPressure" Jul 09 22:10:07.438281 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:07.438272 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-251.ec2.internal" event="NodeHasSufficientPID" Jul 09 22:10:07.439295 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:07.439282 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-251.ec2.internal" Jul 09 22:10:07.439338 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:07.439308 2579 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 09 22:10:07.439932 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:07.439916 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-251.ec2.internal" event="NodeHasSufficientMemory" Jul 09 22:10:07.440007 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:07.439942 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-251.ec2.internal" event="NodeHasNoDiskPressure" Jul 09 22:10:07.440007 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:07.439958 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-251.ec2.internal" event="NodeHasSufficientPID" Jul 09 22:10:07.466692 ip-10-0-136-251 kubenswrapper[2579]: E0709 22:10:07.466667 2579 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-136-251.ec2.internal\" not found" node="ip-10-0-136-251.ec2.internal" Jul 09 22:10:07.469873 ip-10-0-136-251 kubenswrapper[2579]: E0709 22:10:07.469853 2579 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-136-251.ec2.internal\" not found" node="ip-10-0-136-251.ec2.internal" Jul 09 22:10:07.520550 ip-10-0-136-251 kubenswrapper[2579]: E0709 22:10:07.520517 2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-251.ec2.internal\" not found" Jul 09 22:10:07.523929 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:07.523905 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/021762e3fafa981d5236e3af644982e1-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-136-251.ec2.internal\" (UID: \"021762e3fafa981d5236e3af644982e1\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-251.ec2.internal" Jul 09 22:10:07.523986 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:07.523938 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/021762e3fafa981d5236e3af644982e1-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-136-251.ec2.internal\" (UID: \"021762e3fafa981d5236e3af644982e1\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-251.ec2.internal" Jul 09 22:10:07.523986 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:07.523960 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/50613fbded229913a0a9abdd811c3ca4-config\") pod \"kube-apiserver-proxy-ip-10-0-136-251.ec2.internal\" (UID: \"50613fbded229913a0a9abdd811c3ca4\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-136-251.ec2.internal" Jul 09 22:10:07.621067 ip-10-0-136-251 kubenswrapper[2579]: E0709 22:10:07.621027 2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-251.ec2.internal\" not found" Jul 09 22:10:07.624318 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:07.624301 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/021762e3fafa981d5236e3af644982e1-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-136-251.ec2.internal\" (UID: \"021762e3fafa981d5236e3af644982e1\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-251.ec2.internal" Jul 09 22:10:07.624391 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:07.624335 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/021762e3fafa981d5236e3af644982e1-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-136-251.ec2.internal\" (UID: \"021762e3fafa981d5236e3af644982e1\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-251.ec2.internal" Jul 09 22:10:07.624391 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:07.624359 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/50613fbded229913a0a9abdd811c3ca4-config\") pod \"kube-apiserver-proxy-ip-10-0-136-251.ec2.internal\" (UID: \"50613fbded229913a0a9abdd811c3ca4\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-136-251.ec2.internal" Jul 09 22:10:07.624482 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:07.624398 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/021762e3fafa981d5236e3af644982e1-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-136-251.ec2.internal\" (UID: \"021762e3fafa981d5236e3af644982e1\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-251.ec2.internal" Jul 09 22:10:07.624482 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:07.624395 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/50613fbded229913a0a9abdd811c3ca4-config\") pod \"kube-apiserver-proxy-ip-10-0-136-251.ec2.internal\" (UID: \"50613fbded229913a0a9abdd811c3ca4\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-136-251.ec2.internal" Jul 09 22:10:07.624482 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:07.624407 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/021762e3fafa981d5236e3af644982e1-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-136-251.ec2.internal\" (UID: \"021762e3fafa981d5236e3af644982e1\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-251.ec2.internal" Jul 09 22:10:07.721855 ip-10-0-136-251 kubenswrapper[2579]: E0709 22:10:07.721762 2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-251.ec2.internal\" not found" Jul 09 22:10:07.770190 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:07.770156 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-251.ec2.internal" Jul 09 22:10:07.774794 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:07.774775 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-251.ec2.internal" Jul 09 22:10:07.822091 ip-10-0-136-251 kubenswrapper[2579]: E0709 22:10:07.822055 2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-251.ec2.internal\" not found" Jul 09 22:10:09.146038 ip-10-0-136-251 kubenswrapper[2579]: E0709 22:10:09.145998 2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-251.ec2.internal\" not found" Jul 09 22:10:09.246554 ip-10-0-136-251 kubenswrapper[2579]: E0709 22:10:09.246460 2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-251.ec2.internal\" not found" Jul 09 22:10:09.346895 ip-10-0-136-251 kubenswrapper[2579]: E0709 22:10:09.346850 2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-251.ec2.internal\" not found" Jul 09 22:10:09.386178 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:09.386147 2579 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Jul 09 22:10:09.386658 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:09.386345 2579 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 09 22:10:09.447548 ip-10-0-136-251 kubenswrapper[2579]: E0709 22:10:09.447519 2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-251.ec2.internal\" not found" Jul 09 22:10:09.451932 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:09.451895 2579 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Jul 09 22:10:09.471786 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:09.471750 2579 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-08 22:05:08 +0000 UTC" deadline="2028-03-04 06:15:03.173712931 +0000 UTC" Jul 09 22:10:09.471786 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:09.471784 2579 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14480h4m53.701933059s" Jul 09 22:10:09.546076 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:09.545997 2579 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-251.ec2.internal" Jul 09 22:10:09.558501 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:09.558376 2579 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 09 22:10:09.561193 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:09.559995 2579 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-251.ec2.internal" Jul 09 22:10:09.569030 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:09.569011 2579 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 09 22:10:09.616821 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:09.616793 2579 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 09 22:10:09.634444 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:09.634405 2579 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Jul 09 22:10:09.669462 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:10:09.669429 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod50613fbded229913a0a9abdd811c3ca4.slice/crio-2ff48d6b1519924377314fe9f9cbb72de57557739dcf382c05adf25fd3a9d462 WatchSource:0}: Error finding container 2ff48d6b1519924377314fe9f9cbb72de57557739dcf382c05adf25fd3a9d462: Status 404 returned error can't find the container with id 2ff48d6b1519924377314fe9f9cbb72de57557739dcf382c05adf25fd3a9d462 Jul 09 22:10:09.670515 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:10:09.670482 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod021762e3fafa981d5236e3af644982e1.slice/crio-565286e5c968065605f345c5f00f2a6e8412e4219068930949417c8e6ba698e7 WatchSource:0}: Error finding container 565286e5c968065605f345c5f00f2a6e8412e4219068930949417c8e6ba698e7: Status 404 returned error can't find the container with id 565286e5c968065605f345c5f00f2a6e8412e4219068930949417c8e6ba698e7 Jul 09 22:10:09.676669 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:09.676653 2579 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 09 22:10:10.426383 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:10.426351 2579 apiserver.go:52] "Watching apiserver" Jul 09 22:10:10.436952 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:10.436913 2579 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Jul 09 22:10:10.438061 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:10.438033 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-251.ec2.internal","openshift-multus/multus-additional-cni-plugins-5fzf2","openshift-network-diagnostics/network-check-target-9b9zs","openshift-network-operator/iptables-alerter-6t57q","kube-system/global-pull-secret-syncer-bwdgc","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-669lr","openshift-image-registry/node-ca-l578c","openshift-multus/multus-lvcvn","openshift-multus/network-metrics-daemon-8pj95","openshift-ovn-kubernetes/ovnkube-node-r8dsp","kube-system/konnectivity-agent-gcmz7","kube-system/kube-apiserver-proxy-ip-10-0-136-251.ec2.internal","openshift-cluster-node-tuning-operator/tuned-bzbzs","openshift-dns/node-resolver-nr4cq"] Jul 09 22:10:10.439465 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:10.439441 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-l578c" Jul 09 22:10:10.440911 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:10.440672 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-bzbzs" Jul 09 22:10:10.441798 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:10.441780 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9b9zs" Jul 09 22:10:10.441881 ip-10-0-136-251 kubenswrapper[2579]: E0709 22:10:10.441860 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9b9zs" podUID="541ce094-3638-4979-bf7b-782f3df1cbf8" Jul 09 22:10:10.442359 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:10.442333 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Jul 09 22:10:10.442508 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:10.442342 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Jul 09 22:10:10.442614 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:10.442554 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Jul 09 22:10:10.442676 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:10.442495 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-6rjwr\"" Jul 09 22:10:10.442895 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:10.442876 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-6t57q" Jul 09 22:10:10.443511 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:10.443490 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-b5fbb\"" Jul 09 22:10:10.443511 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:10.443510 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Jul 09 22:10:10.443651 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:10.443537 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Jul 09 22:10:10.444230 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:10.444210 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bwdgc" Jul 09 22:10:10.444326 ip-10-0-136-251 kubenswrapper[2579]: E0709 22:10:10.444274 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bwdgc" podUID="b5e569ee-1aea-4eeb-8235-93438e648102" Jul 09 22:10:10.445392 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:10.445309 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Jul 09 22:10:10.445392 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:10.445381 2579 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Jul 09 22:10:10.446019 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:10.445741 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Jul 09 22:10:10.446167 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:10.446126 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Jul 09 22:10:10.446235 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:10.446177 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-hwhtg\"" Jul 09 22:10:10.446379 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:10.446363 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-669lr" Jul 09 22:10:10.447592 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:10.447573 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8pj95" Jul 09 22:10:10.447659 ip-10-0-136-251 kubenswrapper[2579]: E0709 22:10:10.447639 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8pj95" podUID="23b023ec-a254-462d-a0b4-5c22fd905e72" Jul 09 22:10:10.448446 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:10.448333 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Jul 09 22:10:10.449162 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:10.449136 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-nr4cq" Jul 09 22:10:10.449344 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:10.449274 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-vx9jf\"" Jul 09 22:10:10.449532 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:10.449516 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Jul 09 22:10:10.449600 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:10.449561 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Jul 09 22:10:10.450591 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:10.450304 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-5fzf2" Jul 09 22:10:10.451232 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:10.451212 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-prqwd\"" Jul 09 22:10:10.451490 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:10.451473 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Jul 09 22:10:10.452076 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:10.451748 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Jul 09 22:10:10.452076 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:10.452011 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-r8dsp" Jul 09 22:10:10.452697 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:10.452678 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Jul 09 22:10:10.452881 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:10.452862 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Jul 09 22:10:10.452994 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:10.452906 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Jul 09 22:10:10.453056 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:10.452683 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-s55gs\"" Jul 09 22:10:10.453056 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:10.453029 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Jul 09 22:10:10.453152 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:10.452862 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Jul 09 22:10:10.453982 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:10.453809 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-gcmz7" Jul 09 22:10:10.454780 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:10.454642 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Jul 09 22:10:10.454987 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:10.454964 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Jul 09 22:10:10.455149 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:10.455127 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b5e569ee-1aea-4eeb-8235-93438e648102-original-pull-secret\") pod \"global-pull-secret-syncer-bwdgc\" (UID: \"b5e569ee-1aea-4eeb-8235-93438e648102\") " pod="kube-system/global-pull-secret-syncer-bwdgc" Jul 09 22:10:10.455214 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:10.455175 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/af0be020-5b8b-4828-bf98-531265c795b2-serviceca\") pod \"node-ca-l578c\" (UID: \"af0be020-5b8b-4828-bf98-531265c795b2\") " pod="openshift-image-registry/node-ca-l578c" Jul 09 22:10:10.455265 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:10.455210 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/50069e81-757a-482b-b7a2-ac26f2036ebd-etc-modprobe-d\") pod \"tuned-bzbzs\" (UID: \"50069e81-757a-482b-b7a2-ac26f2036ebd\") " pod="openshift-cluster-node-tuning-operator/tuned-bzbzs" Jul 09 22:10:10.455265 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:10.455238 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/50069e81-757a-482b-b7a2-ac26f2036ebd-sys\") pod \"tuned-bzbzs\" (UID: \"50069e81-757a-482b-b7a2-ac26f2036ebd\") " pod="openshift-cluster-node-tuning-operator/tuned-bzbzs" Jul 09 22:10:10.455669 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:10.455270 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/50069e81-757a-482b-b7a2-ac26f2036ebd-lib-modules\") pod \"tuned-bzbzs\" (UID: \"50069e81-757a-482b-b7a2-ac26f2036ebd\") " pod="openshift-cluster-node-tuning-operator/tuned-bzbzs" Jul 09 22:10:10.455669 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:10.455397 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/13602a06-27c2-446f-a5aa-64c917c285c8-kubelet-dir\") pod \"aws-ebs-csi-driver-node-669lr\" (UID: \"13602a06-27c2-446f-a5aa-64c917c285c8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-669lr" Jul 09 22:10:10.455669 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:10.455434 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/13602a06-27c2-446f-a5aa-64c917c285c8-etc-selinux\") pod \"aws-ebs-csi-driver-node-669lr\" (UID: \"13602a06-27c2-446f-a5aa-64c917c285c8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-669lr" Jul 09 22:10:10.455669 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:10.455585 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/b5e569ee-1aea-4eeb-8235-93438e648102-kubelet-config\") pod \"global-pull-secret-syncer-bwdgc\" (UID: \"b5e569ee-1aea-4eeb-8235-93438e648102\") " pod="kube-system/global-pull-secret-syncer-bwdgc" Jul 09 22:10:10.455669 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:10.455612 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/50069e81-757a-482b-b7a2-ac26f2036ebd-etc-systemd\") pod \"tuned-bzbzs\" (UID: \"50069e81-757a-482b-b7a2-ac26f2036ebd\") " pod="openshift-cluster-node-tuning-operator/tuned-bzbzs" Jul 09 22:10:10.455894 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:10.455704 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/50069e81-757a-482b-b7a2-ac26f2036ebd-etc-tuned\") pod \"tuned-bzbzs\" (UID: \"50069e81-757a-482b-b7a2-ac26f2036ebd\") " pod="openshift-cluster-node-tuning-operator/tuned-bzbzs" Jul 09 22:10:10.455894 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:10.455738 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/15cda8f6-b7fd-41f2-8aea-2ef4e9b9669b-iptables-alerter-script\") pod \"iptables-alerter-6t57q\" (UID: \"15cda8f6-b7fd-41f2-8aea-2ef4e9b9669b\") " pod="openshift-network-operator/iptables-alerter-6t57q" Jul 09 22:10:10.455894 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:10.455766 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/b5e569ee-1aea-4eeb-8235-93438e648102-dbus\") pod \"global-pull-secret-syncer-bwdgc\" (UID: \"b5e569ee-1aea-4eeb-8235-93438e648102\") " pod="kube-system/global-pull-secret-syncer-bwdgc" Jul 09 22:10:10.455894 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:10.455796 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/50069e81-757a-482b-b7a2-ac26f2036ebd-etc-sysctl-d\") pod \"tuned-bzbzs\" (UID: \"50069e81-757a-482b-b7a2-ac26f2036ebd\") " pod="openshift-cluster-node-tuning-operator/tuned-bzbzs" Jul 09 22:10:10.455894 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:10.455827 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/50069e81-757a-482b-b7a2-ac26f2036ebd-run\") pod \"tuned-bzbzs\" (UID: \"50069e81-757a-482b-b7a2-ac26f2036ebd\") " pod="openshift-cluster-node-tuning-operator/tuned-bzbzs" Jul 09 22:10:10.455894 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:10.455882 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ghmmf\" (UniqueName: \"kubernetes.io/projected/15cda8f6-b7fd-41f2-8aea-2ef4e9b9669b-kube-api-access-ghmmf\") pod \"iptables-alerter-6t57q\" (UID: \"15cda8f6-b7fd-41f2-8aea-2ef4e9b9669b\") " pod="openshift-network-operator/iptables-alerter-6t57q" Jul 09 22:10:10.456155 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:10.455962 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/13602a06-27c2-446f-a5aa-64c917c285c8-registration-dir\") pod \"aws-ebs-csi-driver-node-669lr\" (UID: \"13602a06-27c2-446f-a5aa-64c917c285c8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-669lr" Jul 09 22:10:10.456155 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:10.456013 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/23b023ec-a254-462d-a0b4-5c22fd905e72-metrics-certs\") pod \"network-metrics-daemon-8pj95\" (UID: \"23b023ec-a254-462d-a0b4-5c22fd905e72\") " pod="openshift-multus/network-metrics-daemon-8pj95" Jul 09 22:10:10.456155 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:10.456046 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-25lz6\" (UniqueName: \"kubernetes.io/projected/af0be020-5b8b-4828-bf98-531265c795b2-kube-api-access-25lz6\") pod \"node-ca-l578c\" (UID: \"af0be020-5b8b-4828-bf98-531265c795b2\") " pod="openshift-image-registry/node-ca-l578c" Jul 09 22:10:10.456155 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:10.456109 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/50069e81-757a-482b-b7a2-ac26f2036ebd-etc-kubernetes\") pod \"tuned-bzbzs\" (UID: \"50069e81-757a-482b-b7a2-ac26f2036ebd\") " pod="openshift-cluster-node-tuning-operator/tuned-bzbzs" Jul 09 22:10:10.456155 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:10.456138 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/50069e81-757a-482b-b7a2-ac26f2036ebd-var-lib-kubelet\") pod \"tuned-bzbzs\" (UID: \"50069e81-757a-482b-b7a2-ac26f2036ebd\") " pod="openshift-cluster-node-tuning-operator/tuned-bzbzs" Jul 09 22:10:10.456944 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:10.456530 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-dr7vx\"" Jul 09 22:10:10.456944 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:10.456619 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/13602a06-27c2-446f-a5aa-64c917c285c8-device-dir\") pod \"aws-ebs-csi-driver-node-669lr\" (UID: \"13602a06-27c2-446f-a5aa-64c917c285c8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-669lr" Jul 09 22:10:10.456944 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:10.456659 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-lvcvn" Jul 09 22:10:10.456944 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:10.456678 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dw6k7\" (UniqueName: \"kubernetes.io/projected/23b023ec-a254-462d-a0b4-5c22fd905e72-kube-api-access-dw6k7\") pod \"network-metrics-daemon-8pj95\" (UID: \"23b023ec-a254-462d-a0b4-5c22fd905e72\") " pod="openshift-multus/network-metrics-daemon-8pj95" Jul 09 22:10:10.456944 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:10.456749 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/af0be020-5b8b-4828-bf98-531265c795b2-host\") pod \"node-ca-l578c\" (UID: \"af0be020-5b8b-4828-bf98-531265c795b2\") " pod="openshift-image-registry/node-ca-l578c" Jul 09 22:10:10.456944 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:10.456781 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/50069e81-757a-482b-b7a2-ac26f2036ebd-etc-sysconfig\") pod \"tuned-bzbzs\" (UID: \"50069e81-757a-482b-b7a2-ac26f2036ebd\") " pod="openshift-cluster-node-tuning-operator/tuned-bzbzs" Jul 09 22:10:10.456944 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:10.456821 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/50069e81-757a-482b-b7a2-ac26f2036ebd-tmp\") pod \"tuned-bzbzs\" (UID: \"50069e81-757a-482b-b7a2-ac26f2036ebd\") " pod="openshift-cluster-node-tuning-operator/tuned-bzbzs" Jul 09 22:10:10.456944 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:10.456874 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/15cda8f6-b7fd-41f2-8aea-2ef4e9b9669b-host-slash\") pod \"iptables-alerter-6t57q\" (UID: \"15cda8f6-b7fd-41f2-8aea-2ef4e9b9669b\") " pod="openshift-network-operator/iptables-alerter-6t57q" Jul 09 22:10:10.456944 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:10.456902 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/50069e81-757a-482b-b7a2-ac26f2036ebd-etc-sysctl-conf\") pod \"tuned-bzbzs\" (UID: \"50069e81-757a-482b-b7a2-ac26f2036ebd\") " pod="openshift-cluster-node-tuning-operator/tuned-bzbzs" Jul 09 22:10:10.457401 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:10.456990 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/50069e81-757a-482b-b7a2-ac26f2036ebd-host\") pod \"tuned-bzbzs\" (UID: \"50069e81-757a-482b-b7a2-ac26f2036ebd\") " pod="openshift-cluster-node-tuning-operator/tuned-bzbzs" Jul 09 22:10:10.457401 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:10.457015 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tpflr\" (UniqueName: \"kubernetes.io/projected/50069e81-757a-482b-b7a2-ac26f2036ebd-kube-api-access-tpflr\") pod \"tuned-bzbzs\" (UID: \"50069e81-757a-482b-b7a2-ac26f2036ebd\") " pod="openshift-cluster-node-tuning-operator/tuned-bzbzs" Jul 09 22:10:10.457401 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:10.457043 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/13602a06-27c2-446f-a5aa-64c917c285c8-socket-dir\") pod \"aws-ebs-csi-driver-node-669lr\" (UID: \"13602a06-27c2-446f-a5aa-64c917c285c8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-669lr" Jul 09 22:10:10.457401 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:10.457071 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/13602a06-27c2-446f-a5aa-64c917c285c8-sys-fs\") pod \"aws-ebs-csi-driver-node-669lr\" (UID: \"13602a06-27c2-446f-a5aa-64c917c285c8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-669lr" Jul 09 22:10:10.457401 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:10.457105 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h57g8\" (UniqueName: \"kubernetes.io/projected/13602a06-27c2-446f-a5aa-64c917c285c8-kube-api-access-h57g8\") pod \"aws-ebs-csi-driver-node-669lr\" (UID: \"13602a06-27c2-446f-a5aa-64c917c285c8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-669lr" Jul 09 22:10:10.457401 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:10.457166 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fx7mr\" (UniqueName: \"kubernetes.io/projected/541ce094-3638-4979-bf7b-782f3df1cbf8-kube-api-access-fx7mr\") pod \"network-check-target-9b9zs\" (UID: \"541ce094-3638-4979-bf7b-782f3df1cbf8\") " pod="openshift-network-diagnostics/network-check-target-9b9zs" Jul 09 22:10:10.458730 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:10.458563 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Jul 09 22:10:10.458730 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:10.458653 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Jul 09 22:10:10.458730 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:10.458679 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Jul 09 22:10:10.458934 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:10.458808 2579 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 09 22:10:10.459326 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:10.459245 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Jul 09 22:10:10.459511 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:10.459289 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Jul 09 22:10:10.460377 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:10.459815 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-vfwm7\"" Jul 09 22:10:10.460377 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:10.460052 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Jul 09 22:10:10.460377 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:10.460082 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Jul 09 22:10:10.460803 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:10.460636 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-8l6qk\"" Jul 09 22:10:10.477626 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:10.477575 2579 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-zz5vm" Jul 09 22:10:10.485373 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:10.485345 2579 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-zz5vm" Jul 09 22:10:10.547268 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:10.547205 2579 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Jul 09 22:10:10.558087 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:10.557912 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/50069e81-757a-482b-b7a2-ac26f2036ebd-var-lib-kubelet\") pod \"tuned-bzbzs\" (UID: \"50069e81-757a-482b-b7a2-ac26f2036ebd\") " pod="openshift-cluster-node-tuning-operator/tuned-bzbzs" Jul 09 22:10:10.558087 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:10.557971 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/7d1ac8fd-5e7b-439c-b48c-44dcf09b2934-run-systemd\") pod \"ovnkube-node-r8dsp\" (UID: \"7d1ac8fd-5e7b-439c-b48c-44dcf09b2934\") " pod="openshift-ovn-kubernetes/ovnkube-node-r8dsp" Jul 09 22:10:10.558087 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:10.558000 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/7d1ac8fd-5e7b-439c-b48c-44dcf09b2934-host-cni-netd\") pod \"ovnkube-node-r8dsp\" (UID: \"7d1ac8fd-5e7b-439c-b48c-44dcf09b2934\") " pod="openshift-ovn-kubernetes/ovnkube-node-r8dsp" Jul 09 22:10:10.558087 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:10.558008 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/50069e81-757a-482b-b7a2-ac26f2036ebd-var-lib-kubelet\") pod \"tuned-bzbzs\" (UID: \"50069e81-757a-482b-b7a2-ac26f2036ebd\") " pod="openshift-cluster-node-tuning-operator/tuned-bzbzs" Jul 09 22:10:10.558087 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:10.558025 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/84b44025-bd7d-4910-ba37-d294bf354258-host-var-lib-kubelet\") pod \"multus-lvcvn\" (UID: \"84b44025-bd7d-4910-ba37-d294bf354258\") " pod="openshift-multus/multus-lvcvn" Jul 09 22:10:10.558087 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:10.558048 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7d1ac8fd-5e7b-439c-b48c-44dcf09b2934-etc-openvswitch\") pod \"ovnkube-node-r8dsp\" (UID: \"7d1ac8fd-5e7b-439c-b48c-44dcf09b2934\") " pod="openshift-ovn-kubernetes/ovnkube-node-r8dsp" Jul 09 22:10:10.558487 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:10.558101 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/7d1ac8fd-5e7b-439c-b48c-44dcf09b2934-ovnkube-config\") pod \"ovnkube-node-r8dsp\" (UID: \"7d1ac8fd-5e7b-439c-b48c-44dcf09b2934\") " pod="openshift-ovn-kubernetes/ovnkube-node-r8dsp" Jul 09 22:10:10.558487 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:10.558142 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/af0be020-5b8b-4828-bf98-531265c795b2-host\") pod \"node-ca-l578c\" (UID: \"af0be020-5b8b-4828-bf98-531265c795b2\") " pod="openshift-image-registry/node-ca-l578c" Jul 09 22:10:10.558487 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:10.558174 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/15cda8f6-b7fd-41f2-8aea-2ef4e9b9669b-host-slash\") pod \"iptables-alerter-6t57q\" (UID: \"15cda8f6-b7fd-41f2-8aea-2ef4e9b9669b\") " pod="openshift-network-operator/iptables-alerter-6t57q" Jul 09 22:10:10.558487 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:10.558201 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/84b44025-bd7d-4910-ba37-d294bf354258-multus-conf-dir\") pod \"multus-lvcvn\" (UID: \"84b44025-bd7d-4910-ba37-d294bf354258\") " pod="openshift-multus/multus-lvcvn" Jul 09 22:10:10.558487 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:10.558236 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/499e80b4-0095-43da-ad29-f143e59a7357-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-5fzf2\" (UID: \"499e80b4-0095-43da-ad29-f143e59a7357\") " pod="openshift-multus/multus-additional-cni-plugins-5fzf2" Jul 09 22:10:10.558487 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:10.558263 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/50069e81-757a-482b-b7a2-ac26f2036ebd-etc-sysctl-conf\") pod \"tuned-bzbzs\" (UID: \"50069e81-757a-482b-b7a2-ac26f2036ebd\") " pod="openshift-cluster-node-tuning-operator/tuned-bzbzs" Jul 09 22:10:10.558487 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:10.558285 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/50069e81-757a-482b-b7a2-ac26f2036ebd-host\") pod \"tuned-bzbzs\" (UID: \"50069e81-757a-482b-b7a2-ac26f2036ebd\") " pod="openshift-cluster-node-tuning-operator/tuned-bzbzs" Jul 09 22:10:10.558487 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:10.558309 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tpflr\" (UniqueName: \"kubernetes.io/projected/50069e81-757a-482b-b7a2-ac26f2036ebd-kube-api-access-tpflr\") pod \"tuned-bzbzs\" (UID: \"50069e81-757a-482b-b7a2-ac26f2036ebd\") " pod="openshift-cluster-node-tuning-operator/tuned-bzbzs" Jul 09 22:10:10.558487 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:10.558399 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/50069e81-757a-482b-b7a2-ac26f2036ebd-host\") pod \"tuned-bzbzs\" (UID: \"50069e81-757a-482b-b7a2-ac26f2036ebd\") " pod="openshift-cluster-node-tuning-operator/tuned-bzbzs" Jul 09 22:10:10.558487 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:10.558427 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/13602a06-27c2-446f-a5aa-64c917c285c8-socket-dir\") pod \"aws-ebs-csi-driver-node-669lr\" (UID: \"13602a06-27c2-446f-a5aa-64c917c285c8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-669lr" Jul 09 22:10:10.558487 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:10.558456 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/84b44025-bd7d-4910-ba37-d294bf354258-etc-kubernetes\") pod \"multus-lvcvn\" (UID: \"84b44025-bd7d-4910-ba37-d294bf354258\") " pod="openshift-multus/multus-lvcvn" Jul 09 22:10:10.558487 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:10.558481 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/13602a06-27c2-446f-a5aa-64c917c285c8-sys-fs\") pod \"aws-ebs-csi-driver-node-669lr\" (UID: \"13602a06-27c2-446f-a5aa-64c917c285c8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-669lr" Jul 09 22:10:10.558990 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:10.558518 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/af0be020-5b8b-4828-bf98-531265c795b2-serviceca\") pod \"node-ca-l578c\" (UID: \"af0be020-5b8b-4828-bf98-531265c795b2\") " pod="openshift-image-registry/node-ca-l578c" Jul 09 22:10:10.558990 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:10.558551 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/50069e81-757a-482b-b7a2-ac26f2036ebd-etc-modprobe-d\") pod \"tuned-bzbzs\" (UID: \"50069e81-757a-482b-b7a2-ac26f2036ebd\") " pod="openshift-cluster-node-tuning-operator/tuned-bzbzs" Jul 09 22:10:10.558990 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:10.558581 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/13602a06-27c2-446f-a5aa-64c917c285c8-etc-selinux\") pod \"aws-ebs-csi-driver-node-669lr\" (UID: \"13602a06-27c2-446f-a5aa-64c917c285c8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-669lr" Jul 09 22:10:10.558990 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:10.558606 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/84b44025-bd7d-4910-ba37-d294bf354258-multus-daemon-config\") pod \"multus-lvcvn\" (UID: \"84b44025-bd7d-4910-ba37-d294bf354258\") " pod="openshift-multus/multus-lvcvn" Jul 09 22:10:10.558990 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:10.558692 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/13602a06-27c2-446f-a5aa-64c917c285c8-sys-fs\") pod \"aws-ebs-csi-driver-node-669lr\" (UID: \"13602a06-27c2-446f-a5aa-64c917c285c8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-669lr" Jul 09 22:10:10.558990 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:10.558750 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/7d1ac8fd-5e7b-439c-b48c-44dcf09b2934-host-run-netns\") pod \"ovnkube-node-r8dsp\" (UID: \"7d1ac8fd-5e7b-439c-b48c-44dcf09b2934\") " pod="openshift-ovn-kubernetes/ovnkube-node-r8dsp" Jul 09 22:10:10.558990 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:10.558783 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7d1ac8fd-5e7b-439c-b48c-44dcf09b2934-host-run-ovn-kubernetes\") pod \"ovnkube-node-r8dsp\" (UID: \"7d1ac8fd-5e7b-439c-b48c-44dcf09b2934\") " pod="openshift-ovn-kubernetes/ovnkube-node-r8dsp" Jul 09 22:10:10.558990 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:10.558810 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z6827\" (UniqueName: \"kubernetes.io/projected/499e80b4-0095-43da-ad29-f143e59a7357-kube-api-access-z6827\") pod \"multus-additional-cni-plugins-5fzf2\" (UID: \"499e80b4-0095-43da-ad29-f143e59a7357\") " pod="openshift-multus/multus-additional-cni-plugins-5fzf2" Jul 09 22:10:10.558990 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:10.558844 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/b5e569ee-1aea-4eeb-8235-93438e648102-kubelet-config\") pod \"global-pull-secret-syncer-bwdgc\" (UID: \"b5e569ee-1aea-4eeb-8235-93438e648102\") " pod="kube-system/global-pull-secret-syncer-bwdgc" Jul 09 22:10:10.558990 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:10.558870 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/50069e81-757a-482b-b7a2-ac26f2036ebd-etc-tuned\") pod \"tuned-bzbzs\" (UID: \"50069e81-757a-482b-b7a2-ac26f2036ebd\") " pod="openshift-cluster-node-tuning-operator/tuned-bzbzs" Jul 09 22:10:10.558990 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:10.558896 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/15cda8f6-b7fd-41f2-8aea-2ef4e9b9669b-iptables-alerter-script\") pod \"iptables-alerter-6t57q\" (UID: \"15cda8f6-b7fd-41f2-8aea-2ef4e9b9669b\") " pod="openshift-network-operator/iptables-alerter-6t57q" Jul 09 22:10:10.558990 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:10.558934 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/84b44025-bd7d-4910-ba37-d294bf354258-os-release\") pod \"multus-lvcvn\" (UID: \"84b44025-bd7d-4910-ba37-d294bf354258\") " pod="openshift-multus/multus-lvcvn" Jul 09 22:10:10.558990 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:10.558960 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/7d1ac8fd-5e7b-439c-b48c-44dcf09b2934-host-kubelet\") pod \"ovnkube-node-r8dsp\" (UID: \"7d1ac8fd-5e7b-439c-b48c-44dcf09b2934\") " pod="openshift-ovn-kubernetes/ovnkube-node-r8dsp" Jul 09 22:10:10.558990 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:10.558985 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/7d1ac8fd-5e7b-439c-b48c-44dcf09b2934-env-overrides\") pod \"ovnkube-node-r8dsp\" (UID: \"7d1ac8fd-5e7b-439c-b48c-44dcf09b2934\") " pod="openshift-ovn-kubernetes/ovnkube-node-r8dsp" Jul 09 22:10:10.558990 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:10.558754 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/50069e81-757a-482b-b7a2-ac26f2036ebd-etc-sysctl-conf\") pod \"tuned-bzbzs\" (UID: \"50069e81-757a-482b-b7a2-ac26f2036ebd\") " pod="openshift-cluster-node-tuning-operator/tuned-bzbzs" Jul 09 22:10:10.559758 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:10.559011 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/13602a06-27c2-446f-a5aa-64c917c285c8-socket-dir\") pod \"aws-ebs-csi-driver-node-669lr\" (UID: \"13602a06-27c2-446f-a5aa-64c917c285c8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-669lr" Jul 09 22:10:10.559758 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:10.559084 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/7d1ac8fd-5e7b-439c-b48c-44dcf09b2934-ovnkube-script-lib\") pod \"ovnkube-node-r8dsp\" (UID: \"7d1ac8fd-5e7b-439c-b48c-44dcf09b2934\") " pod="openshift-ovn-kubernetes/ovnkube-node-r8dsp" Jul 09 22:10:10.559758 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:10.559113 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/499e80b4-0095-43da-ad29-f143e59a7357-cni-binary-copy\") pod \"multus-additional-cni-plugins-5fzf2\" (UID: \"499e80b4-0095-43da-ad29-f143e59a7357\") " pod="openshift-multus/multus-additional-cni-plugins-5fzf2" Jul 09 22:10:10.559758 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:10.559142 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/50069e81-757a-482b-b7a2-ac26f2036ebd-run\") pod \"tuned-bzbzs\" (UID: \"50069e81-757a-482b-b7a2-ac26f2036ebd\") " pod="openshift-cluster-node-tuning-operator/tuned-bzbzs" Jul 09 22:10:10.559758 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:10.559170 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ghmmf\" (UniqueName: \"kubernetes.io/projected/15cda8f6-b7fd-41f2-8aea-2ef4e9b9669b-kube-api-access-ghmmf\") pod \"iptables-alerter-6t57q\" (UID: \"15cda8f6-b7fd-41f2-8aea-2ef4e9b9669b\") " pod="openshift-network-operator/iptables-alerter-6t57q" Jul 09 22:10:10.559758 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:10.559172 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/af0be020-5b8b-4828-bf98-531265c795b2-serviceca\") pod \"node-ca-l578c\" (UID: \"af0be020-5b8b-4828-bf98-531265c795b2\") " pod="openshift-image-registry/node-ca-l578c" Jul 09 22:10:10.559758 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:10.559195 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/84b44025-bd7d-4910-ba37-d294bf354258-system-cni-dir\") pod \"multus-lvcvn\" (UID: \"84b44025-bd7d-4910-ba37-d294bf354258\") " pod="openshift-multus/multus-lvcvn" Jul 09 22:10:10.559758 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:10.559222 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/84b44025-bd7d-4910-ba37-d294bf354258-multus-cni-dir\") pod \"multus-lvcvn\" (UID: \"84b44025-bd7d-4910-ba37-d294bf354258\") " pod="openshift-multus/multus-lvcvn" Jul 09 22:10:10.559758 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:10.559247 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/6807e2e6-d9a4-475a-9fb4-4b958127840f-hosts-file\") pod \"node-resolver-nr4cq\" (UID: \"6807e2e6-d9a4-475a-9fb4-4b958127840f\") " pod="openshift-dns/node-resolver-nr4cq" Jul 09 22:10:10.559758 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:10.559273 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/7d1ac8fd-5e7b-439c-b48c-44dcf09b2934-log-socket\") pod \"ovnkube-node-r8dsp\" (UID: \"7d1ac8fd-5e7b-439c-b48c-44dcf09b2934\") " pod="openshift-ovn-kubernetes/ovnkube-node-r8dsp" Jul 09 22:10:10.559758 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:10.559300 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/499e80b4-0095-43da-ad29-f143e59a7357-system-cni-dir\") pod \"multus-additional-cni-plugins-5fzf2\" (UID: \"499e80b4-0095-43da-ad29-f143e59a7357\") " pod="openshift-multus/multus-additional-cni-plugins-5fzf2" Jul 09 22:10:10.559758 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:10.559311 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/af0be020-5b8b-4828-bf98-531265c795b2-host\") pod \"node-ca-l578c\" (UID: \"af0be020-5b8b-4828-bf98-531265c795b2\") " pod="openshift-image-registry/node-ca-l578c" Jul 09 22:10:10.559758 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:10.559327 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/50069e81-757a-482b-b7a2-ac26f2036ebd-etc-kubernetes\") pod \"tuned-bzbzs\" (UID: \"50069e81-757a-482b-b7a2-ac26f2036ebd\") " pod="openshift-cluster-node-tuning-operator/tuned-bzbzs" Jul 09 22:10:10.559758 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:10.559382 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/15cda8f6-b7fd-41f2-8aea-2ef4e9b9669b-host-slash\") pod \"iptables-alerter-6t57q\" (UID: \"15cda8f6-b7fd-41f2-8aea-2ef4e9b9669b\") " pod="openshift-network-operator/iptables-alerter-6t57q" Jul 09 22:10:10.559758 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:10.559480 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/50069e81-757a-482b-b7a2-ac26f2036ebd-etc-modprobe-d\") pod \"tuned-bzbzs\" (UID: \"50069e81-757a-482b-b7a2-ac26f2036ebd\") " pod="openshift-cluster-node-tuning-operator/tuned-bzbzs" Jul 09 22:10:10.559758 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:10.559513 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/b5e569ee-1aea-4eeb-8235-93438e648102-kubelet-config\") pod \"global-pull-secret-syncer-bwdgc\" (UID: \"b5e569ee-1aea-4eeb-8235-93438e648102\") " pod="kube-system/global-pull-secret-syncer-bwdgc" Jul 09 22:10:10.559758 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:10.559585 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/13602a06-27c2-446f-a5aa-64c917c285c8-etc-selinux\") pod \"aws-ebs-csi-driver-node-669lr\" (UID: \"13602a06-27c2-446f-a5aa-64c917c285c8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-669lr" Jul 09 22:10:10.560558 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:10.559607 2579 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Jul 09 22:10:10.560558 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:10.559687 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/50069e81-757a-482b-b7a2-ac26f2036ebd-run\") pod \"tuned-bzbzs\" (UID: \"50069e81-757a-482b-b7a2-ac26f2036ebd\") " pod="openshift-cluster-node-tuning-operator/tuned-bzbzs" Jul 09 22:10:10.560558 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:10.559731 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/84b44025-bd7d-4910-ba37-d294bf354258-cnibin\") pod \"multus-lvcvn\" (UID: \"84b44025-bd7d-4910-ba37-d294bf354258\") " pod="openshift-multus/multus-lvcvn" Jul 09 22:10:10.560558 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:10.559756 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/7d1ac8fd-5e7b-439c-b48c-44dcf09b2934-host-cni-bin\") pod \"ovnkube-node-r8dsp\" (UID: \"7d1ac8fd-5e7b-439c-b48c-44dcf09b2934\") " pod="openshift-ovn-kubernetes/ovnkube-node-r8dsp" Jul 09 22:10:10.560558 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:10.559786 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/13602a06-27c2-446f-a5aa-64c917c285c8-device-dir\") pod \"aws-ebs-csi-driver-node-669lr\" (UID: \"13602a06-27c2-446f-a5aa-64c917c285c8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-669lr" Jul 09 22:10:10.560558 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:10.559843 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dw6k7\" (UniqueName: \"kubernetes.io/projected/23b023ec-a254-462d-a0b4-5c22fd905e72-kube-api-access-dw6k7\") pod \"network-metrics-daemon-8pj95\" (UID: \"23b023ec-a254-462d-a0b4-5c22fd905e72\") " pod="openshift-multus/network-metrics-daemon-8pj95" Jul 09 22:10:10.560558 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:10.559869 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/84b44025-bd7d-4910-ba37-d294bf354258-multus-socket-dir-parent\") pod \"multus-lvcvn\" (UID: \"84b44025-bd7d-4910-ba37-d294bf354258\") " pod="openshift-multus/multus-lvcvn" Jul 09 22:10:10.560558 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:10.559897 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/84b44025-bd7d-4910-ba37-d294bf354258-host-var-lib-cni-bin\") pod \"multus-lvcvn\" (UID: \"84b44025-bd7d-4910-ba37-d294bf354258\") " pod="openshift-multus/multus-lvcvn" Jul 09 22:10:10.560558 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:10.559901 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/15cda8f6-b7fd-41f2-8aea-2ef4e9b9669b-iptables-alerter-script\") pod \"iptables-alerter-6t57q\" (UID: \"15cda8f6-b7fd-41f2-8aea-2ef4e9b9669b\") " pod="openshift-network-operator/iptables-alerter-6t57q" Jul 09 22:10:10.560558 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:10.559938 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/84b44025-bd7d-4910-ba37-d294bf354258-hostroot\") pod \"multus-lvcvn\" (UID: \"84b44025-bd7d-4910-ba37-d294bf354258\") " pod="openshift-multus/multus-lvcvn" Jul 09 22:10:10.560558 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:10.559981 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/50069e81-757a-482b-b7a2-ac26f2036ebd-etc-kubernetes\") pod \"tuned-bzbzs\" (UID: \"50069e81-757a-482b-b7a2-ac26f2036ebd\") " pod="openshift-cluster-node-tuning-operator/tuned-bzbzs" Jul 09 22:10:10.560558 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:10.559982 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cf2ms\" (UniqueName: \"kubernetes.io/projected/6807e2e6-d9a4-475a-9fb4-4b958127840f-kube-api-access-cf2ms\") pod \"node-resolver-nr4cq\" (UID: \"6807e2e6-d9a4-475a-9fb4-4b958127840f\") " pod="openshift-dns/node-resolver-nr4cq" Jul 09 22:10:10.560558 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:10.560037 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/7d1ac8fd-5e7b-439c-b48c-44dcf09b2934-systemd-units\") pod \"ovnkube-node-r8dsp\" (UID: \"7d1ac8fd-5e7b-439c-b48c-44dcf09b2934\") " pod="openshift-ovn-kubernetes/ovnkube-node-r8dsp" Jul 09 22:10:10.560558 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:10.560061 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/7d1ac8fd-5e7b-439c-b48c-44dcf09b2934-run-ovn\") pod \"ovnkube-node-r8dsp\" (UID: \"7d1ac8fd-5e7b-439c-b48c-44dcf09b2934\") " pod="openshift-ovn-kubernetes/ovnkube-node-r8dsp" Jul 09 22:10:10.560558 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:10.560088 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/50069e81-757a-482b-b7a2-ac26f2036ebd-etc-sysconfig\") pod \"tuned-bzbzs\" (UID: \"50069e81-757a-482b-b7a2-ac26f2036ebd\") " pod="openshift-cluster-node-tuning-operator/tuned-bzbzs" Jul 09 22:10:10.560558 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:10.560125 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/50069e81-757a-482b-b7a2-ac26f2036ebd-tmp\") pod \"tuned-bzbzs\" (UID: \"50069e81-757a-482b-b7a2-ac26f2036ebd\") " pod="openshift-cluster-node-tuning-operator/tuned-bzbzs" Jul 09 22:10:10.560558 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:10.560150 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/84b44025-bd7d-4910-ba37-d294bf354258-host-run-k8s-cni-cncf-io\") pod \"multus-lvcvn\" (UID: \"84b44025-bd7d-4910-ba37-d294bf354258\") " pod="openshift-multus/multus-lvcvn" Jul 09 22:10:10.561389 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:10.560242 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/5946d6db-f1f5-4fee-827b-1427b1bdd90f-agent-certs\") pod \"konnectivity-agent-gcmz7\" (UID: \"5946d6db-f1f5-4fee-827b-1427b1bdd90f\") " pod="kube-system/konnectivity-agent-gcmz7" Jul 09 22:10:10.561389 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:10.560289 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/50069e81-757a-482b-b7a2-ac26f2036ebd-etc-sysconfig\") pod \"tuned-bzbzs\" (UID: \"50069e81-757a-482b-b7a2-ac26f2036ebd\") " pod="openshift-cluster-node-tuning-operator/tuned-bzbzs" Jul 09 22:10:10.561389 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:10.560295 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/5946d6db-f1f5-4fee-827b-1427b1bdd90f-konnectivity-ca\") pod \"konnectivity-agent-gcmz7\" (UID: \"5946d6db-f1f5-4fee-827b-1427b1bdd90f\") " pod="kube-system/konnectivity-agent-gcmz7" Jul 09 22:10:10.561389 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:10.560335 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/499e80b4-0095-43da-ad29-f143e59a7357-os-release\") pod \"multus-additional-cni-plugins-5fzf2\" (UID: \"499e80b4-0095-43da-ad29-f143e59a7357\") " pod="openshift-multus/multus-additional-cni-plugins-5fzf2" Jul 09 22:10:10.561389 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:10.560331 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/13602a06-27c2-446f-a5aa-64c917c285c8-device-dir\") pod \"aws-ebs-csi-driver-node-669lr\" (UID: \"13602a06-27c2-446f-a5aa-64c917c285c8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-669lr" Jul 09 22:10:10.561389 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:10.560363 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/499e80b4-0095-43da-ad29-f143e59a7357-tuning-conf-dir\") pod \"multus-additional-cni-plugins-5fzf2\" (UID: \"499e80b4-0095-43da-ad29-f143e59a7357\") " pod="openshift-multus/multus-additional-cni-plugins-5fzf2" Jul 09 22:10:10.561389 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:10.560402 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/499e80b4-0095-43da-ad29-f143e59a7357-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-5fzf2\" (UID: \"499e80b4-0095-43da-ad29-f143e59a7357\") " pod="openshift-multus/multus-additional-cni-plugins-5fzf2" Jul 09 22:10:10.561389 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:10.560439 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/84b44025-bd7d-4910-ba37-d294bf354258-cni-binary-copy\") pod \"multus-lvcvn\" (UID: \"84b44025-bd7d-4910-ba37-d294bf354258\") " pod="openshift-multus/multus-lvcvn" Jul 09 22:10:10.561389 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:10.560470 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/7d1ac8fd-5e7b-439c-b48c-44dcf09b2934-ovn-node-metrics-cert\") pod \"ovnkube-node-r8dsp\" (UID: \"7d1ac8fd-5e7b-439c-b48c-44dcf09b2934\") " pod="openshift-ovn-kubernetes/ovnkube-node-r8dsp" Jul 09 22:10:10.561389 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:10.560495 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/499e80b4-0095-43da-ad29-f143e59a7357-cnibin\") pod \"multus-additional-cni-plugins-5fzf2\" (UID: \"499e80b4-0095-43da-ad29-f143e59a7357\") " pod="openshift-multus/multus-additional-cni-plugins-5fzf2" Jul 09 22:10:10.561389 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:10.560560 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h57g8\" (UniqueName: \"kubernetes.io/projected/13602a06-27c2-446f-a5aa-64c917c285c8-kube-api-access-h57g8\") pod \"aws-ebs-csi-driver-node-669lr\" (UID: \"13602a06-27c2-446f-a5aa-64c917c285c8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-669lr" Jul 09 22:10:10.561389 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:10.560607 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fx7mr\" (UniqueName: \"kubernetes.io/projected/541ce094-3638-4979-bf7b-782f3df1cbf8-kube-api-access-fx7mr\") pod \"network-check-target-9b9zs\" (UID: \"541ce094-3638-4979-bf7b-782f3df1cbf8\") " pod="openshift-network-diagnostics/network-check-target-9b9zs" Jul 09 22:10:10.561389 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:10.560719 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/84b44025-bd7d-4910-ba37-d294bf354258-host-run-multus-certs\") pod \"multus-lvcvn\" (UID: \"84b44025-bd7d-4910-ba37-d294bf354258\") " pod="openshift-multus/multus-lvcvn" Jul 09 22:10:10.561389 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:10.560751 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7d1ac8fd-5e7b-439c-b48c-44dcf09b2934-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-r8dsp\" (UID: \"7d1ac8fd-5e7b-439c-b48c-44dcf09b2934\") " pod="openshift-ovn-kubernetes/ovnkube-node-r8dsp" Jul 09 22:10:10.561389 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:10.560786 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b5e569ee-1aea-4eeb-8235-93438e648102-original-pull-secret\") pod \"global-pull-secret-syncer-bwdgc\" (UID: \"b5e569ee-1aea-4eeb-8235-93438e648102\") " pod="kube-system/global-pull-secret-syncer-bwdgc" Jul 09 22:10:10.561389 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:10.560812 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/50069e81-757a-482b-b7a2-ac26f2036ebd-sys\") pod \"tuned-bzbzs\" (UID: \"50069e81-757a-482b-b7a2-ac26f2036ebd\") " pod="openshift-cluster-node-tuning-operator/tuned-bzbzs" Jul 09 22:10:10.562124 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:10.560839 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/50069e81-757a-482b-b7a2-ac26f2036ebd-lib-modules\") pod \"tuned-bzbzs\" (UID: \"50069e81-757a-482b-b7a2-ac26f2036ebd\") " pod="openshift-cluster-node-tuning-operator/tuned-bzbzs" Jul 09 22:10:10.562124 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:10.560896 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/13602a06-27c2-446f-a5aa-64c917c285c8-kubelet-dir\") pod \"aws-ebs-csi-driver-node-669lr\" (UID: \"13602a06-27c2-446f-a5aa-64c917c285c8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-669lr" Jul 09 22:10:10.562124 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:10.560940 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/84b44025-bd7d-4910-ba37-d294bf354258-host-var-lib-cni-multus\") pod \"multus-lvcvn\" (UID: \"84b44025-bd7d-4910-ba37-d294bf354258\") " pod="openshift-multus/multus-lvcvn" Jul 09 22:10:10.562124 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:10.560979 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7d1ac8fd-5e7b-439c-b48c-44dcf09b2934-run-openvswitch\") pod \"ovnkube-node-r8dsp\" (UID: \"7d1ac8fd-5e7b-439c-b48c-44dcf09b2934\") " pod="openshift-ovn-kubernetes/ovnkube-node-r8dsp" Jul 09 22:10:10.562124 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:10.561004 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/50069e81-757a-482b-b7a2-ac26f2036ebd-etc-systemd\") pod \"tuned-bzbzs\" (UID: \"50069e81-757a-482b-b7a2-ac26f2036ebd\") " pod="openshift-cluster-node-tuning-operator/tuned-bzbzs" Jul 09 22:10:10.562124 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:10.561030 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5fdxd\" (UniqueName: \"kubernetes.io/projected/84b44025-bd7d-4910-ba37-d294bf354258-kube-api-access-5fdxd\") pod \"multus-lvcvn\" (UID: \"84b44025-bd7d-4910-ba37-d294bf354258\") " pod="openshift-multus/multus-lvcvn" Jul 09 22:10:10.562124 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:10.561089 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/7d1ac8fd-5e7b-439c-b48c-44dcf09b2934-host-slash\") pod \"ovnkube-node-r8dsp\" (UID: \"7d1ac8fd-5e7b-439c-b48c-44dcf09b2934\") " pod="openshift-ovn-kubernetes/ovnkube-node-r8dsp" Jul 09 22:10:10.562124 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:10.561118 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7d1ac8fd-5e7b-439c-b48c-44dcf09b2934-var-lib-openvswitch\") pod \"ovnkube-node-r8dsp\" (UID: \"7d1ac8fd-5e7b-439c-b48c-44dcf09b2934\") " pod="openshift-ovn-kubernetes/ovnkube-node-r8dsp" Jul 09 22:10:10.562124 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:10.561160 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/7d1ac8fd-5e7b-439c-b48c-44dcf09b2934-node-log\") pod \"ovnkube-node-r8dsp\" (UID: \"7d1ac8fd-5e7b-439c-b48c-44dcf09b2934\") " pod="openshift-ovn-kubernetes/ovnkube-node-r8dsp" Jul 09 22:10:10.562124 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:10.561186 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/b5e569ee-1aea-4eeb-8235-93438e648102-dbus\") pod \"global-pull-secret-syncer-bwdgc\" (UID: \"b5e569ee-1aea-4eeb-8235-93438e648102\") " pod="kube-system/global-pull-secret-syncer-bwdgc" Jul 09 22:10:10.562124 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:10.561213 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/50069e81-757a-482b-b7a2-ac26f2036ebd-etc-sysctl-d\") pod \"tuned-bzbzs\" (UID: \"50069e81-757a-482b-b7a2-ac26f2036ebd\") " pod="openshift-cluster-node-tuning-operator/tuned-bzbzs" Jul 09 22:10:10.562124 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:10.561241 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/13602a06-27c2-446f-a5aa-64c917c285c8-registration-dir\") pod \"aws-ebs-csi-driver-node-669lr\" (UID: \"13602a06-27c2-446f-a5aa-64c917c285c8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-669lr" Jul 09 22:10:10.562124 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:10.561267 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/23b023ec-a254-462d-a0b4-5c22fd905e72-metrics-certs\") pod \"network-metrics-daemon-8pj95\" (UID: \"23b023ec-a254-462d-a0b4-5c22fd905e72\") " pod="openshift-multus/network-metrics-daemon-8pj95" Jul 09 22:10:10.562124 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:10.561340 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/50069e81-757a-482b-b7a2-ac26f2036ebd-lib-modules\") pod \"tuned-bzbzs\" (UID: \"50069e81-757a-482b-b7a2-ac26f2036ebd\") " pod="openshift-cluster-node-tuning-operator/tuned-bzbzs" Jul 09 22:10:10.562124 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:10.561350 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/84b44025-bd7d-4910-ba37-d294bf354258-host-run-netns\") pod \"multus-lvcvn\" (UID: \"84b44025-bd7d-4910-ba37-d294bf354258\") " pod="openshift-multus/multus-lvcvn" Jul 09 22:10:10.562124 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:10.561377 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6807e2e6-d9a4-475a-9fb4-4b958127840f-tmp-dir\") pod \"node-resolver-nr4cq\" (UID: \"6807e2e6-d9a4-475a-9fb4-4b958127840f\") " pod="openshift-dns/node-resolver-nr4cq" Jul 09 22:10:10.562124 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:10.561403 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tzbvt\" (UniqueName: \"kubernetes.io/projected/7d1ac8fd-5e7b-439c-b48c-44dcf09b2934-kube-api-access-tzbvt\") pod \"ovnkube-node-r8dsp\" (UID: \"7d1ac8fd-5e7b-439c-b48c-44dcf09b2934\") " pod="openshift-ovn-kubernetes/ovnkube-node-r8dsp" Jul 09 22:10:10.562910 ip-10-0-136-251 kubenswrapper[2579]: E0709 22:10:10.561422 2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 09 22:10:10.562910 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:10.561432 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-25lz6\" (UniqueName: \"kubernetes.io/projected/af0be020-5b8b-4828-bf98-531265c795b2-kube-api-access-25lz6\") pod \"node-ca-l578c\" (UID: \"af0be020-5b8b-4828-bf98-531265c795b2\") " pod="openshift-image-registry/node-ca-l578c" Jul 09 22:10:10.562910 ip-10-0-136-251 kubenswrapper[2579]: E0709 22:10:10.561507 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b5e569ee-1aea-4eeb-8235-93438e648102-original-pull-secret podName:b5e569ee-1aea-4eeb-8235-93438e648102 nodeName:}" failed. No retries permitted until 2026-07-09 22:10:11.061466722 +0000 UTC m=+2.972919406 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b5e569ee-1aea-4eeb-8235-93438e648102-original-pull-secret") pod "global-pull-secret-syncer-bwdgc" (UID: "b5e569ee-1aea-4eeb-8235-93438e648102") : object "kube-system"/"original-pull-secret" not registered Jul 09 22:10:10.562910 ip-10-0-136-251 kubenswrapper[2579]: E0709 22:10:10.561662 2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 09 22:10:10.562910 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:10.561712 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/50069e81-757a-482b-b7a2-ac26f2036ebd-sys\") pod \"tuned-bzbzs\" (UID: \"50069e81-757a-482b-b7a2-ac26f2036ebd\") " pod="openshift-cluster-node-tuning-operator/tuned-bzbzs" Jul 09 22:10:10.562910 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:10.561761 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/50069e81-757a-482b-b7a2-ac26f2036ebd-etc-systemd\") pod \"tuned-bzbzs\" (UID: \"50069e81-757a-482b-b7a2-ac26f2036ebd\") " pod="openshift-cluster-node-tuning-operator/tuned-bzbzs" Jul 09 22:10:10.562910 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:10.561815 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/13602a06-27c2-446f-a5aa-64c917c285c8-kubelet-dir\") pod \"aws-ebs-csi-driver-node-669lr\" (UID: \"13602a06-27c2-446f-a5aa-64c917c285c8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-669lr" Jul 09 22:10:10.562910 ip-10-0-136-251 kubenswrapper[2579]: E0709 22:10:10.561719 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/23b023ec-a254-462d-a0b4-5c22fd905e72-metrics-certs podName:23b023ec-a254-462d-a0b4-5c22fd905e72 nodeName:}" failed. No retries permitted until 2026-07-09 22:10:11.06170217 +0000 UTC m=+2.973154856 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/23b023ec-a254-462d-a0b4-5c22fd905e72-metrics-certs") pod "network-metrics-daemon-8pj95" (UID: "23b023ec-a254-462d-a0b4-5c22fd905e72") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 09 22:10:10.562910 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:10.561851 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/b5e569ee-1aea-4eeb-8235-93438e648102-dbus\") pod \"global-pull-secret-syncer-bwdgc\" (UID: \"b5e569ee-1aea-4eeb-8235-93438e648102\") " pod="kube-system/global-pull-secret-syncer-bwdgc" Jul 09 22:10:10.562910 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:10.561956 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/50069e81-757a-482b-b7a2-ac26f2036ebd-etc-sysctl-d\") pod \"tuned-bzbzs\" (UID: \"50069e81-757a-482b-b7a2-ac26f2036ebd\") " pod="openshift-cluster-node-tuning-operator/tuned-bzbzs" Jul 09 22:10:10.562910 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:10.561999 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/13602a06-27c2-446f-a5aa-64c917c285c8-registration-dir\") pod \"aws-ebs-csi-driver-node-669lr\" (UID: \"13602a06-27c2-446f-a5aa-64c917c285c8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-669lr" Jul 09 22:10:10.567134 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:10.567051 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-251.ec2.internal" event={"ID":"021762e3fafa981d5236e3af644982e1","Type":"ContainerStarted","Data":"565286e5c968065605f345c5f00f2a6e8412e4219068930949417c8e6ba698e7"} Jul 09 22:10:10.568122 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:10.568058 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/50069e81-757a-482b-b7a2-ac26f2036ebd-tmp\") pod \"tuned-bzbzs\" (UID: \"50069e81-757a-482b-b7a2-ac26f2036ebd\") " pod="openshift-cluster-node-tuning-operator/tuned-bzbzs" Jul 09 22:10:10.568612 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:10.568555 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-251.ec2.internal" event={"ID":"50613fbded229913a0a9abdd811c3ca4","Type":"ContainerStarted","Data":"2ff48d6b1519924377314fe9f9cbb72de57557739dcf382c05adf25fd3a9d462"} Jul 09 22:10:10.569376 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:10.569306 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/50069e81-757a-482b-b7a2-ac26f2036ebd-etc-tuned\") pod \"tuned-bzbzs\" (UID: \"50069e81-757a-482b-b7a2-ac26f2036ebd\") " pod="openshift-cluster-node-tuning-operator/tuned-bzbzs" Jul 09 22:10:10.573355 ip-10-0-136-251 kubenswrapper[2579]: E0709 22:10:10.573331 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 09 22:10:10.573452 ip-10-0-136-251 kubenswrapper[2579]: E0709 22:10:10.573361 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 09 22:10:10.573452 ip-10-0-136-251 kubenswrapper[2579]: E0709 22:10:10.573374 2579 projected.go:196] Error preparing data for projected volume kube-api-access-fx7mr for pod openshift-network-diagnostics/network-check-target-9b9zs: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 09 22:10:10.573452 ip-10-0-136-251 kubenswrapper[2579]: E0709 22:10:10.573426 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/541ce094-3638-4979-bf7b-782f3df1cbf8-kube-api-access-fx7mr podName:541ce094-3638-4979-bf7b-782f3df1cbf8 nodeName:}" failed. No retries permitted until 2026-07-09 22:10:11.073410098 +0000 UTC m=+2.984862783 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-fx7mr" (UniqueName: "kubernetes.io/projected/541ce094-3638-4979-bf7b-782f3df1cbf8-kube-api-access-fx7mr") pod "network-check-target-9b9zs" (UID: "541ce094-3638-4979-bf7b-782f3df1cbf8") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 09 22:10:10.575654 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:10.575599 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dw6k7\" (UniqueName: \"kubernetes.io/projected/23b023ec-a254-462d-a0b4-5c22fd905e72-kube-api-access-dw6k7\") pod \"network-metrics-daemon-8pj95\" (UID: \"23b023ec-a254-462d-a0b4-5c22fd905e72\") " pod="openshift-multus/network-metrics-daemon-8pj95" Jul 09 22:10:10.576433 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:10.576407 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tpflr\" (UniqueName: \"kubernetes.io/projected/50069e81-757a-482b-b7a2-ac26f2036ebd-kube-api-access-tpflr\") pod \"tuned-bzbzs\" (UID: \"50069e81-757a-482b-b7a2-ac26f2036ebd\") " pod="openshift-cluster-node-tuning-operator/tuned-bzbzs" Jul 09 22:10:10.577322 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:10.577277 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-25lz6\" (UniqueName: \"kubernetes.io/projected/af0be020-5b8b-4828-bf98-531265c795b2-kube-api-access-25lz6\") pod \"node-ca-l578c\" (UID: \"af0be020-5b8b-4828-bf98-531265c795b2\") " pod="openshift-image-registry/node-ca-l578c" Jul 09 22:10:10.577953 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:10.577896 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ghmmf\" (UniqueName: \"kubernetes.io/projected/15cda8f6-b7fd-41f2-8aea-2ef4e9b9669b-kube-api-access-ghmmf\") pod \"iptables-alerter-6t57q\" (UID: \"15cda8f6-b7fd-41f2-8aea-2ef4e9b9669b\") " pod="openshift-network-operator/iptables-alerter-6t57q" Jul 09 22:10:10.580309 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:10.580274 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h57g8\" (UniqueName: \"kubernetes.io/projected/13602a06-27c2-446f-a5aa-64c917c285c8-kube-api-access-h57g8\") pod \"aws-ebs-csi-driver-node-669lr\" (UID: \"13602a06-27c2-446f-a5aa-64c917c285c8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-669lr" Jul 09 22:10:10.661803 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:10.661771 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/7d1ac8fd-5e7b-439c-b48c-44dcf09b2934-run-systemd\") pod \"ovnkube-node-r8dsp\" (UID: \"7d1ac8fd-5e7b-439c-b48c-44dcf09b2934\") " pod="openshift-ovn-kubernetes/ovnkube-node-r8dsp" Jul 09 22:10:10.661803 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:10.661811 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/7d1ac8fd-5e7b-439c-b48c-44dcf09b2934-host-cni-netd\") pod \"ovnkube-node-r8dsp\" (UID: \"7d1ac8fd-5e7b-439c-b48c-44dcf09b2934\") " pod="openshift-ovn-kubernetes/ovnkube-node-r8dsp" Jul 09 22:10:10.662082 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:10.661830 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/84b44025-bd7d-4910-ba37-d294bf354258-host-var-lib-kubelet\") pod \"multus-lvcvn\" (UID: \"84b44025-bd7d-4910-ba37-d294bf354258\") " pod="openshift-multus/multus-lvcvn" Jul 09 22:10:10.662082 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:10.661853 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7d1ac8fd-5e7b-439c-b48c-44dcf09b2934-etc-openvswitch\") pod \"ovnkube-node-r8dsp\" (UID: \"7d1ac8fd-5e7b-439c-b48c-44dcf09b2934\") " pod="openshift-ovn-kubernetes/ovnkube-node-r8dsp" Jul 09 22:10:10.662082 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:10.661875 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/7d1ac8fd-5e7b-439c-b48c-44dcf09b2934-ovnkube-config\") pod \"ovnkube-node-r8dsp\" (UID: \"7d1ac8fd-5e7b-439c-b48c-44dcf09b2934\") " pod="openshift-ovn-kubernetes/ovnkube-node-r8dsp" Jul 09 22:10:10.662082 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:10.661893 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/7d1ac8fd-5e7b-439c-b48c-44dcf09b2934-run-systemd\") pod \"ovnkube-node-r8dsp\" (UID: \"7d1ac8fd-5e7b-439c-b48c-44dcf09b2934\") " pod="openshift-ovn-kubernetes/ovnkube-node-r8dsp" Jul 09 22:10:10.662082 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:10.661928 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/7d1ac8fd-5e7b-439c-b48c-44dcf09b2934-host-cni-netd\") pod \"ovnkube-node-r8dsp\" (UID: \"7d1ac8fd-5e7b-439c-b48c-44dcf09b2934\") " pod="openshift-ovn-kubernetes/ovnkube-node-r8dsp" Jul 09 22:10:10.662082 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:10.661939 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/84b44025-bd7d-4910-ba37-d294bf354258-host-var-lib-kubelet\") pod \"multus-lvcvn\" (UID: \"84b44025-bd7d-4910-ba37-d294bf354258\") " pod="openshift-multus/multus-lvcvn" Jul 09 22:10:10.662082 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:10.661939 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7d1ac8fd-5e7b-439c-b48c-44dcf09b2934-etc-openvswitch\") pod \"ovnkube-node-r8dsp\" (UID: \"7d1ac8fd-5e7b-439c-b48c-44dcf09b2934\") " pod="openshift-ovn-kubernetes/ovnkube-node-r8dsp" Jul 09 22:10:10.662082 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:10.661945 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/84b44025-bd7d-4910-ba37-d294bf354258-multus-conf-dir\") pod \"multus-lvcvn\" (UID: \"84b44025-bd7d-4910-ba37-d294bf354258\") " pod="openshift-multus/multus-lvcvn" Jul 09 22:10:10.662082 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:10.661900 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/84b44025-bd7d-4910-ba37-d294bf354258-multus-conf-dir\") pod \"multus-lvcvn\" (UID: \"84b44025-bd7d-4910-ba37-d294bf354258\") " pod="openshift-multus/multus-lvcvn" Jul 09 22:10:10.662082 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:10.662007 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/499e80b4-0095-43da-ad29-f143e59a7357-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-5fzf2\" (UID: \"499e80b4-0095-43da-ad29-f143e59a7357\") " pod="openshift-multus/multus-additional-cni-plugins-5fzf2" Jul 09 22:10:10.662082 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:10.662037 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/84b44025-bd7d-4910-ba37-d294bf354258-etc-kubernetes\") pod \"multus-lvcvn\" (UID: \"84b44025-bd7d-4910-ba37-d294bf354258\") " pod="openshift-multus/multus-lvcvn" Jul 09 22:10:10.662082 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:10.662063 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/84b44025-bd7d-4910-ba37-d294bf354258-multus-daemon-config\") pod \"multus-lvcvn\" (UID: \"84b44025-bd7d-4910-ba37-d294bf354258\") " pod="openshift-multus/multus-lvcvn" Jul 09 22:10:10.662659 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:10.662096 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/7d1ac8fd-5e7b-439c-b48c-44dcf09b2934-host-run-netns\") pod \"ovnkube-node-r8dsp\" (UID: \"7d1ac8fd-5e7b-439c-b48c-44dcf09b2934\") " pod="openshift-ovn-kubernetes/ovnkube-node-r8dsp" Jul 09 22:10:10.662659 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:10.662112 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7d1ac8fd-5e7b-439c-b48c-44dcf09b2934-host-run-ovn-kubernetes\") pod \"ovnkube-node-r8dsp\" (UID: \"7d1ac8fd-5e7b-439c-b48c-44dcf09b2934\") " pod="openshift-ovn-kubernetes/ovnkube-node-r8dsp" Jul 09 22:10:10.662659 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:10.662121 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/84b44025-bd7d-4910-ba37-d294bf354258-etc-kubernetes\") pod \"multus-lvcvn\" (UID: \"84b44025-bd7d-4910-ba37-d294bf354258\") " pod="openshift-multus/multus-lvcvn" Jul 09 22:10:10.662659 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:10.662129 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z6827\" (UniqueName: \"kubernetes.io/projected/499e80b4-0095-43da-ad29-f143e59a7357-kube-api-access-z6827\") pod \"multus-additional-cni-plugins-5fzf2\" (UID: \"499e80b4-0095-43da-ad29-f143e59a7357\") " pod="openshift-multus/multus-additional-cni-plugins-5fzf2" Jul 09 22:10:10.662659 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:10.662160 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/84b44025-bd7d-4910-ba37-d294bf354258-os-release\") pod \"multus-lvcvn\" (UID: \"84b44025-bd7d-4910-ba37-d294bf354258\") " pod="openshift-multus/multus-lvcvn" Jul 09 22:10:10.662659 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:10.662185 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/7d1ac8fd-5e7b-439c-b48c-44dcf09b2934-host-kubelet\") pod \"ovnkube-node-r8dsp\" (UID: \"7d1ac8fd-5e7b-439c-b48c-44dcf09b2934\") " pod="openshift-ovn-kubernetes/ovnkube-node-r8dsp" Jul 09 22:10:10.662659 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:10.662210 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/7d1ac8fd-5e7b-439c-b48c-44dcf09b2934-env-overrides\") pod \"ovnkube-node-r8dsp\" (UID: \"7d1ac8fd-5e7b-439c-b48c-44dcf09b2934\") " pod="openshift-ovn-kubernetes/ovnkube-node-r8dsp" Jul 09 22:10:10.662659 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:10.662233 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/7d1ac8fd-5e7b-439c-b48c-44dcf09b2934-ovnkube-script-lib\") pod \"ovnkube-node-r8dsp\" (UID: \"7d1ac8fd-5e7b-439c-b48c-44dcf09b2934\") " pod="openshift-ovn-kubernetes/ovnkube-node-r8dsp" Jul 09 22:10:10.662659 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:10.662257 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/499e80b4-0095-43da-ad29-f143e59a7357-cni-binary-copy\") pod \"multus-additional-cni-plugins-5fzf2\" (UID: \"499e80b4-0095-43da-ad29-f143e59a7357\") " pod="openshift-multus/multus-additional-cni-plugins-5fzf2" Jul 09 22:10:10.662659 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:10.662283 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/84b44025-bd7d-4910-ba37-d294bf354258-system-cni-dir\") pod \"multus-lvcvn\" (UID: \"84b44025-bd7d-4910-ba37-d294bf354258\") " pod="openshift-multus/multus-lvcvn" Jul 09 22:10:10.662659 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:10.662305 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/84b44025-bd7d-4910-ba37-d294bf354258-multus-cni-dir\") pod \"multus-lvcvn\" (UID: \"84b44025-bd7d-4910-ba37-d294bf354258\") " pod="openshift-multus/multus-lvcvn" Jul 09 22:10:10.662659 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:10.662329 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/6807e2e6-d9a4-475a-9fb4-4b958127840f-hosts-file\") pod \"node-resolver-nr4cq\" (UID: \"6807e2e6-d9a4-475a-9fb4-4b958127840f\") " pod="openshift-dns/node-resolver-nr4cq" Jul 09 22:10:10.662659 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:10.662352 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/7d1ac8fd-5e7b-439c-b48c-44dcf09b2934-log-socket\") pod \"ovnkube-node-r8dsp\" (UID: \"7d1ac8fd-5e7b-439c-b48c-44dcf09b2934\") " pod="openshift-ovn-kubernetes/ovnkube-node-r8dsp" Jul 09 22:10:10.662659 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:10.662378 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/499e80b4-0095-43da-ad29-f143e59a7357-system-cni-dir\") pod \"multus-additional-cni-plugins-5fzf2\" (UID: \"499e80b4-0095-43da-ad29-f143e59a7357\") " pod="openshift-multus/multus-additional-cni-plugins-5fzf2" Jul 09 22:10:10.662659 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:10.662405 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/84b44025-bd7d-4910-ba37-d294bf354258-cnibin\") pod \"multus-lvcvn\" (UID: \"84b44025-bd7d-4910-ba37-d294bf354258\") " pod="openshift-multus/multus-lvcvn" Jul 09 22:10:10.662659 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:10.662428 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/7d1ac8fd-5e7b-439c-b48c-44dcf09b2934-host-cni-bin\") pod \"ovnkube-node-r8dsp\" (UID: \"7d1ac8fd-5e7b-439c-b48c-44dcf09b2934\") " pod="openshift-ovn-kubernetes/ovnkube-node-r8dsp" Jul 09 22:10:10.662659 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:10.662453 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/84b44025-bd7d-4910-ba37-d294bf354258-multus-socket-dir-parent\") pod \"multus-lvcvn\" (UID: \"84b44025-bd7d-4910-ba37-d294bf354258\") " pod="openshift-multus/multus-lvcvn" Jul 09 22:10:10.663464 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:10.662476 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/84b44025-bd7d-4910-ba37-d294bf354258-host-var-lib-cni-bin\") pod \"multus-lvcvn\" (UID: \"84b44025-bd7d-4910-ba37-d294bf354258\") " pod="openshift-multus/multus-lvcvn" Jul 09 22:10:10.663464 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:10.662499 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/84b44025-bd7d-4910-ba37-d294bf354258-hostroot\") pod \"multus-lvcvn\" (UID: \"84b44025-bd7d-4910-ba37-d294bf354258\") " pod="openshift-multus/multus-lvcvn" Jul 09 22:10:10.663464 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:10.662523 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cf2ms\" (UniqueName: \"kubernetes.io/projected/6807e2e6-d9a4-475a-9fb4-4b958127840f-kube-api-access-cf2ms\") pod \"node-resolver-nr4cq\" (UID: \"6807e2e6-d9a4-475a-9fb4-4b958127840f\") " pod="openshift-dns/node-resolver-nr4cq" Jul 09 22:10:10.663464 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:10.662531 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/499e80b4-0095-43da-ad29-f143e59a7357-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-5fzf2\" (UID: \"499e80b4-0095-43da-ad29-f143e59a7357\") " pod="openshift-multus/multus-additional-cni-plugins-5fzf2" Jul 09 22:10:10.663464 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:10.662531 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/7d1ac8fd-5e7b-439c-b48c-44dcf09b2934-ovnkube-config\") pod \"ovnkube-node-r8dsp\" (UID: \"7d1ac8fd-5e7b-439c-b48c-44dcf09b2934\") " pod="openshift-ovn-kubernetes/ovnkube-node-r8dsp" Jul 09 22:10:10.663464 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:10.662550 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/7d1ac8fd-5e7b-439c-b48c-44dcf09b2934-systemd-units\") pod \"ovnkube-node-r8dsp\" (UID: \"7d1ac8fd-5e7b-439c-b48c-44dcf09b2934\") " pod="openshift-ovn-kubernetes/ovnkube-node-r8dsp" Jul 09 22:10:10.663464 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:10.662579 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/7d1ac8fd-5e7b-439c-b48c-44dcf09b2934-host-run-netns\") pod \"ovnkube-node-r8dsp\" (UID: \"7d1ac8fd-5e7b-439c-b48c-44dcf09b2934\") " pod="openshift-ovn-kubernetes/ovnkube-node-r8dsp" Jul 09 22:10:10.663464 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:10.662626 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/7d1ac8fd-5e7b-439c-b48c-44dcf09b2934-run-ovn\") pod \"ovnkube-node-r8dsp\" (UID: \"7d1ac8fd-5e7b-439c-b48c-44dcf09b2934\") " pod="openshift-ovn-kubernetes/ovnkube-node-r8dsp" Jul 09 22:10:10.663464 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:10.662628 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/7d1ac8fd-5e7b-439c-b48c-44dcf09b2934-log-socket\") pod \"ovnkube-node-r8dsp\" (UID: \"7d1ac8fd-5e7b-439c-b48c-44dcf09b2934\") " pod="openshift-ovn-kubernetes/ovnkube-node-r8dsp" Jul 09 22:10:10.663464 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:10.662655 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/84b44025-bd7d-4910-ba37-d294bf354258-host-run-k8s-cni-cncf-io\") pod \"multus-lvcvn\" (UID: \"84b44025-bd7d-4910-ba37-d294bf354258\") " pod="openshift-multus/multus-lvcvn" Jul 09 22:10:10.663464 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:10.662661 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/7d1ac8fd-5e7b-439c-b48c-44dcf09b2934-systemd-units\") pod \"ovnkube-node-r8dsp\" (UID: \"7d1ac8fd-5e7b-439c-b48c-44dcf09b2934\") " pod="openshift-ovn-kubernetes/ovnkube-node-r8dsp" Jul 09 22:10:10.663464 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:10.662698 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/84b44025-bd7d-4910-ba37-d294bf354258-hostroot\") pod \"multus-lvcvn\" (UID: \"84b44025-bd7d-4910-ba37-d294bf354258\") " pod="openshift-multus/multus-lvcvn" Jul 09 22:10:10.663464 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:10.662701 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/7d1ac8fd-5e7b-439c-b48c-44dcf09b2934-host-kubelet\") pod \"ovnkube-node-r8dsp\" (UID: \"7d1ac8fd-5e7b-439c-b48c-44dcf09b2934\") " pod="openshift-ovn-kubernetes/ovnkube-node-r8dsp" Jul 09 22:10:10.663464 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:10.662727 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/5946d6db-f1f5-4fee-827b-1427b1bdd90f-agent-certs\") pod \"konnectivity-agent-gcmz7\" (UID: \"5946d6db-f1f5-4fee-827b-1427b1bdd90f\") " pod="kube-system/konnectivity-agent-gcmz7" Jul 09 22:10:10.663464 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:10.662774 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/5946d6db-f1f5-4fee-827b-1427b1bdd90f-konnectivity-ca\") pod \"konnectivity-agent-gcmz7\" (UID: \"5946d6db-f1f5-4fee-827b-1427b1bdd90f\") " pod="kube-system/konnectivity-agent-gcmz7" Jul 09 22:10:10.663464 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:10.662778 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/84b44025-bd7d-4910-ba37-d294bf354258-multus-daemon-config\") pod \"multus-lvcvn\" (UID: \"84b44025-bd7d-4910-ba37-d294bf354258\") " pod="openshift-multus/multus-lvcvn" Jul 09 22:10:10.663464 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:10.662788 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/84b44025-bd7d-4910-ba37-d294bf354258-cnibin\") pod \"multus-lvcvn\" (UID: \"84b44025-bd7d-4910-ba37-d294bf354258\") " pod="openshift-multus/multus-lvcvn" Jul 09 22:10:10.663464 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:10.662828 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/6807e2e6-d9a4-475a-9fb4-4b958127840f-hosts-file\") pod \"node-resolver-nr4cq\" (UID: \"6807e2e6-d9a4-475a-9fb4-4b958127840f\") " pod="openshift-dns/node-resolver-nr4cq" Jul 09 22:10:10.664354 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:10.662820 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/84b44025-bd7d-4910-ba37-d294bf354258-host-run-k8s-cni-cncf-io\") pod \"multus-lvcvn\" (UID: \"84b44025-bd7d-4910-ba37-d294bf354258\") " pod="openshift-multus/multus-lvcvn" Jul 09 22:10:10.664354 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:10.662831 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/7d1ac8fd-5e7b-439c-b48c-44dcf09b2934-host-cni-bin\") pod \"ovnkube-node-r8dsp\" (UID: \"7d1ac8fd-5e7b-439c-b48c-44dcf09b2934\") " pod="openshift-ovn-kubernetes/ovnkube-node-r8dsp" Jul 09 22:10:10.664354 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:10.662851 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/84b44025-bd7d-4910-ba37-d294bf354258-os-release\") pod \"multus-lvcvn\" (UID: \"84b44025-bd7d-4910-ba37-d294bf354258\") " pod="openshift-multus/multus-lvcvn" Jul 09 22:10:10.664354 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:10.662863 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7d1ac8fd-5e7b-439c-b48c-44dcf09b2934-host-run-ovn-kubernetes\") pod \"ovnkube-node-r8dsp\" (UID: \"7d1ac8fd-5e7b-439c-b48c-44dcf09b2934\") " pod="openshift-ovn-kubernetes/ovnkube-node-r8dsp" Jul 09 22:10:10.664354 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:10.662862 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/499e80b4-0095-43da-ad29-f143e59a7357-os-release\") pod \"multus-additional-cni-plugins-5fzf2\" (UID: \"499e80b4-0095-43da-ad29-f143e59a7357\") " pod="openshift-multus/multus-additional-cni-plugins-5fzf2" Jul 09 22:10:10.664354 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:10.662881 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/499e80b4-0095-43da-ad29-f143e59a7357-system-cni-dir\") pod \"multus-additional-cni-plugins-5fzf2\" (UID: \"499e80b4-0095-43da-ad29-f143e59a7357\") " pod="openshift-multus/multus-additional-cni-plugins-5fzf2" Jul 09 22:10:10.664354 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:10.662888 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/7d1ac8fd-5e7b-439c-b48c-44dcf09b2934-run-ovn\") pod \"ovnkube-node-r8dsp\" (UID: \"7d1ac8fd-5e7b-439c-b48c-44dcf09b2934\") " pod="openshift-ovn-kubernetes/ovnkube-node-r8dsp" Jul 09 22:10:10.664354 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:10.662899 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/499e80b4-0095-43da-ad29-f143e59a7357-tuning-conf-dir\") pod \"multus-additional-cni-plugins-5fzf2\" (UID: \"499e80b4-0095-43da-ad29-f143e59a7357\") " pod="openshift-multus/multus-additional-cni-plugins-5fzf2" Jul 09 22:10:10.664354 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:10.662935 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/499e80b4-0095-43da-ad29-f143e59a7357-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-5fzf2\" (UID: \"499e80b4-0095-43da-ad29-f143e59a7357\") " pod="openshift-multus/multus-additional-cni-plugins-5fzf2" Jul 09 22:10:10.664354 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:10.662935 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/499e80b4-0095-43da-ad29-f143e59a7357-os-release\") pod \"multus-additional-cni-plugins-5fzf2\" (UID: \"499e80b4-0095-43da-ad29-f143e59a7357\") " pod="openshift-multus/multus-additional-cni-plugins-5fzf2" Jul 09 22:10:10.664354 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:10.662952 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/84b44025-bd7d-4910-ba37-d294bf354258-cni-binary-copy\") pod \"multus-lvcvn\" (UID: \"84b44025-bd7d-4910-ba37-d294bf354258\") " pod="openshift-multus/multus-lvcvn" Jul 09 22:10:10.664354 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:10.662979 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/84b44025-bd7d-4910-ba37-d294bf354258-host-var-lib-cni-bin\") pod \"multus-lvcvn\" (UID: \"84b44025-bd7d-4910-ba37-d294bf354258\") " pod="openshift-multus/multus-lvcvn" Jul 09 22:10:10.664354 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:10.663025 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/7d1ac8fd-5e7b-439c-b48c-44dcf09b2934-ovn-node-metrics-cert\") pod \"ovnkube-node-r8dsp\" (UID: \"7d1ac8fd-5e7b-439c-b48c-44dcf09b2934\") " pod="openshift-ovn-kubernetes/ovnkube-node-r8dsp" Jul 09 22:10:10.664354 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:10.663052 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/499e80b4-0095-43da-ad29-f143e59a7357-cnibin\") pod \"multus-additional-cni-plugins-5fzf2\" (UID: \"499e80b4-0095-43da-ad29-f143e59a7357\") " pod="openshift-multus/multus-additional-cni-plugins-5fzf2" Jul 09 22:10:10.664354 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:10.663096 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/84b44025-bd7d-4910-ba37-d294bf354258-host-run-multus-certs\") pod \"multus-lvcvn\" (UID: \"84b44025-bd7d-4910-ba37-d294bf354258\") " pod="openshift-multus/multus-lvcvn" Jul 09 22:10:10.664354 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:10.663121 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7d1ac8fd-5e7b-439c-b48c-44dcf09b2934-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-r8dsp\" (UID: \"7d1ac8fd-5e7b-439c-b48c-44dcf09b2934\") " pod="openshift-ovn-kubernetes/ovnkube-node-r8dsp" Jul 09 22:10:10.664354 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:10.663270 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/7d1ac8fd-5e7b-439c-b48c-44dcf09b2934-env-overrides\") pod \"ovnkube-node-r8dsp\" (UID: \"7d1ac8fd-5e7b-439c-b48c-44dcf09b2934\") " pod="openshift-ovn-kubernetes/ovnkube-node-r8dsp" Jul 09 22:10:10.665065 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:10.663293 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/84b44025-bd7d-4910-ba37-d294bf354258-multus-socket-dir-parent\") pod \"multus-lvcvn\" (UID: \"84b44025-bd7d-4910-ba37-d294bf354258\") " pod="openshift-multus/multus-lvcvn" Jul 09 22:10:10.665065 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:10.663473 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/499e80b4-0095-43da-ad29-f143e59a7357-cni-binary-copy\") pod \"multus-additional-cni-plugins-5fzf2\" (UID: \"499e80b4-0095-43da-ad29-f143e59a7357\") " pod="openshift-multus/multus-additional-cni-plugins-5fzf2" Jul 09 22:10:10.665065 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:10.663620 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/499e80b4-0095-43da-ad29-f143e59a7357-tuning-conf-dir\") pod \"multus-additional-cni-plugins-5fzf2\" (UID: \"499e80b4-0095-43da-ad29-f143e59a7357\") " pod="openshift-multus/multus-additional-cni-plugins-5fzf2" Jul 09 22:10:10.665065 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:10.663637 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/7d1ac8fd-5e7b-439c-b48c-44dcf09b2934-ovnkube-script-lib\") pod \"ovnkube-node-r8dsp\" (UID: \"7d1ac8fd-5e7b-439c-b48c-44dcf09b2934\") " pod="openshift-ovn-kubernetes/ovnkube-node-r8dsp" Jul 09 22:10:10.665065 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:10.663696 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/499e80b4-0095-43da-ad29-f143e59a7357-cnibin\") pod \"multus-additional-cni-plugins-5fzf2\" (UID: \"499e80b4-0095-43da-ad29-f143e59a7357\") " pod="openshift-multus/multus-additional-cni-plugins-5fzf2" Jul 09 22:10:10.665065 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:10.663740 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/84b44025-bd7d-4910-ba37-d294bf354258-host-run-multus-certs\") pod \"multus-lvcvn\" (UID: \"84b44025-bd7d-4910-ba37-d294bf354258\") " pod="openshift-multus/multus-lvcvn" Jul 09 22:10:10.665065 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:10.663764 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/84b44025-bd7d-4910-ba37-d294bf354258-cni-binary-copy\") pod \"multus-lvcvn\" (UID: \"84b44025-bd7d-4910-ba37-d294bf354258\") " pod="openshift-multus/multus-lvcvn" Jul 09 22:10:10.665065 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:10.663786 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7d1ac8fd-5e7b-439c-b48c-44dcf09b2934-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-r8dsp\" (UID: \"7d1ac8fd-5e7b-439c-b48c-44dcf09b2934\") " pod="openshift-ovn-kubernetes/ovnkube-node-r8dsp" Jul 09 22:10:10.665065 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:10.663827 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/84b44025-bd7d-4910-ba37-d294bf354258-host-var-lib-cni-multus\") pod \"multus-lvcvn\" (UID: \"84b44025-bd7d-4910-ba37-d294bf354258\") " pod="openshift-multus/multus-lvcvn" Jul 09 22:10:10.665065 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:10.663842 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/84b44025-bd7d-4910-ba37-d294bf354258-multus-cni-dir\") pod \"multus-lvcvn\" (UID: \"84b44025-bd7d-4910-ba37-d294bf354258\") " pod="openshift-multus/multus-lvcvn" Jul 09 22:10:10.665065 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:10.663859 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7d1ac8fd-5e7b-439c-b48c-44dcf09b2934-run-openvswitch\") pod \"ovnkube-node-r8dsp\" (UID: \"7d1ac8fd-5e7b-439c-b48c-44dcf09b2934\") " pod="openshift-ovn-kubernetes/ovnkube-node-r8dsp" Jul 09 22:10:10.665065 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:10.663900 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/84b44025-bd7d-4910-ba37-d294bf354258-system-cni-dir\") pod \"multus-lvcvn\" (UID: \"84b44025-bd7d-4910-ba37-d294bf354258\") " pod="openshift-multus/multus-lvcvn" Jul 09 22:10:10.665065 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:10.663910 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/84b44025-bd7d-4910-ba37-d294bf354258-host-var-lib-cni-multus\") pod \"multus-lvcvn\" (UID: \"84b44025-bd7d-4910-ba37-d294bf354258\") " pod="openshift-multus/multus-lvcvn" Jul 09 22:10:10.665065 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:10.663961 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7d1ac8fd-5e7b-439c-b48c-44dcf09b2934-run-openvswitch\") pod \"ovnkube-node-r8dsp\" (UID: \"7d1ac8fd-5e7b-439c-b48c-44dcf09b2934\") " pod="openshift-ovn-kubernetes/ovnkube-node-r8dsp" Jul 09 22:10:10.665065 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:10.663964 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5fdxd\" (UniqueName: \"kubernetes.io/projected/84b44025-bd7d-4910-ba37-d294bf354258-kube-api-access-5fdxd\") pod \"multus-lvcvn\" (UID: \"84b44025-bd7d-4910-ba37-d294bf354258\") " pod="openshift-multus/multus-lvcvn" Jul 09 22:10:10.665065 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:10.664002 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/7d1ac8fd-5e7b-439c-b48c-44dcf09b2934-host-slash\") pod \"ovnkube-node-r8dsp\" (UID: \"7d1ac8fd-5e7b-439c-b48c-44dcf09b2934\") " pod="openshift-ovn-kubernetes/ovnkube-node-r8dsp" Jul 09 22:10:10.665065 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:10.664080 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/499e80b4-0095-43da-ad29-f143e59a7357-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-5fzf2\" (UID: \"499e80b4-0095-43da-ad29-f143e59a7357\") " pod="openshift-multus/multus-additional-cni-plugins-5fzf2" Jul 09 22:10:10.665751 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:10.664125 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7d1ac8fd-5e7b-439c-b48c-44dcf09b2934-var-lib-openvswitch\") pod \"ovnkube-node-r8dsp\" (UID: \"7d1ac8fd-5e7b-439c-b48c-44dcf09b2934\") " pod="openshift-ovn-kubernetes/ovnkube-node-r8dsp" Jul 09 22:10:10.665751 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:10.664130 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/5946d6db-f1f5-4fee-827b-1427b1bdd90f-konnectivity-ca\") pod \"konnectivity-agent-gcmz7\" (UID: \"5946d6db-f1f5-4fee-827b-1427b1bdd90f\") " pod="kube-system/konnectivity-agent-gcmz7" Jul 09 22:10:10.665751 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:10.664129 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/7d1ac8fd-5e7b-439c-b48c-44dcf09b2934-host-slash\") pod \"ovnkube-node-r8dsp\" (UID: \"7d1ac8fd-5e7b-439c-b48c-44dcf09b2934\") " pod="openshift-ovn-kubernetes/ovnkube-node-r8dsp" Jul 09 22:10:10.665751 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:10.664153 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/7d1ac8fd-5e7b-439c-b48c-44dcf09b2934-node-log\") pod \"ovnkube-node-r8dsp\" (UID: \"7d1ac8fd-5e7b-439c-b48c-44dcf09b2934\") " pod="openshift-ovn-kubernetes/ovnkube-node-r8dsp" Jul 09 22:10:10.665751 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:10.664181 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7d1ac8fd-5e7b-439c-b48c-44dcf09b2934-var-lib-openvswitch\") pod \"ovnkube-node-r8dsp\" (UID: \"7d1ac8fd-5e7b-439c-b48c-44dcf09b2934\") " pod="openshift-ovn-kubernetes/ovnkube-node-r8dsp" Jul 09 22:10:10.665751 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:10.664187 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/7d1ac8fd-5e7b-439c-b48c-44dcf09b2934-node-log\") pod \"ovnkube-node-r8dsp\" (UID: \"7d1ac8fd-5e7b-439c-b48c-44dcf09b2934\") " pod="openshift-ovn-kubernetes/ovnkube-node-r8dsp" Jul 09 22:10:10.665751 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:10.664225 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/84b44025-bd7d-4910-ba37-d294bf354258-host-run-netns\") pod \"multus-lvcvn\" (UID: \"84b44025-bd7d-4910-ba37-d294bf354258\") " pod="openshift-multus/multus-lvcvn" Jul 09 22:10:10.665751 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:10.664254 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6807e2e6-d9a4-475a-9fb4-4b958127840f-tmp-dir\") pod \"node-resolver-nr4cq\" (UID: \"6807e2e6-d9a4-475a-9fb4-4b958127840f\") " pod="openshift-dns/node-resolver-nr4cq" Jul 09 22:10:10.665751 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:10.664286 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tzbvt\" (UniqueName: \"kubernetes.io/projected/7d1ac8fd-5e7b-439c-b48c-44dcf09b2934-kube-api-access-tzbvt\") pod \"ovnkube-node-r8dsp\" (UID: \"7d1ac8fd-5e7b-439c-b48c-44dcf09b2934\") " pod="openshift-ovn-kubernetes/ovnkube-node-r8dsp" Jul 09 22:10:10.665751 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:10.664300 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/84b44025-bd7d-4910-ba37-d294bf354258-host-run-netns\") pod \"multus-lvcvn\" (UID: \"84b44025-bd7d-4910-ba37-d294bf354258\") " pod="openshift-multus/multus-lvcvn" Jul 09 22:10:10.665751 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:10.664557 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6807e2e6-d9a4-475a-9fb4-4b958127840f-tmp-dir\") pod \"node-resolver-nr4cq\" (UID: \"6807e2e6-d9a4-475a-9fb4-4b958127840f\") " pod="openshift-dns/node-resolver-nr4cq" Jul 09 22:10:10.666251 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:10.666178 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/7d1ac8fd-5e7b-439c-b48c-44dcf09b2934-ovn-node-metrics-cert\") pod \"ovnkube-node-r8dsp\" (UID: \"7d1ac8fd-5e7b-439c-b48c-44dcf09b2934\") " pod="openshift-ovn-kubernetes/ovnkube-node-r8dsp" Jul 09 22:10:10.666440 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:10.666411 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/5946d6db-f1f5-4fee-827b-1427b1bdd90f-agent-certs\") pod \"konnectivity-agent-gcmz7\" (UID: \"5946d6db-f1f5-4fee-827b-1427b1bdd90f\") " pod="kube-system/konnectivity-agent-gcmz7" Jul 09 22:10:10.671899 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:10.671872 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tzbvt\" (UniqueName: \"kubernetes.io/projected/7d1ac8fd-5e7b-439c-b48c-44dcf09b2934-kube-api-access-tzbvt\") pod \"ovnkube-node-r8dsp\" (UID: \"7d1ac8fd-5e7b-439c-b48c-44dcf09b2934\") " pod="openshift-ovn-kubernetes/ovnkube-node-r8dsp" Jul 09 22:10:10.672155 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:10.672122 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5fdxd\" (UniqueName: \"kubernetes.io/projected/84b44025-bd7d-4910-ba37-d294bf354258-kube-api-access-5fdxd\") pod \"multus-lvcvn\" (UID: \"84b44025-bd7d-4910-ba37-d294bf354258\") " pod="openshift-multus/multus-lvcvn" Jul 09 22:10:10.672956 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:10.672934 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z6827\" (UniqueName: \"kubernetes.io/projected/499e80b4-0095-43da-ad29-f143e59a7357-kube-api-access-z6827\") pod \"multus-additional-cni-plugins-5fzf2\" (UID: \"499e80b4-0095-43da-ad29-f143e59a7357\") " pod="openshift-multus/multus-additional-cni-plugins-5fzf2" Jul 09 22:10:10.673349 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:10.673329 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cf2ms\" (UniqueName: \"kubernetes.io/projected/6807e2e6-d9a4-475a-9fb4-4b958127840f-kube-api-access-cf2ms\") pod \"node-resolver-nr4cq\" (UID: \"6807e2e6-d9a4-475a-9fb4-4b958127840f\") " pod="openshift-dns/node-resolver-nr4cq" Jul 09 22:10:10.753217 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:10.753130 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-l578c" Jul 09 22:10:10.761653 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:10:10.761627 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaf0be020_5b8b_4828_bf98_531265c795b2.slice/crio-8ada4ff978f43d3ac370714d41e996b38cedd113a484f76c10f33306b2c52fa6 WatchSource:0}: Error finding container 8ada4ff978f43d3ac370714d41e996b38cedd113a484f76c10f33306b2c52fa6: Status 404 returned error can't find the container with id 8ada4ff978f43d3ac370714d41e996b38cedd113a484f76c10f33306b2c52fa6 Jul 09 22:10:10.761756 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:10.761722 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-bzbzs" Jul 09 22:10:10.769770 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:10:10.769739 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod50069e81_757a_482b_b7a2_ac26f2036ebd.slice/crio-12b11cb5c810328ea35d47f20e019514b8344db6b5c9dab086d2676e5edb2bee WatchSource:0}: Error finding container 12b11cb5c810328ea35d47f20e019514b8344db6b5c9dab086d2676e5edb2bee: Status 404 returned error can't find the container with id 12b11cb5c810328ea35d47f20e019514b8344db6b5c9dab086d2676e5edb2bee Jul 09 22:10:10.772750 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:10.772689 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-6t57q" Jul 09 22:10:10.780210 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:10.780186 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-669lr" Jul 09 22:10:10.783666 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:10:10.783640 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod15cda8f6_b7fd_41f2_8aea_2ef4e9b9669b.slice/crio-cb3510693cb5be8c92097f8c8beb05a28878f825a0d11cdf5bf5f50dac534109 WatchSource:0}: Error finding container cb3510693cb5be8c92097f8c8beb05a28878f825a0d11cdf5bf5f50dac534109: Status 404 returned error can't find the container with id cb3510693cb5be8c92097f8c8beb05a28878f825a0d11cdf5bf5f50dac534109 Jul 09 22:10:10.788041 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:10.788018 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-nr4cq" Jul 09 22:10:10.789115 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:10:10.789093 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod13602a06_27c2_446f_a5aa_64c917c285c8.slice/crio-828d553eed9904b8012dca02836c7cbc45d58bd57b768e943ff76c1d771bdf5b WatchSource:0}: Error finding container 828d553eed9904b8012dca02836c7cbc45d58bd57b768e943ff76c1d771bdf5b: Status 404 returned error can't find the container with id 828d553eed9904b8012dca02836c7cbc45d58bd57b768e943ff76c1d771bdf5b Jul 09 22:10:10.796491 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:10.796421 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-5fzf2" Jul 09 22:10:10.796491 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:10:10.796464 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6807e2e6_d9a4_475a_9fb4_4b958127840f.slice/crio-71e324be567dd00252f7f7ca8fce930daf436b0ca7ca96201b149676de1fdf50 WatchSource:0}: Error finding container 71e324be567dd00252f7f7ca8fce930daf436b0ca7ca96201b149676de1fdf50: Status 404 returned error can't find the container with id 71e324be567dd00252f7f7ca8fce930daf436b0ca7ca96201b149676de1fdf50 Jul 09 22:10:10.805137 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:10:10.805107 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod499e80b4_0095_43da_ad29_f143e59a7357.slice/crio-e88a24efc49b9e5c54d4696d5979f4ed4f8541f0fca8a3aa4388ba8b727ce11a WatchSource:0}: Error finding container e88a24efc49b9e5c54d4696d5979f4ed4f8541f0fca8a3aa4388ba8b727ce11a: Status 404 returned error can't find the container with id e88a24efc49b9e5c54d4696d5979f4ed4f8541f0fca8a3aa4388ba8b727ce11a Jul 09 22:10:10.805783 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:10.805763 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-r8dsp" Jul 09 22:10:10.812990 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:10.812970 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-gcmz7" Jul 09 22:10:10.814256 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:10:10.814230 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7d1ac8fd_5e7b_439c_b48c_44dcf09b2934.slice/crio-bdcb0b83c6f9cb6ccfd51a76fa03731578722cae69e6f81a26124fdbf63a9cd5 WatchSource:0}: Error finding container bdcb0b83c6f9cb6ccfd51a76fa03731578722cae69e6f81a26124fdbf63a9cd5: Status 404 returned error can't find the container with id bdcb0b83c6f9cb6ccfd51a76fa03731578722cae69e6f81a26124fdbf63a9cd5 Jul 09 22:10:10.820237 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:10.820209 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-lvcvn" Jul 09 22:10:10.823855 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:10:10.823828 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5946d6db_f1f5_4fee_827b_1427b1bdd90f.slice/crio-0f497d1a5b4cb0bfeb480255f10abcca759101c4a4fe62f97bc62c44dec1c0bb WatchSource:0}: Error finding container 0f497d1a5b4cb0bfeb480255f10abcca759101c4a4fe62f97bc62c44dec1c0bb: Status 404 returned error can't find the container with id 0f497d1a5b4cb0bfeb480255f10abcca759101c4a4fe62f97bc62c44dec1c0bb Jul 09 22:10:10.828866 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:10:10.828840 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod84b44025_bd7d_4910_ba37_d294bf354258.slice/crio-a239012d8cf5835c0de4544e8f2b58200803b7b35149c2296fe21a555ded3b32 WatchSource:0}: Error finding container a239012d8cf5835c0de4544e8f2b58200803b7b35149c2296fe21a555ded3b32: Status 404 returned error can't find the container with id a239012d8cf5835c0de4544e8f2b58200803b7b35149c2296fe21a555ded3b32 Jul 09 22:10:10.908056 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:10.908015 2579 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 09 22:10:11.066088 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:11.066005 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b5e569ee-1aea-4eeb-8235-93438e648102-original-pull-secret\") pod \"global-pull-secret-syncer-bwdgc\" (UID: \"b5e569ee-1aea-4eeb-8235-93438e648102\") " pod="kube-system/global-pull-secret-syncer-bwdgc" Jul 09 22:10:11.066088 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:11.066051 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/23b023ec-a254-462d-a0b4-5c22fd905e72-metrics-certs\") pod \"network-metrics-daemon-8pj95\" (UID: \"23b023ec-a254-462d-a0b4-5c22fd905e72\") " pod="openshift-multus/network-metrics-daemon-8pj95" Jul 09 22:10:11.066302 ip-10-0-136-251 kubenswrapper[2579]: E0709 22:10:11.066219 2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 09 22:10:11.066302 ip-10-0-136-251 kubenswrapper[2579]: E0709 22:10:11.066282 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/23b023ec-a254-462d-a0b4-5c22fd905e72-metrics-certs podName:23b023ec-a254-462d-a0b4-5c22fd905e72 nodeName:}" failed. No retries permitted until 2026-07-09 22:10:12.066263708 +0000 UTC m=+3.977716392 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/23b023ec-a254-462d-a0b4-5c22fd905e72-metrics-certs") pod "network-metrics-daemon-8pj95" (UID: "23b023ec-a254-462d-a0b4-5c22fd905e72") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 09 22:10:11.066526 ip-10-0-136-251 kubenswrapper[2579]: E0709 22:10:11.066502 2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 09 22:10:11.066627 ip-10-0-136-251 kubenswrapper[2579]: E0709 22:10:11.066565 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b5e569ee-1aea-4eeb-8235-93438e648102-original-pull-secret podName:b5e569ee-1aea-4eeb-8235-93438e648102 nodeName:}" failed. No retries permitted until 2026-07-09 22:10:12.066550222 +0000 UTC m=+3.978002917 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b5e569ee-1aea-4eeb-8235-93438e648102-original-pull-secret") pod "global-pull-secret-syncer-bwdgc" (UID: "b5e569ee-1aea-4eeb-8235-93438e648102") : object "kube-system"/"original-pull-secret" not registered Jul 09 22:10:11.166788 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:11.166740 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fx7mr\" (UniqueName: \"kubernetes.io/projected/541ce094-3638-4979-bf7b-782f3df1cbf8-kube-api-access-fx7mr\") pod \"network-check-target-9b9zs\" (UID: \"541ce094-3638-4979-bf7b-782f3df1cbf8\") " pod="openshift-network-diagnostics/network-check-target-9b9zs" Jul 09 22:10:11.166981 ip-10-0-136-251 kubenswrapper[2579]: E0709 22:10:11.166902 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 09 22:10:11.166981 ip-10-0-136-251 kubenswrapper[2579]: E0709 22:10:11.166945 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 09 22:10:11.166981 ip-10-0-136-251 kubenswrapper[2579]: E0709 22:10:11.166960 2579 projected.go:196] Error preparing data for projected volume kube-api-access-fx7mr for pod openshift-network-diagnostics/network-check-target-9b9zs: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 09 22:10:11.167153 ip-10-0-136-251 kubenswrapper[2579]: E0709 22:10:11.167025 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/541ce094-3638-4979-bf7b-782f3df1cbf8-kube-api-access-fx7mr podName:541ce094-3638-4979-bf7b-782f3df1cbf8 nodeName:}" failed. No retries permitted until 2026-07-09 22:10:12.167006222 +0000 UTC m=+4.078458907 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-fx7mr" (UniqueName: "kubernetes.io/projected/541ce094-3638-4979-bf7b-782f3df1cbf8-kube-api-access-fx7mr") pod "network-check-target-9b9zs" (UID: "541ce094-3638-4979-bf7b-782f3df1cbf8") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 09 22:10:11.488018 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:11.487929 2579 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-08 22:05:10 +0000 UTC" deadline="2028-03-09 18:38:03.161042194 +0000 UTC" Jul 09 22:10:11.488018 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:11.487964 2579 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14612h27m51.673082359s" Jul 09 22:10:11.574415 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:11.574376 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-669lr" event={"ID":"13602a06-27c2-446f-a5aa-64c917c285c8","Type":"ContainerStarted","Data":"828d553eed9904b8012dca02836c7cbc45d58bd57b768e943ff76c1d771bdf5b"} Jul 09 22:10:11.576883 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:11.576853 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-6t57q" event={"ID":"15cda8f6-b7fd-41f2-8aea-2ef4e9b9669b","Type":"ContainerStarted","Data":"cb3510693cb5be8c92097f8c8beb05a28878f825a0d11cdf5bf5f50dac534109"} Jul 09 22:10:11.580101 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:11.580074 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-l578c" event={"ID":"af0be020-5b8b-4828-bf98-531265c795b2","Type":"ContainerStarted","Data":"8ada4ff978f43d3ac370714d41e996b38cedd113a484f76c10f33306b2c52fa6"} Jul 09 22:10:11.581502 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:11.581444 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-gcmz7" event={"ID":"5946d6db-f1f5-4fee-827b-1427b1bdd90f","Type":"ContainerStarted","Data":"0f497d1a5b4cb0bfeb480255f10abcca759101c4a4fe62f97bc62c44dec1c0bb"} Jul 09 22:10:11.583149 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:11.583126 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-5fzf2" event={"ID":"499e80b4-0095-43da-ad29-f143e59a7357","Type":"ContainerStarted","Data":"e88a24efc49b9e5c54d4696d5979f4ed4f8541f0fca8a3aa4388ba8b727ce11a"} Jul 09 22:10:11.584528 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:11.584506 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-bzbzs" event={"ID":"50069e81-757a-482b-b7a2-ac26f2036ebd","Type":"ContainerStarted","Data":"12b11cb5c810328ea35d47f20e019514b8344db6b5c9dab086d2676e5edb2bee"} Jul 09 22:10:11.586142 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:11.586117 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-lvcvn" event={"ID":"84b44025-bd7d-4910-ba37-d294bf354258","Type":"ContainerStarted","Data":"a239012d8cf5835c0de4544e8f2b58200803b7b35149c2296fe21a555ded3b32"} Jul 09 22:10:11.587983 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:11.587944 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-r8dsp" event={"ID":"7d1ac8fd-5e7b-439c-b48c-44dcf09b2934","Type":"ContainerStarted","Data":"bdcb0b83c6f9cb6ccfd51a76fa03731578722cae69e6f81a26124fdbf63a9cd5"} Jul 09 22:10:11.589971 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:11.589909 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-nr4cq" event={"ID":"6807e2e6-d9a4-475a-9fb4-4b958127840f","Type":"ContainerStarted","Data":"71e324be567dd00252f7f7ca8fce930daf436b0ca7ca96201b149676de1fdf50"} Jul 09 22:10:12.081230 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:12.081180 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b5e569ee-1aea-4eeb-8235-93438e648102-original-pull-secret\") pod \"global-pull-secret-syncer-bwdgc\" (UID: \"b5e569ee-1aea-4eeb-8235-93438e648102\") " pod="kube-system/global-pull-secret-syncer-bwdgc" Jul 09 22:10:12.081424 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:12.081247 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/23b023ec-a254-462d-a0b4-5c22fd905e72-metrics-certs\") pod \"network-metrics-daemon-8pj95\" (UID: \"23b023ec-a254-462d-a0b4-5c22fd905e72\") " pod="openshift-multus/network-metrics-daemon-8pj95" Jul 09 22:10:12.081487 ip-10-0-136-251 kubenswrapper[2579]: E0709 22:10:12.081426 2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 09 22:10:12.081538 ip-10-0-136-251 kubenswrapper[2579]: E0709 22:10:12.081491 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/23b023ec-a254-462d-a0b4-5c22fd905e72-metrics-certs podName:23b023ec-a254-462d-a0b4-5c22fd905e72 nodeName:}" failed. No retries permitted until 2026-07-09 22:10:14.081470719 +0000 UTC m=+5.992923417 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/23b023ec-a254-462d-a0b4-5c22fd905e72-metrics-certs") pod "network-metrics-daemon-8pj95" (UID: "23b023ec-a254-462d-a0b4-5c22fd905e72") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 09 22:10:12.081978 ip-10-0-136-251 kubenswrapper[2579]: E0709 22:10:12.081955 2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 09 22:10:12.082057 ip-10-0-136-251 kubenswrapper[2579]: E0709 22:10:12.082018 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b5e569ee-1aea-4eeb-8235-93438e648102-original-pull-secret podName:b5e569ee-1aea-4eeb-8235-93438e648102 nodeName:}" failed. No retries permitted until 2026-07-09 22:10:14.081999069 +0000 UTC m=+5.993451751 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b5e569ee-1aea-4eeb-8235-93438e648102-original-pull-secret") pod "global-pull-secret-syncer-bwdgc" (UID: "b5e569ee-1aea-4eeb-8235-93438e648102") : object "kube-system"/"original-pull-secret" not registered Jul 09 22:10:12.182271 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:12.182233 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fx7mr\" (UniqueName: \"kubernetes.io/projected/541ce094-3638-4979-bf7b-782f3df1cbf8-kube-api-access-fx7mr\") pod \"network-check-target-9b9zs\" (UID: \"541ce094-3638-4979-bf7b-782f3df1cbf8\") " pod="openshift-network-diagnostics/network-check-target-9b9zs" Jul 09 22:10:12.182452 ip-10-0-136-251 kubenswrapper[2579]: E0709 22:10:12.182435 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 09 22:10:12.182515 ip-10-0-136-251 kubenswrapper[2579]: E0709 22:10:12.182458 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 09 22:10:12.182515 ip-10-0-136-251 kubenswrapper[2579]: E0709 22:10:12.182470 2579 projected.go:196] Error preparing data for projected volume kube-api-access-fx7mr for pod openshift-network-diagnostics/network-check-target-9b9zs: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 09 22:10:12.182614 ip-10-0-136-251 kubenswrapper[2579]: E0709 22:10:12.182527 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/541ce094-3638-4979-bf7b-782f3df1cbf8-kube-api-access-fx7mr podName:541ce094-3638-4979-bf7b-782f3df1cbf8 nodeName:}" failed. No retries permitted until 2026-07-09 22:10:14.182509353 +0000 UTC m=+6.093962037 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-fx7mr" (UniqueName: "kubernetes.io/projected/541ce094-3638-4979-bf7b-782f3df1cbf8-kube-api-access-fx7mr") pod "network-check-target-9b9zs" (UID: "541ce094-3638-4979-bf7b-782f3df1cbf8") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 09 22:10:12.488566 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:12.488443 2579 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-08 22:05:10 +0000 UTC" deadline="2028-03-05 20:15:41.423619698 +0000 UTC" Jul 09 22:10:12.488566 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:12.488510 2579 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14518h5m28.935114144s" Jul 09 22:10:12.560227 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:12.559748 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8pj95" Jul 09 22:10:12.560227 ip-10-0-136-251 kubenswrapper[2579]: E0709 22:10:12.559850 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8pj95" podUID="23b023ec-a254-462d-a0b4-5c22fd905e72" Jul 09 22:10:12.560227 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:12.559979 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9b9zs" Jul 09 22:10:12.560227 ip-10-0-136-251 kubenswrapper[2579]: E0709 22:10:12.560032 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9b9zs" podUID="541ce094-3638-4979-bf7b-782f3df1cbf8" Jul 09 22:10:12.560227 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:12.560105 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bwdgc" Jul 09 22:10:12.560227 ip-10-0-136-251 kubenswrapper[2579]: E0709 22:10:12.560220 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bwdgc" podUID="b5e569ee-1aea-4eeb-8235-93438e648102" Jul 09 22:10:13.618635 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:13.618355 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-251.ec2.internal" event={"ID":"50613fbded229913a0a9abdd811c3ca4","Type":"ContainerStarted","Data":"1910badec553af41c0a6853d0c47f8efe2359a2b61701a7542f881903c71fb15"} Jul 09 22:10:13.625234 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:13.625198 2579 generic.go:356] "Generic (PLEG): container finished" podID="021762e3fafa981d5236e3af644982e1" containerID="9e1bd0912196873eb53543c2e62bdb8a2f4ec31b20b92e12cac87c68c6cb8897" exitCode=0 Jul 09 22:10:13.625409 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:13.625259 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-251.ec2.internal" event={"ID":"021762e3fafa981d5236e3af644982e1","Type":"ContainerDied","Data":"9e1bd0912196873eb53543c2e62bdb8a2f4ec31b20b92e12cac87c68c6cb8897"} Jul 09 22:10:13.634589 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:13.634528 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-251.ec2.internal" podStartSLOduration=4.634507683 podStartE2EDuration="4.634507683s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 22:10:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 22:10:13.634007984 +0000 UTC m=+5.545460685" watchObservedRunningTime="2026-07-09 22:10:13.634507683 +0000 UTC m=+5.545960383" Jul 09 22:10:14.098679 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:14.098644 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b5e569ee-1aea-4eeb-8235-93438e648102-original-pull-secret\") pod \"global-pull-secret-syncer-bwdgc\" (UID: \"b5e569ee-1aea-4eeb-8235-93438e648102\") " pod="kube-system/global-pull-secret-syncer-bwdgc" Jul 09 22:10:14.098857 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:14.098698 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/23b023ec-a254-462d-a0b4-5c22fd905e72-metrics-certs\") pod \"network-metrics-daemon-8pj95\" (UID: \"23b023ec-a254-462d-a0b4-5c22fd905e72\") " pod="openshift-multus/network-metrics-daemon-8pj95" Jul 09 22:10:14.098857 ip-10-0-136-251 kubenswrapper[2579]: E0709 22:10:14.098826 2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 09 22:10:14.099010 ip-10-0-136-251 kubenswrapper[2579]: E0709 22:10:14.098876 2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 09 22:10:14.099010 ip-10-0-136-251 kubenswrapper[2579]: E0709 22:10:14.098889 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/23b023ec-a254-462d-a0b4-5c22fd905e72-metrics-certs podName:23b023ec-a254-462d-a0b4-5c22fd905e72 nodeName:}" failed. No retries permitted until 2026-07-09 22:10:18.098869665 +0000 UTC m=+10.010322356 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/23b023ec-a254-462d-a0b4-5c22fd905e72-metrics-certs") pod "network-metrics-daemon-8pj95" (UID: "23b023ec-a254-462d-a0b4-5c22fd905e72") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 09 22:10:14.099010 ip-10-0-136-251 kubenswrapper[2579]: E0709 22:10:14.098969 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b5e569ee-1aea-4eeb-8235-93438e648102-original-pull-secret podName:b5e569ee-1aea-4eeb-8235-93438e648102 nodeName:}" failed. No retries permitted until 2026-07-09 22:10:18.09895083 +0000 UTC m=+10.010403519 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b5e569ee-1aea-4eeb-8235-93438e648102-original-pull-secret") pod "global-pull-secret-syncer-bwdgc" (UID: "b5e569ee-1aea-4eeb-8235-93438e648102") : object "kube-system"/"original-pull-secret" not registered Jul 09 22:10:14.199903 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:14.199866 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fx7mr\" (UniqueName: \"kubernetes.io/projected/541ce094-3638-4979-bf7b-782f3df1cbf8-kube-api-access-fx7mr\") pod \"network-check-target-9b9zs\" (UID: \"541ce094-3638-4979-bf7b-782f3df1cbf8\") " pod="openshift-network-diagnostics/network-check-target-9b9zs" Jul 09 22:10:14.200127 ip-10-0-136-251 kubenswrapper[2579]: E0709 22:10:14.200110 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 09 22:10:14.200232 ip-10-0-136-251 kubenswrapper[2579]: E0709 22:10:14.200135 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 09 22:10:14.200232 ip-10-0-136-251 kubenswrapper[2579]: E0709 22:10:14.200152 2579 projected.go:196] Error preparing data for projected volume kube-api-access-fx7mr for pod openshift-network-diagnostics/network-check-target-9b9zs: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 09 22:10:14.200232 ip-10-0-136-251 kubenswrapper[2579]: E0709 22:10:14.200209 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/541ce094-3638-4979-bf7b-782f3df1cbf8-kube-api-access-fx7mr podName:541ce094-3638-4979-bf7b-782f3df1cbf8 nodeName:}" failed. No retries permitted until 2026-07-09 22:10:18.200190468 +0000 UTC m=+10.111643161 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-fx7mr" (UniqueName: "kubernetes.io/projected/541ce094-3638-4979-bf7b-782f3df1cbf8-kube-api-access-fx7mr") pod "network-check-target-9b9zs" (UID: "541ce094-3638-4979-bf7b-782f3df1cbf8") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 09 22:10:14.559832 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:14.559134 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9b9zs" Jul 09 22:10:14.559832 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:14.559167 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bwdgc" Jul 09 22:10:14.559832 ip-10-0-136-251 kubenswrapper[2579]: E0709 22:10:14.559266 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9b9zs" podUID="541ce094-3638-4979-bf7b-782f3df1cbf8" Jul 09 22:10:14.559832 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:14.559283 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8pj95" Jul 09 22:10:14.559832 ip-10-0-136-251 kubenswrapper[2579]: E0709 22:10:14.559365 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8pj95" podUID="23b023ec-a254-462d-a0b4-5c22fd905e72" Jul 09 22:10:14.559832 ip-10-0-136-251 kubenswrapper[2579]: E0709 22:10:14.559436 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bwdgc" podUID="b5e569ee-1aea-4eeb-8235-93438e648102" Jul 09 22:10:16.559063 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:16.558982 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bwdgc" Jul 09 22:10:16.559063 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:16.559007 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8pj95" Jul 09 22:10:16.559612 ip-10-0-136-251 kubenswrapper[2579]: E0709 22:10:16.559124 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bwdgc" podUID="b5e569ee-1aea-4eeb-8235-93438e648102" Jul 09 22:10:16.559612 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:16.558982 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9b9zs" Jul 09 22:10:16.559612 ip-10-0-136-251 kubenswrapper[2579]: E0709 22:10:16.559541 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9b9zs" podUID="541ce094-3638-4979-bf7b-782f3df1cbf8" Jul 09 22:10:16.559742 ip-10-0-136-251 kubenswrapper[2579]: E0709 22:10:16.559632 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8pj95" podUID="23b023ec-a254-462d-a0b4-5c22fd905e72" Jul 09 22:10:18.135726 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:18.135685 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b5e569ee-1aea-4eeb-8235-93438e648102-original-pull-secret\") pod \"global-pull-secret-syncer-bwdgc\" (UID: \"b5e569ee-1aea-4eeb-8235-93438e648102\") " pod="kube-system/global-pull-secret-syncer-bwdgc" Jul 09 22:10:18.136215 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:18.135740 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/23b023ec-a254-462d-a0b4-5c22fd905e72-metrics-certs\") pod \"network-metrics-daemon-8pj95\" (UID: \"23b023ec-a254-462d-a0b4-5c22fd905e72\") " pod="openshift-multus/network-metrics-daemon-8pj95" Jul 09 22:10:18.136215 ip-10-0-136-251 kubenswrapper[2579]: E0709 22:10:18.135822 2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 09 22:10:18.136215 ip-10-0-136-251 kubenswrapper[2579]: E0709 22:10:18.135864 2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 09 22:10:18.136215 ip-10-0-136-251 kubenswrapper[2579]: E0709 22:10:18.135903 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b5e569ee-1aea-4eeb-8235-93438e648102-original-pull-secret podName:b5e569ee-1aea-4eeb-8235-93438e648102 nodeName:}" failed. No retries permitted until 2026-07-09 22:10:26.135879317 +0000 UTC m=+18.047332012 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b5e569ee-1aea-4eeb-8235-93438e648102-original-pull-secret") pod "global-pull-secret-syncer-bwdgc" (UID: "b5e569ee-1aea-4eeb-8235-93438e648102") : object "kube-system"/"original-pull-secret" not registered Jul 09 22:10:18.136215 ip-10-0-136-251 kubenswrapper[2579]: E0709 22:10:18.135943 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/23b023ec-a254-462d-a0b4-5c22fd905e72-metrics-certs podName:23b023ec-a254-462d-a0b4-5c22fd905e72 nodeName:}" failed. No retries permitted until 2026-07-09 22:10:26.135914595 +0000 UTC m=+18.047367273 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/23b023ec-a254-462d-a0b4-5c22fd905e72-metrics-certs") pod "network-metrics-daemon-8pj95" (UID: "23b023ec-a254-462d-a0b4-5c22fd905e72") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 09 22:10:18.236859 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:18.236710 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fx7mr\" (UniqueName: \"kubernetes.io/projected/541ce094-3638-4979-bf7b-782f3df1cbf8-kube-api-access-fx7mr\") pod \"network-check-target-9b9zs\" (UID: \"541ce094-3638-4979-bf7b-782f3df1cbf8\") " pod="openshift-network-diagnostics/network-check-target-9b9zs" Jul 09 22:10:18.237053 ip-10-0-136-251 kubenswrapper[2579]: E0709 22:10:18.236958 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 09 22:10:18.237053 ip-10-0-136-251 kubenswrapper[2579]: E0709 22:10:18.236982 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 09 22:10:18.237053 ip-10-0-136-251 kubenswrapper[2579]: E0709 22:10:18.237001 2579 projected.go:196] Error preparing data for projected volume kube-api-access-fx7mr for pod openshift-network-diagnostics/network-check-target-9b9zs: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 09 22:10:18.237189 ip-10-0-136-251 kubenswrapper[2579]: E0709 22:10:18.237069 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/541ce094-3638-4979-bf7b-782f3df1cbf8-kube-api-access-fx7mr podName:541ce094-3638-4979-bf7b-782f3df1cbf8 nodeName:}" failed. No retries permitted until 2026-07-09 22:10:26.237047605 +0000 UTC m=+18.148500318 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-fx7mr" (UniqueName: "kubernetes.io/projected/541ce094-3638-4979-bf7b-782f3df1cbf8-kube-api-access-fx7mr") pod "network-check-target-9b9zs" (UID: "541ce094-3638-4979-bf7b-782f3df1cbf8") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 09 22:10:18.558987 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:18.558894 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bwdgc" Jul 09 22:10:18.559154 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:18.559019 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8pj95" Jul 09 22:10:18.559154 ip-10-0-136-251 kubenswrapper[2579]: E0709 22:10:18.559026 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bwdgc" podUID="b5e569ee-1aea-4eeb-8235-93438e648102" Jul 09 22:10:18.559229 ip-10-0-136-251 kubenswrapper[2579]: E0709 22:10:18.559150 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8pj95" podUID="23b023ec-a254-462d-a0b4-5c22fd905e72" Jul 09 22:10:18.559229 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:18.559198 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9b9zs" Jul 09 22:10:18.559287 ip-10-0-136-251 kubenswrapper[2579]: E0709 22:10:18.559251 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9b9zs" podUID="541ce094-3638-4979-bf7b-782f3df1cbf8" Jul 09 22:10:20.558713 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:20.558664 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8pj95" Jul 09 22:10:20.559180 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:20.558674 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9b9zs" Jul 09 22:10:20.559180 ip-10-0-136-251 kubenswrapper[2579]: E0709 22:10:20.558795 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8pj95" podUID="23b023ec-a254-462d-a0b4-5c22fd905e72" Jul 09 22:10:20.559180 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:20.558828 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bwdgc" Jul 09 22:10:20.559180 ip-10-0-136-251 kubenswrapper[2579]: E0709 22:10:20.558908 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9b9zs" podUID="541ce094-3638-4979-bf7b-782f3df1cbf8" Jul 09 22:10:20.559180 ip-10-0-136-251 kubenswrapper[2579]: E0709 22:10:20.559011 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bwdgc" podUID="b5e569ee-1aea-4eeb-8235-93438e648102" Jul 09 22:10:21.673119 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:21.673082 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-251.ec2.internal" event={"ID":"021762e3fafa981d5236e3af644982e1","Type":"ContainerStarted","Data":"fbf1246e60d3aee77bca8e38496b0aadecbbf48ace1e5bbe1af99e3fa3819254"} Jul 09 22:10:22.558774 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:22.558668 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9b9zs" Jul 09 22:10:22.558774 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:22.558668 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bwdgc" Jul 09 22:10:22.559007 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:22.558797 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8pj95" Jul 09 22:10:22.559007 ip-10-0-136-251 kubenswrapper[2579]: E0709 22:10:22.558791 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9b9zs" podUID="541ce094-3638-4979-bf7b-782f3df1cbf8" Jul 09 22:10:22.559007 ip-10-0-136-251 kubenswrapper[2579]: E0709 22:10:22.558886 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bwdgc" podUID="b5e569ee-1aea-4eeb-8235-93438e648102" Jul 09 22:10:22.559007 ip-10-0-136-251 kubenswrapper[2579]: E0709 22:10:22.558984 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8pj95" podUID="23b023ec-a254-462d-a0b4-5c22fd905e72" Jul 09 22:10:22.678045 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:22.677600 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-nr4cq" event={"ID":"6807e2e6-d9a4-475a-9fb4-4b958127840f","Type":"ContainerStarted","Data":"f670b143ee4dc349cf50d3de179a4376f691cd7d367770f8d485095311977fbf"} Jul 09 22:10:22.679368 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:22.679305 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-669lr" event={"ID":"13602a06-27c2-446f-a5aa-64c917c285c8","Type":"ContainerStarted","Data":"f2b7f41d432de6d902edafe34abb5b987a3a1bdc14908eaac46876b31eda23e1"} Jul 09 22:10:22.681036 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:22.681010 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-l578c" event={"ID":"af0be020-5b8b-4828-bf98-531265c795b2","Type":"ContainerStarted","Data":"3ee46777335a674249fbd82b4f9ed44d190b90f6b0420ec5e477b5a675818545"} Jul 09 22:10:22.682747 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:22.682717 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-gcmz7" event={"ID":"5946d6db-f1f5-4fee-827b-1427b1bdd90f","Type":"ContainerStarted","Data":"10d058f13ef946de10008a54d7c01f7528c29abf3e63e9e1306adeb024daea30"} Jul 09 22:10:22.684281 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:22.684256 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-5fzf2" event={"ID":"499e80b4-0095-43da-ad29-f143e59a7357","Type":"ContainerStarted","Data":"edd8954873c426d84d8e28cd66d377c611526ea0dc9cead814d3515dda5d0164"} Jul 09 22:10:22.686126 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:22.686077 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-bzbzs" event={"ID":"50069e81-757a-482b-b7a2-ac26f2036ebd","Type":"ContainerStarted","Data":"bd5958a8117d5b55ec17a57485c8e1e9189850e6f74c2f6eead57587725c561d"} Jul 09 22:10:22.692986 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:22.692943 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-nr4cq" podStartSLOduration=4.102709313 podStartE2EDuration="14.69291381s" totalImagesPullingTime="10.590204497s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 22:10:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 22:10:22.692424052 +0000 UTC m=+14.603876776" watchObservedRunningTime="2026-07-09 22:10:22.69291381 +0000 UTC m=+14.604366509" Jul 09 22:10:22.693083 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:22.693029 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-251.ec2.internal" podStartSLOduration=13.69302293 podStartE2EDuration="13.69302293s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 22:10:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 22:10:21.68970149 +0000 UTC m=+13.601154210" watchObservedRunningTime="2026-07-09 22:10:22.69302293 +0000 UTC m=+14.604475633" Jul 09 22:10:22.707204 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:22.707155 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-gcmz7" podStartSLOduration=4.159169439 podStartE2EDuration="14.707142656s" totalImagesPullingTime="10.547973217s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 22:10:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 22:10:22.706795876 +0000 UTC m=+14.618248576" watchObservedRunningTime="2026-07-09 22:10:22.707142656 +0000 UTC m=+14.618595368" Jul 09 22:10:22.721658 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:22.721613 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-l578c" podStartSLOduration=4.13701426 podStartE2EDuration="14.721595068s" totalImagesPullingTime="10.584580808s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 22:10:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 22:10:22.720661805 +0000 UTC m=+14.632114540" watchObservedRunningTime="2026-07-09 22:10:22.721595068 +0000 UTC m=+14.633047769" Jul 09 22:10:23.691038 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:23.691004 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-6t57q" event={"ID":"15cda8f6-b7fd-41f2-8aea-2ef4e9b9669b","Type":"ContainerStarted","Data":"53843d4ed85393b0284123ae4f52816ffb12455cfdc6e57002996a1ecd813a3f"} Jul 09 22:10:23.706743 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:23.706444 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-6t57q" podStartSLOduration=5.122908109 podStartE2EDuration="15.706424578s" totalImagesPullingTime="10.583516469s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 22:10:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 22:10:23.70641104 +0000 UTC m=+15.617863740" watchObservedRunningTime="2026-07-09 22:10:23.706424578 +0000 UTC m=+15.617877279" Jul 09 22:10:23.707132 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:23.707100 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-bzbzs" podStartSLOduration=4.901721392 podStartE2EDuration="15.70709067s" totalImagesPullingTime="10.805369278s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 22:10:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 22:10:22.766088015 +0000 UTC m=+14.677540738" watchObservedRunningTime="2026-07-09 22:10:23.70709067 +0000 UTC m=+15.618543414" Jul 09 22:10:24.558719 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:24.558681 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bwdgc" Jul 09 22:10:24.558887 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:24.558729 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8pj95" Jul 09 22:10:24.558887 ip-10-0-136-251 kubenswrapper[2579]: E0709 22:10:24.558835 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bwdgc" podUID="b5e569ee-1aea-4eeb-8235-93438e648102" Jul 09 22:10:24.559024 ip-10-0-136-251 kubenswrapper[2579]: E0709 22:10:24.558902 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8pj95" podUID="23b023ec-a254-462d-a0b4-5c22fd905e72" Jul 09 22:10:24.559024 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:24.558931 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9b9zs" Jul 09 22:10:24.559122 ip-10-0-136-251 kubenswrapper[2579]: E0709 22:10:24.559037 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9b9zs" podUID="541ce094-3638-4979-bf7b-782f3df1cbf8" Jul 09 22:10:26.200558 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:26.200509 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b5e569ee-1aea-4eeb-8235-93438e648102-original-pull-secret\") pod \"global-pull-secret-syncer-bwdgc\" (UID: \"b5e569ee-1aea-4eeb-8235-93438e648102\") " pod="kube-system/global-pull-secret-syncer-bwdgc" Jul 09 22:10:26.201443 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:26.200573 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/23b023ec-a254-462d-a0b4-5c22fd905e72-metrics-certs\") pod \"network-metrics-daemon-8pj95\" (UID: \"23b023ec-a254-462d-a0b4-5c22fd905e72\") " pod="openshift-multus/network-metrics-daemon-8pj95" Jul 09 22:10:26.201443 ip-10-0-136-251 kubenswrapper[2579]: E0709 22:10:26.200676 2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 09 22:10:26.201443 ip-10-0-136-251 kubenswrapper[2579]: E0709 22:10:26.200679 2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 09 22:10:26.201443 ip-10-0-136-251 kubenswrapper[2579]: E0709 22:10:26.200729 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/23b023ec-a254-462d-a0b4-5c22fd905e72-metrics-certs podName:23b023ec-a254-462d-a0b4-5c22fd905e72 nodeName:}" failed. No retries permitted until 2026-07-09 22:10:42.200714694 +0000 UTC m=+34.112167377 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/23b023ec-a254-462d-a0b4-5c22fd905e72-metrics-certs") pod "network-metrics-daemon-8pj95" (UID: "23b023ec-a254-462d-a0b4-5c22fd905e72") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 09 22:10:26.201443 ip-10-0-136-251 kubenswrapper[2579]: E0709 22:10:26.200742 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b5e569ee-1aea-4eeb-8235-93438e648102-original-pull-secret podName:b5e569ee-1aea-4eeb-8235-93438e648102 nodeName:}" failed. No retries permitted until 2026-07-09 22:10:42.200736413 +0000 UTC m=+34.112189091 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b5e569ee-1aea-4eeb-8235-93438e648102-original-pull-secret") pod "global-pull-secret-syncer-bwdgc" (UID: "b5e569ee-1aea-4eeb-8235-93438e648102") : object "kube-system"/"original-pull-secret" not registered Jul 09 22:10:26.301855 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:26.301823 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fx7mr\" (UniqueName: \"kubernetes.io/projected/541ce094-3638-4979-bf7b-782f3df1cbf8-kube-api-access-fx7mr\") pod \"network-check-target-9b9zs\" (UID: \"541ce094-3638-4979-bf7b-782f3df1cbf8\") " pod="openshift-network-diagnostics/network-check-target-9b9zs" Jul 09 22:10:26.302030 ip-10-0-136-251 kubenswrapper[2579]: E0709 22:10:26.302008 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 09 22:10:26.302077 ip-10-0-136-251 kubenswrapper[2579]: E0709 22:10:26.302039 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 09 22:10:26.302077 ip-10-0-136-251 kubenswrapper[2579]: E0709 22:10:26.302054 2579 projected.go:196] Error preparing data for projected volume kube-api-access-fx7mr for pod openshift-network-diagnostics/network-check-target-9b9zs: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 09 22:10:26.302157 ip-10-0-136-251 kubenswrapper[2579]: E0709 22:10:26.302122 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/541ce094-3638-4979-bf7b-782f3df1cbf8-kube-api-access-fx7mr podName:541ce094-3638-4979-bf7b-782f3df1cbf8 nodeName:}" failed. No retries permitted until 2026-07-09 22:10:42.302107951 +0000 UTC m=+34.213560633 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-fx7mr" (UniqueName: "kubernetes.io/projected/541ce094-3638-4979-bf7b-782f3df1cbf8-kube-api-access-fx7mr") pod "network-check-target-9b9zs" (UID: "541ce094-3638-4979-bf7b-782f3df1cbf8") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 09 22:10:26.558252 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:26.558214 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8pj95" Jul 09 22:10:26.558252 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:26.558249 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bwdgc" Jul 09 22:10:26.558456 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:26.558214 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9b9zs" Jul 09 22:10:26.558456 ip-10-0-136-251 kubenswrapper[2579]: E0709 22:10:26.558343 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8pj95" podUID="23b023ec-a254-462d-a0b4-5c22fd905e72" Jul 09 22:10:26.558556 ip-10-0-136-251 kubenswrapper[2579]: E0709 22:10:26.558451 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bwdgc" podUID="b5e569ee-1aea-4eeb-8235-93438e648102" Jul 09 22:10:26.558556 ip-10-0-136-251 kubenswrapper[2579]: E0709 22:10:26.558531 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9b9zs" podUID="541ce094-3638-4979-bf7b-782f3df1cbf8" Jul 09 22:10:26.698192 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:26.698154 2579 generic.go:356] "Generic (PLEG): container finished" podID="499e80b4-0095-43da-ad29-f143e59a7357" containerID="edd8954873c426d84d8e28cd66d377c611526ea0dc9cead814d3515dda5d0164" exitCode=0 Jul 09 22:10:26.698389 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:26.698213 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-5fzf2" event={"ID":"499e80b4-0095-43da-ad29-f143e59a7357","Type":"ContainerDied","Data":"edd8954873c426d84d8e28cd66d377c611526ea0dc9cead814d3515dda5d0164"} Jul 09 22:10:27.397663 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:27.397625 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-gcmz7" Jul 09 22:10:27.398404 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:27.398287 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-gcmz7" Jul 09 22:10:28.559280 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:28.559244 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9b9zs" Jul 09 22:10:28.559746 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:28.559324 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8pj95" Jul 09 22:10:28.559746 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:28.559363 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bwdgc" Jul 09 22:10:28.559746 ip-10-0-136-251 kubenswrapper[2579]: E0709 22:10:28.559383 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9b9zs" podUID="541ce094-3638-4979-bf7b-782f3df1cbf8" Jul 09 22:10:28.559746 ip-10-0-136-251 kubenswrapper[2579]: E0709 22:10:28.559435 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bwdgc" podUID="b5e569ee-1aea-4eeb-8235-93438e648102" Jul 09 22:10:28.559746 ip-10-0-136-251 kubenswrapper[2579]: E0709 22:10:28.559527 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8pj95" podUID="23b023ec-a254-462d-a0b4-5c22fd905e72" Jul 09 22:10:30.558853 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:30.558816 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9b9zs" Jul 09 22:10:30.559341 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:30.558823 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bwdgc" Jul 09 22:10:30.559341 ip-10-0-136-251 kubenswrapper[2579]: E0709 22:10:30.558971 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9b9zs" podUID="541ce094-3638-4979-bf7b-782f3df1cbf8" Jul 09 22:10:30.559341 ip-10-0-136-251 kubenswrapper[2579]: E0709 22:10:30.559065 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bwdgc" podUID="b5e569ee-1aea-4eeb-8235-93438e648102" Jul 09 22:10:30.559341 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:30.558824 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8pj95" Jul 09 22:10:30.559341 ip-10-0-136-251 kubenswrapper[2579]: E0709 22:10:30.559179 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8pj95" podUID="23b023ec-a254-462d-a0b4-5c22fd905e72" Jul 09 22:10:30.761188 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:30.761013 2579 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Jul 09 22:10:31.525581 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:31.525327 2579 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-09T22:10:30.761185634Z","UUID":"4da51aae-cb7a-45ae-a841-c727974bf0d3","Handler":null,"Name":"","Endpoint":""} Jul 09 22:10:31.526870 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:31.526847 2579 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Jul 09 22:10:31.527000 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:31.526882 2579 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Jul 09 22:10:31.711803 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:31.711771 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-lvcvn" event={"ID":"84b44025-bd7d-4910-ba37-d294bf354258","Type":"ContainerStarted","Data":"52d64e6032becc289d50ed29d035f6fc9f42b496a5d16c102663a7c0af3febb6"} Jul 09 22:10:31.714825 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:31.714794 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-r8dsp" event={"ID":"7d1ac8fd-5e7b-439c-b48c-44dcf09b2934","Type":"ContainerStarted","Data":"8e5f136618ad8adf4cbb11f8c7d57abcd7f108c8ad44bd9a7a636c0a7e58c8fc"} Jul 09 22:10:31.714973 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:31.714854 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-r8dsp" event={"ID":"7d1ac8fd-5e7b-439c-b48c-44dcf09b2934","Type":"ContainerStarted","Data":"c1fa4345dd0de083bdaf464e9df89c108dd186a63ecc3d404f442b4eba94dce4"} Jul 09 22:10:31.714973 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:31.714885 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-r8dsp" event={"ID":"7d1ac8fd-5e7b-439c-b48c-44dcf09b2934","Type":"ContainerStarted","Data":"1d14d4808230bf19dcaa55ff338a1cc3288e83b8ce27ba428b24f0a4ac423d80"} Jul 09 22:10:31.714973 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:31.714912 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-r8dsp" event={"ID":"7d1ac8fd-5e7b-439c-b48c-44dcf09b2934","Type":"ContainerStarted","Data":"e418b5997125add771917071e8b399fd69a9e803c70ae08d49fb85936ca5a708"} Jul 09 22:10:31.714973 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:31.714963 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-r8dsp" event={"ID":"7d1ac8fd-5e7b-439c-b48c-44dcf09b2934","Type":"ContainerStarted","Data":"4ce148139f8e8302ae2934e23f5c0e59d94c72789847edbf466da3c62f83411d"} Jul 09 22:10:31.715169 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:31.715024 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-r8dsp" event={"ID":"7d1ac8fd-5e7b-439c-b48c-44dcf09b2934","Type":"ContainerStarted","Data":"877f525ba4d8ea0bac49f207143c641293ee2ec98846f4f35f012c848ed43f31"} Jul 09 22:10:31.716456 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:31.716435 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-669lr" event={"ID":"13602a06-27c2-446f-a5aa-64c917c285c8","Type":"ContainerStarted","Data":"94652e56130f545810f7d7e9d52d9bca0df47cfb010128f2f17c123aa51f8bcb"} Jul 09 22:10:31.728714 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:31.728671 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-lvcvn" podStartSLOduration=3.918635349 podStartE2EDuration="23.728659437s" totalImagesPullingTime="19.810024088s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 22:10:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 22:10:31.728050326 +0000 UTC m=+23.639503025" watchObservedRunningTime="2026-07-09 22:10:31.728659437 +0000 UTC m=+23.640112137" Jul 09 22:10:32.562332 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:32.562298 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9b9zs" Jul 09 22:10:32.562332 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:32.562332 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bwdgc" Jul 09 22:10:32.562615 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:32.562298 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8pj95" Jul 09 22:10:32.562615 ip-10-0-136-251 kubenswrapper[2579]: E0709 22:10:32.562404 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9b9zs" podUID="541ce094-3638-4979-bf7b-782f3df1cbf8" Jul 09 22:10:32.562615 ip-10-0-136-251 kubenswrapper[2579]: E0709 22:10:32.562491 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8pj95" podUID="23b023ec-a254-462d-a0b4-5c22fd905e72" Jul 09 22:10:32.562615 ip-10-0-136-251 kubenswrapper[2579]: E0709 22:10:32.562557 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bwdgc" podUID="b5e569ee-1aea-4eeb-8235-93438e648102" Jul 09 22:10:32.719896 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:32.719852 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-669lr" event={"ID":"13602a06-27c2-446f-a5aa-64c917c285c8","Type":"ContainerStarted","Data":"232c50eca0b8d25676b8592831d406d4bd921d0bf929801b44fa56c9921c9cbc"} Jul 09 22:10:32.738383 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:32.738338 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-669lr" podStartSLOduration=4.040874941 podStartE2EDuration="24.73832125s" totalImagesPullingTime="20.697446309s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 22:10:08 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 22:10:32.737419641 +0000 UTC m=+24.648872341" watchObservedRunningTime="2026-07-09 22:10:32.73832125 +0000 UTC m=+24.649773967" Jul 09 22:10:33.726622 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:33.726580 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-r8dsp" event={"ID":"7d1ac8fd-5e7b-439c-b48c-44dcf09b2934","Type":"ContainerStarted","Data":"6b28dc7fa682d8e3b28152f6aacf1bd50fa582ac278f867dc8dde8f669afe16b"} Jul 09 22:10:34.561747 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:34.561718 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bwdgc" Jul 09 22:10:34.561930 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:34.561813 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8pj95" Jul 09 22:10:34.561930 ip-10-0-136-251 kubenswrapper[2579]: E0709 22:10:34.561825 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bwdgc" podUID="b5e569ee-1aea-4eeb-8235-93438e648102" Jul 09 22:10:34.561930 ip-10-0-136-251 kubenswrapper[2579]: E0709 22:10:34.561893 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8pj95" podUID="23b023ec-a254-462d-a0b4-5c22fd905e72" Jul 09 22:10:34.561930 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:34.561904 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9b9zs" Jul 09 22:10:34.562144 ip-10-0-136-251 kubenswrapper[2579]: E0709 22:10:34.562017 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9b9zs" podUID="541ce094-3638-4979-bf7b-782f3df1cbf8" Jul 09 22:10:34.706495 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:34.706454 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-gcmz7" Jul 09 22:10:34.706668 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:34.706608 2579 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness" Jul 09 22:10:34.707237 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:34.707217 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-gcmz7" Jul 09 22:10:36.561462 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:36.561265 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bwdgc" Jul 09 22:10:36.562023 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:36.561312 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8pj95" Jul 09 22:10:36.562023 ip-10-0-136-251 kubenswrapper[2579]: E0709 22:10:36.561548 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bwdgc" podUID="b5e569ee-1aea-4eeb-8235-93438e648102" Jul 09 22:10:36.562023 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:36.561324 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9b9zs" Jul 09 22:10:36.562023 ip-10-0-136-251 kubenswrapper[2579]: E0709 22:10:36.561672 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8pj95" podUID="23b023ec-a254-462d-a0b4-5c22fd905e72" Jul 09 22:10:36.562023 ip-10-0-136-251 kubenswrapper[2579]: E0709 22:10:36.561727 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9b9zs" podUID="541ce094-3638-4979-bf7b-782f3df1cbf8" Jul 09 22:10:36.735661 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:36.735584 2579 generic.go:356] "Generic (PLEG): container finished" podID="499e80b4-0095-43da-ad29-f143e59a7357" containerID="c7aa8f078e73c626908d4a7b3ff21639a8efd3acd62398fe572f449fae515e56" exitCode=0 Jul 09 22:10:36.735807 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:36.735655 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-5fzf2" event={"ID":"499e80b4-0095-43da-ad29-f143e59a7357","Type":"ContainerDied","Data":"c7aa8f078e73c626908d4a7b3ff21639a8efd3acd62398fe572f449fae515e56"} Jul 09 22:10:36.739447 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:36.739425 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-r8dsp" event={"ID":"7d1ac8fd-5e7b-439c-b48c-44dcf09b2934","Type":"ContainerStarted","Data":"edc2279148e53ba18295847664b71b81b896e7d6e4d4fef4b9d6672de3480fa7"} Jul 09 22:10:36.739702 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:36.739681 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-r8dsp" Jul 09 22:10:36.739778 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:36.739739 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-r8dsp" Jul 09 22:10:36.756024 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:36.755993 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-r8dsp" Jul 09 22:10:36.785126 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:36.785084 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-r8dsp" podStartSLOduration=9.001196161 podStartE2EDuration="28.785069319s" totalImagesPullingTime="19.783873158s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 22:10:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 22:10:36.784660669 +0000 UTC m=+28.696113392" watchObservedRunningTime="2026-07-09 22:10:36.785069319 +0000 UTC m=+28.696522018" Jul 09 22:10:37.742674 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:37.742642 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-r8dsp" Jul 09 22:10:37.757745 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:37.757723 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-r8dsp" Jul 09 22:10:38.219932 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:38.219893 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-bwdgc"] Jul 09 22:10:38.220093 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:38.220020 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bwdgc" Jul 09 22:10:38.220147 ip-10-0-136-251 kubenswrapper[2579]: E0709 22:10:38.220104 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bwdgc" podUID="b5e569ee-1aea-4eeb-8235-93438e648102" Jul 09 22:10:38.224136 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:38.224109 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-8pj95"] Jul 09 22:10:38.224251 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:38.224230 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8pj95" Jul 09 22:10:38.224361 ip-10-0-136-251 kubenswrapper[2579]: E0709 22:10:38.224340 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8pj95" podUID="23b023ec-a254-462d-a0b4-5c22fd905e72" Jul 09 22:10:38.224722 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:38.224696 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-9b9zs"] Jul 09 22:10:38.224854 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:38.224793 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9b9zs" Jul 09 22:10:38.224942 ip-10-0-136-251 kubenswrapper[2579]: E0709 22:10:38.224878 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9b9zs" podUID="541ce094-3638-4979-bf7b-782f3df1cbf8" Jul 09 22:10:38.746863 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:38.746828 2579 generic.go:356] "Generic (PLEG): container finished" podID="499e80b4-0095-43da-ad29-f143e59a7357" containerID="b956a43a6327d4bc0510d7ab8f8b304075378e5d0dd4e04db914bf0327d6a6e2" exitCode=0 Jul 09 22:10:38.747222 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:38.746897 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-5fzf2" event={"ID":"499e80b4-0095-43da-ad29-f143e59a7357","Type":"ContainerDied","Data":"b956a43a6327d4bc0510d7ab8f8b304075378e5d0dd4e04db914bf0327d6a6e2"} Jul 09 22:10:39.558932 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:39.558894 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9b9zs" Jul 09 22:10:39.559055 ip-10-0-136-251 kubenswrapper[2579]: E0709 22:10:39.559009 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9b9zs" podUID="541ce094-3638-4979-bf7b-782f3df1cbf8" Jul 09 22:10:39.751957 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:39.751739 2579 generic.go:356] "Generic (PLEG): container finished" podID="499e80b4-0095-43da-ad29-f143e59a7357" containerID="aa6a2b1bbd07444c11bccfcc2941123382e14d5601b6e3e43755899e7f1cdfba" exitCode=0 Jul 09 22:10:39.752283 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:39.751814 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-5fzf2" event={"ID":"499e80b4-0095-43da-ad29-f143e59a7357","Type":"ContainerDied","Data":"aa6a2b1bbd07444c11bccfcc2941123382e14d5601b6e3e43755899e7f1cdfba"} Jul 09 22:10:40.558324 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:40.558285 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8pj95" Jul 09 22:10:40.558490 ip-10-0-136-251 kubenswrapper[2579]: E0709 22:10:40.558436 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8pj95" podUID="23b023ec-a254-462d-a0b4-5c22fd905e72" Jul 09 22:10:40.558553 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:40.558492 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bwdgc" Jul 09 22:10:40.558622 ip-10-0-136-251 kubenswrapper[2579]: E0709 22:10:40.558598 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bwdgc" podUID="b5e569ee-1aea-4eeb-8235-93438e648102" Jul 09 22:10:41.559047 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:41.559012 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9b9zs" Jul 09 22:10:41.559529 ip-10-0-136-251 kubenswrapper[2579]: E0709 22:10:41.559128 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9b9zs" podUID="541ce094-3638-4979-bf7b-782f3df1cbf8" Jul 09 22:10:42.226674 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:42.226637 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b5e569ee-1aea-4eeb-8235-93438e648102-original-pull-secret\") pod \"global-pull-secret-syncer-bwdgc\" (UID: \"b5e569ee-1aea-4eeb-8235-93438e648102\") " pod="kube-system/global-pull-secret-syncer-bwdgc" Jul 09 22:10:42.226674 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:42.226679 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/23b023ec-a254-462d-a0b4-5c22fd905e72-metrics-certs\") pod \"network-metrics-daemon-8pj95\" (UID: \"23b023ec-a254-462d-a0b4-5c22fd905e72\") " pod="openshift-multus/network-metrics-daemon-8pj95" Jul 09 22:10:42.227030 ip-10-0-136-251 kubenswrapper[2579]: E0709 22:10:42.226777 2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 09 22:10:42.227030 ip-10-0-136-251 kubenswrapper[2579]: E0709 22:10:42.226790 2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 09 22:10:42.227030 ip-10-0-136-251 kubenswrapper[2579]: E0709 22:10:42.226826 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/23b023ec-a254-462d-a0b4-5c22fd905e72-metrics-certs podName:23b023ec-a254-462d-a0b4-5c22fd905e72 nodeName:}" failed. No retries permitted until 2026-07-09 22:11:14.226812174 +0000 UTC m=+66.138264852 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/23b023ec-a254-462d-a0b4-5c22fd905e72-metrics-certs") pod "network-metrics-daemon-8pj95" (UID: "23b023ec-a254-462d-a0b4-5c22fd905e72") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 09 22:10:42.227030 ip-10-0-136-251 kubenswrapper[2579]: E0709 22:10:42.226853 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b5e569ee-1aea-4eeb-8235-93438e648102-original-pull-secret podName:b5e569ee-1aea-4eeb-8235-93438e648102 nodeName:}" failed. No retries permitted until 2026-07-09 22:11:14.226834646 +0000 UTC m=+66.138287338 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b5e569ee-1aea-4eeb-8235-93438e648102-original-pull-secret") pod "global-pull-secret-syncer-bwdgc" (UID: "b5e569ee-1aea-4eeb-8235-93438e648102") : object "kube-system"/"original-pull-secret" not registered Jul 09 22:10:42.327813 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:42.327781 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fx7mr\" (UniqueName: \"kubernetes.io/projected/541ce094-3638-4979-bf7b-782f3df1cbf8-kube-api-access-fx7mr\") pod \"network-check-target-9b9zs\" (UID: \"541ce094-3638-4979-bf7b-782f3df1cbf8\") " pod="openshift-network-diagnostics/network-check-target-9b9zs" Jul 09 22:10:42.327990 ip-10-0-136-251 kubenswrapper[2579]: E0709 22:10:42.327901 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 09 22:10:42.327990 ip-10-0-136-251 kubenswrapper[2579]: E0709 22:10:42.327930 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 09 22:10:42.327990 ip-10-0-136-251 kubenswrapper[2579]: E0709 22:10:42.327940 2579 projected.go:196] Error preparing data for projected volume kube-api-access-fx7mr for pod openshift-network-diagnostics/network-check-target-9b9zs: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 09 22:10:42.327990 ip-10-0-136-251 kubenswrapper[2579]: E0709 22:10:42.327988 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/541ce094-3638-4979-bf7b-782f3df1cbf8-kube-api-access-fx7mr podName:541ce094-3638-4979-bf7b-782f3df1cbf8 nodeName:}" failed. No retries permitted until 2026-07-09 22:11:14.327976573 +0000 UTC m=+66.239429251 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-fx7mr" (UniqueName: "kubernetes.io/projected/541ce094-3638-4979-bf7b-782f3df1cbf8-kube-api-access-fx7mr") pod "network-check-target-9b9zs" (UID: "541ce094-3638-4979-bf7b-782f3df1cbf8") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 09 22:10:42.558632 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:42.558598 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8pj95" Jul 09 22:10:42.558830 ip-10-0-136-251 kubenswrapper[2579]: E0709 22:10:42.558752 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8pj95" podUID="23b023ec-a254-462d-a0b4-5c22fd905e72" Jul 09 22:10:42.558830 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:42.558807 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bwdgc" Jul 09 22:10:42.558991 ip-10-0-136-251 kubenswrapper[2579]: E0709 22:10:42.558953 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bwdgc" podUID="b5e569ee-1aea-4eeb-8235-93438e648102" Jul 09 22:10:43.380030 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:43.379946 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-251.ec2.internal" event="NodeReady" Jul 09 22:10:43.380573 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:43.380097 2579 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Jul 09 22:10:43.412400 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:43.412363 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-5957744766-cmxpg"] Jul 09 22:10:43.471647 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:43.471604 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-6bffdf9778-kggxk"] Jul 09 22:10:43.471815 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:43.471729 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5957744766-cmxpg" Jul 09 22:10:43.474741 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:43.474714 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Jul 09 22:10:43.474882 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:43.474786 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Jul 09 22:10:43.474882 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:43.474799 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Jul 09 22:10:43.474882 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:43.474848 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-jn276\"" Jul 09 22:10:43.479624 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:43.479600 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Jul 09 22:10:43.490783 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:43.490750 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-6bffdf9778-kggxk"] Jul 09 22:10:43.490783 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:43.490785 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-vptbn"] Jul 09 22:10:43.491071 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:43.491048 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-6bffdf9778-kggxk" Jul 09 22:10:43.493333 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:43.493316 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\"" Jul 09 22:10:43.493431 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:43.493365 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-br9j6\"" Jul 09 22:10:43.493431 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:43.493316 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\"" Jul 09 22:10:43.516745 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:43.516726 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-5957744766-cmxpg"] Jul 09 22:10:43.516861 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:43.516759 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-tnwg2"] Jul 09 22:10:43.516861 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:43.516838 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-vptbn" Jul 09 22:10:43.519176 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:43.519138 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Jul 09 22:10:43.519306 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:43.519289 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-r6x6z\"" Jul 09 22:10:43.519402 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:43.519369 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Jul 09 22:10:43.540326 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:43.540306 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-tnwg2"] Jul 09 22:10:43.540444 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:43.540335 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-vptbn"] Jul 09 22:10:43.540503 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:43.540453 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-tnwg2" Jul 09 22:10:43.543252 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:43.543205 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Jul 09 22:10:43.543360 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:43.543263 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-r7pcm\"" Jul 09 22:10:43.543517 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:43.543489 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Jul 09 22:10:43.543604 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:43.543572 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Jul 09 22:10:43.558281 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:43.558260 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9b9zs" Jul 09 22:10:43.561128 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:43.561107 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-sjfh2\"" Jul 09 22:10:43.561215 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:43.561159 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 09 22:10:43.561215 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:43.561120 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 09 22:10:43.638382 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:43.638303 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/5a6d8c82-ca6e-49b1-b660-78501b981273-config-volume\") pod \"dns-default-vptbn\" (UID: \"5a6d8c82-ca6e-49b1-b660-78501b981273\") " pod="openshift-dns/dns-default-vptbn" Jul 09 22:10:43.638382 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:43.638334 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5a6d8c82-ca6e-49b1-b660-78501b981273-tmp-dir\") pod \"dns-default-vptbn\" (UID: \"5a6d8c82-ca6e-49b1-b660-78501b981273\") " pod="openshift-dns/dns-default-vptbn" Jul 09 22:10:43.638382 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:43.638356 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7djmn\" (UniqueName: \"kubernetes.io/projected/5afda08d-c04b-46ee-b245-6c884c49c78a-kube-api-access-7djmn\") pod \"ingress-canary-tnwg2\" (UID: \"5afda08d-c04b-46ee-b245-6c884c49c78a\") " pod="openshift-ingress-canary/ingress-canary-tnwg2" Jul 09 22:10:43.638598 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:43.638408 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/e8d76e6f-c1a2-4c70-8c4e-ce0b0480d75a-installation-pull-secrets\") pod \"image-registry-5957744766-cmxpg\" (UID: \"e8d76e6f-c1a2-4c70-8c4e-ce0b0480d75a\") " pod="openshift-image-registry/image-registry-5957744766-cmxpg" Jul 09 22:10:43.638598 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:43.638449 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5afda08d-c04b-46ee-b245-6c884c49c78a-cert\") pod \"ingress-canary-tnwg2\" (UID: \"5afda08d-c04b-46ee-b245-6c884c49c78a\") " pod="openshift-ingress-canary/ingress-canary-tnwg2" Jul 09 22:10:43.638598 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:43.638492 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/e8d76e6f-c1a2-4c70-8c4e-ce0b0480d75a-ca-trust-extracted\") pod \"image-registry-5957744766-cmxpg\" (UID: \"e8d76e6f-c1a2-4c70-8c4e-ce0b0480d75a\") " pod="openshift-image-registry/image-registry-5957744766-cmxpg" Jul 09 22:10:43.638598 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:43.638511 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/e8d76e6f-c1a2-4c70-8c4e-ce0b0480d75a-bound-sa-token\") pod \"image-registry-5957744766-cmxpg\" (UID: \"e8d76e6f-c1a2-4c70-8c4e-ce0b0480d75a\") " pod="openshift-image-registry/image-registry-5957744766-cmxpg" Jul 09 22:10:43.638598 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:43.638543 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/2b7da825-82cf-48d9-a4fb-50c6adbc34e0-nginx-conf\") pod \"networking-console-plugin-6bffdf9778-kggxk\" (UID: \"2b7da825-82cf-48d9-a4fb-50c6adbc34e0\") " pod="openshift-network-console/networking-console-plugin-6bffdf9778-kggxk" Jul 09 22:10:43.638598 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:43.638570 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vdhgh\" (UniqueName: \"kubernetes.io/projected/e8d76e6f-c1a2-4c70-8c4e-ce0b0480d75a-kube-api-access-vdhgh\") pod \"image-registry-5957744766-cmxpg\" (UID: \"e8d76e6f-c1a2-4c70-8c4e-ce0b0480d75a\") " pod="openshift-image-registry/image-registry-5957744766-cmxpg" Jul 09 22:10:43.638598 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:43.638586 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f4hph\" (UniqueName: \"kubernetes.io/projected/5a6d8c82-ca6e-49b1-b660-78501b981273-kube-api-access-f4hph\") pod \"dns-default-vptbn\" (UID: \"5a6d8c82-ca6e-49b1-b660-78501b981273\") " pod="openshift-dns/dns-default-vptbn" Jul 09 22:10:43.638821 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:43.638629 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/e8d76e6f-c1a2-4c70-8c4e-ce0b0480d75a-image-registry-private-configuration\") pod \"image-registry-5957744766-cmxpg\" (UID: \"e8d76e6f-c1a2-4c70-8c4e-ce0b0480d75a\") " pod="openshift-image-registry/image-registry-5957744766-cmxpg" Jul 09 22:10:43.638821 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:43.638646 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/e8d76e6f-c1a2-4c70-8c4e-ce0b0480d75a-registry-tls\") pod \"image-registry-5957744766-cmxpg\" (UID: \"e8d76e6f-c1a2-4c70-8c4e-ce0b0480d75a\") " pod="openshift-image-registry/image-registry-5957744766-cmxpg" Jul 09 22:10:43.638821 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:43.638661 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/5a6d8c82-ca6e-49b1-b660-78501b981273-metrics-tls\") pod \"dns-default-vptbn\" (UID: \"5a6d8c82-ca6e-49b1-b660-78501b981273\") " pod="openshift-dns/dns-default-vptbn" Jul 09 22:10:43.638821 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:43.638696 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/2b7da825-82cf-48d9-a4fb-50c6adbc34e0-networking-console-plugin-cert\") pod \"networking-console-plugin-6bffdf9778-kggxk\" (UID: \"2b7da825-82cf-48d9-a4fb-50c6adbc34e0\") " pod="openshift-network-console/networking-console-plugin-6bffdf9778-kggxk" Jul 09 22:10:43.638821 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:43.638734 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/e8d76e6f-c1a2-4c70-8c4e-ce0b0480d75a-registry-certificates\") pod \"image-registry-5957744766-cmxpg\" (UID: \"e8d76e6f-c1a2-4c70-8c4e-ce0b0480d75a\") " pod="openshift-image-registry/image-registry-5957744766-cmxpg" Jul 09 22:10:43.638821 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:43.638778 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/e8d76e6f-c1a2-4c70-8c4e-ce0b0480d75a-trusted-ca\") pod \"image-registry-5957744766-cmxpg\" (UID: \"e8d76e6f-c1a2-4c70-8c4e-ce0b0480d75a\") " pod="openshift-image-registry/image-registry-5957744766-cmxpg" Jul 09 22:10:43.739709 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:43.739677 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vdhgh\" (UniqueName: \"kubernetes.io/projected/e8d76e6f-c1a2-4c70-8c4e-ce0b0480d75a-kube-api-access-vdhgh\") pod \"image-registry-5957744766-cmxpg\" (UID: \"e8d76e6f-c1a2-4c70-8c4e-ce0b0480d75a\") " pod="openshift-image-registry/image-registry-5957744766-cmxpg" Jul 09 22:10:43.739709 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:43.739712 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f4hph\" (UniqueName: \"kubernetes.io/projected/5a6d8c82-ca6e-49b1-b660-78501b981273-kube-api-access-f4hph\") pod \"dns-default-vptbn\" (UID: \"5a6d8c82-ca6e-49b1-b660-78501b981273\") " pod="openshift-dns/dns-default-vptbn" Jul 09 22:10:43.739946 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:43.739740 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/e8d76e6f-c1a2-4c70-8c4e-ce0b0480d75a-image-registry-private-configuration\") pod \"image-registry-5957744766-cmxpg\" (UID: \"e8d76e6f-c1a2-4c70-8c4e-ce0b0480d75a\") " pod="openshift-image-registry/image-registry-5957744766-cmxpg" Jul 09 22:10:43.739946 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:43.739865 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/e8d76e6f-c1a2-4c70-8c4e-ce0b0480d75a-registry-tls\") pod \"image-registry-5957744766-cmxpg\" (UID: \"e8d76e6f-c1a2-4c70-8c4e-ce0b0480d75a\") " pod="openshift-image-registry/image-registry-5957744766-cmxpg" Jul 09 22:10:43.739946 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:43.739906 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/5a6d8c82-ca6e-49b1-b660-78501b981273-metrics-tls\") pod \"dns-default-vptbn\" (UID: \"5a6d8c82-ca6e-49b1-b660-78501b981273\") " pod="openshift-dns/dns-default-vptbn" Jul 09 22:10:43.740065 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:43.739968 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/2b7da825-82cf-48d9-a4fb-50c6adbc34e0-networking-console-plugin-cert\") pod \"networking-console-plugin-6bffdf9778-kggxk\" (UID: \"2b7da825-82cf-48d9-a4fb-50c6adbc34e0\") " pod="openshift-network-console/networking-console-plugin-6bffdf9778-kggxk" Jul 09 22:10:43.740065 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:43.740002 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/e8d76e6f-c1a2-4c70-8c4e-ce0b0480d75a-registry-certificates\") pod \"image-registry-5957744766-cmxpg\" (UID: \"e8d76e6f-c1a2-4c70-8c4e-ce0b0480d75a\") " pod="openshift-image-registry/image-registry-5957744766-cmxpg" Jul 09 22:10:43.740065 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:43.740041 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/e8d76e6f-c1a2-4c70-8c4e-ce0b0480d75a-trusted-ca\") pod \"image-registry-5957744766-cmxpg\" (UID: \"e8d76e6f-c1a2-4c70-8c4e-ce0b0480d75a\") " pod="openshift-image-registry/image-registry-5957744766-cmxpg" Jul 09 22:10:43.740256 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:43.740068 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/5a6d8c82-ca6e-49b1-b660-78501b981273-config-volume\") pod \"dns-default-vptbn\" (UID: \"5a6d8c82-ca6e-49b1-b660-78501b981273\") " pod="openshift-dns/dns-default-vptbn" Jul 09 22:10:43.740256 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:43.740094 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5a6d8c82-ca6e-49b1-b660-78501b981273-tmp-dir\") pod \"dns-default-vptbn\" (UID: \"5a6d8c82-ca6e-49b1-b660-78501b981273\") " pod="openshift-dns/dns-default-vptbn" Jul 09 22:10:43.740256 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:43.740120 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7djmn\" (UniqueName: \"kubernetes.io/projected/5afda08d-c04b-46ee-b245-6c884c49c78a-kube-api-access-7djmn\") pod \"ingress-canary-tnwg2\" (UID: \"5afda08d-c04b-46ee-b245-6c884c49c78a\") " pod="openshift-ingress-canary/ingress-canary-tnwg2" Jul 09 22:10:43.740256 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:43.740143 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/e8d76e6f-c1a2-4c70-8c4e-ce0b0480d75a-installation-pull-secrets\") pod \"image-registry-5957744766-cmxpg\" (UID: \"e8d76e6f-c1a2-4c70-8c4e-ce0b0480d75a\") " pod="openshift-image-registry/image-registry-5957744766-cmxpg" Jul 09 22:10:43.740256 ip-10-0-136-251 kubenswrapper[2579]: E0709 22:10:43.740163 2579 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 09 22:10:43.740256 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:43.740171 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5afda08d-c04b-46ee-b245-6c884c49c78a-cert\") pod \"ingress-canary-tnwg2\" (UID: \"5afda08d-c04b-46ee-b245-6c884c49c78a\") " pod="openshift-ingress-canary/ingress-canary-tnwg2" Jul 09 22:10:43.740256 ip-10-0-136-251 kubenswrapper[2579]: E0709 22:10:43.740221 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2b7da825-82cf-48d9-a4fb-50c6adbc34e0-networking-console-plugin-cert podName:2b7da825-82cf-48d9-a4fb-50c6adbc34e0 nodeName:}" failed. No retries permitted until 2026-07-09 22:10:44.240203259 +0000 UTC m=+36.151655937 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/2b7da825-82cf-48d9-a4fb-50c6adbc34e0-networking-console-plugin-cert") pod "networking-console-plugin-6bffdf9778-kggxk" (UID: "2b7da825-82cf-48d9-a4fb-50c6adbc34e0") : secret "networking-console-plugin-cert" not found Jul 09 22:10:43.740256 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:43.740237 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/e8d76e6f-c1a2-4c70-8c4e-ce0b0480d75a-ca-trust-extracted\") pod \"image-registry-5957744766-cmxpg\" (UID: \"e8d76e6f-c1a2-4c70-8c4e-ce0b0480d75a\") " pod="openshift-image-registry/image-registry-5957744766-cmxpg" Jul 09 22:10:43.740600 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:43.740260 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/e8d76e6f-c1a2-4c70-8c4e-ce0b0480d75a-bound-sa-token\") pod \"image-registry-5957744766-cmxpg\" (UID: \"e8d76e6f-c1a2-4c70-8c4e-ce0b0480d75a\") " pod="openshift-image-registry/image-registry-5957744766-cmxpg" Jul 09 22:10:43.740600 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:43.740288 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/2b7da825-82cf-48d9-a4fb-50c6adbc34e0-nginx-conf\") pod \"networking-console-plugin-6bffdf9778-kggxk\" (UID: \"2b7da825-82cf-48d9-a4fb-50c6adbc34e0\") " pod="openshift-network-console/networking-console-plugin-6bffdf9778-kggxk" Jul 09 22:10:43.741138 ip-10-0-136-251 kubenswrapper[2579]: E0709 22:10:43.740715 2579 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 09 22:10:43.741138 ip-10-0-136-251 kubenswrapper[2579]: E0709 22:10:43.740734 2579 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5957744766-cmxpg: secret "image-registry-tls" not found Jul 09 22:10:43.741138 ip-10-0-136-251 kubenswrapper[2579]: E0709 22:10:43.740777 2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 09 22:10:43.741138 ip-10-0-136-251 kubenswrapper[2579]: E0709 22:10:43.740833 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5a6d8c82-ca6e-49b1-b660-78501b981273-metrics-tls podName:5a6d8c82-ca6e-49b1-b660-78501b981273 nodeName:}" failed. No retries permitted until 2026-07-09 22:10:44.24081592 +0000 UTC m=+36.152268617 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/5a6d8c82-ca6e-49b1-b660-78501b981273-metrics-tls") pod "dns-default-vptbn" (UID: "5a6d8c82-ca6e-49b1-b660-78501b981273") : secret "dns-default-metrics-tls" not found Jul 09 22:10:43.741138 ip-10-0-136-251 kubenswrapper[2579]: E0709 22:10:43.740850 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e8d76e6f-c1a2-4c70-8c4e-ce0b0480d75a-registry-tls podName:e8d76e6f-c1a2-4c70-8c4e-ce0b0480d75a nodeName:}" failed. No retries permitted until 2026-07-09 22:10:44.240840746 +0000 UTC m=+36.152293427 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/e8d76e6f-c1a2-4c70-8c4e-ce0b0480d75a-registry-tls") pod "image-registry-5957744766-cmxpg" (UID: "e8d76e6f-c1a2-4c70-8c4e-ce0b0480d75a") : secret "image-registry-tls" not found Jul 09 22:10:43.741138 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:43.740853 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/5a6d8c82-ca6e-49b1-b660-78501b981273-config-volume\") pod \"dns-default-vptbn\" (UID: \"5a6d8c82-ca6e-49b1-b660-78501b981273\") " pod="openshift-dns/dns-default-vptbn" Jul 09 22:10:43.741138 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:43.741038 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/e8d76e6f-c1a2-4c70-8c4e-ce0b0480d75a-registry-certificates\") pod \"image-registry-5957744766-cmxpg\" (UID: \"e8d76e6f-c1a2-4c70-8c4e-ce0b0480d75a\") " pod="openshift-image-registry/image-registry-5957744766-cmxpg" Jul 09 22:10:43.741138 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:43.741076 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/2b7da825-82cf-48d9-a4fb-50c6adbc34e0-nginx-conf\") pod \"networking-console-plugin-6bffdf9778-kggxk\" (UID: \"2b7da825-82cf-48d9-a4fb-50c6adbc34e0\") " pod="openshift-network-console/networking-console-plugin-6bffdf9778-kggxk" Jul 09 22:10:43.741576 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:43.741228 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/e8d76e6f-c1a2-4c70-8c4e-ce0b0480d75a-trusted-ca\") pod \"image-registry-5957744766-cmxpg\" (UID: \"e8d76e6f-c1a2-4c70-8c4e-ce0b0480d75a\") " pod="openshift-image-registry/image-registry-5957744766-cmxpg" Jul 09 22:10:43.741576 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:43.741227 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5a6d8c82-ca6e-49b1-b660-78501b981273-tmp-dir\") pod \"dns-default-vptbn\" (UID: \"5a6d8c82-ca6e-49b1-b660-78501b981273\") " pod="openshift-dns/dns-default-vptbn" Jul 09 22:10:43.741576 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:43.741229 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/e8d76e6f-c1a2-4c70-8c4e-ce0b0480d75a-ca-trust-extracted\") pod \"image-registry-5957744766-cmxpg\" (UID: \"e8d76e6f-c1a2-4c70-8c4e-ce0b0480d75a\") " pod="openshift-image-registry/image-registry-5957744766-cmxpg" Jul 09 22:10:43.744948 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:43.744913 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5afda08d-c04b-46ee-b245-6c884c49c78a-cert\") pod \"ingress-canary-tnwg2\" (UID: \"5afda08d-c04b-46ee-b245-6c884c49c78a\") " pod="openshift-ingress-canary/ingress-canary-tnwg2" Jul 09 22:10:43.744948 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:43.744940 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/e8d76e6f-c1a2-4c70-8c4e-ce0b0480d75a-image-registry-private-configuration\") pod \"image-registry-5957744766-cmxpg\" (UID: \"e8d76e6f-c1a2-4c70-8c4e-ce0b0480d75a\") " pod="openshift-image-registry/image-registry-5957744766-cmxpg" Jul 09 22:10:43.745104 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:43.744972 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/e8d76e6f-c1a2-4c70-8c4e-ce0b0480d75a-installation-pull-secrets\") pod \"image-registry-5957744766-cmxpg\" (UID: \"e8d76e6f-c1a2-4c70-8c4e-ce0b0480d75a\") " pod="openshift-image-registry/image-registry-5957744766-cmxpg" Jul 09 22:10:43.749420 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:43.749396 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f4hph\" (UniqueName: \"kubernetes.io/projected/5a6d8c82-ca6e-49b1-b660-78501b981273-kube-api-access-f4hph\") pod \"dns-default-vptbn\" (UID: \"5a6d8c82-ca6e-49b1-b660-78501b981273\") " pod="openshift-dns/dns-default-vptbn" Jul 09 22:10:43.749662 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:43.749642 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vdhgh\" (UniqueName: \"kubernetes.io/projected/e8d76e6f-c1a2-4c70-8c4e-ce0b0480d75a-kube-api-access-vdhgh\") pod \"image-registry-5957744766-cmxpg\" (UID: \"e8d76e6f-c1a2-4c70-8c4e-ce0b0480d75a\") " pod="openshift-image-registry/image-registry-5957744766-cmxpg" Jul 09 22:10:43.749803 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:43.749779 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/e8d76e6f-c1a2-4c70-8c4e-ce0b0480d75a-bound-sa-token\") pod \"image-registry-5957744766-cmxpg\" (UID: \"e8d76e6f-c1a2-4c70-8c4e-ce0b0480d75a\") " pod="openshift-image-registry/image-registry-5957744766-cmxpg" Jul 09 22:10:43.750366 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:43.750272 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7djmn\" (UniqueName: \"kubernetes.io/projected/5afda08d-c04b-46ee-b245-6c884c49c78a-kube-api-access-7djmn\") pod \"ingress-canary-tnwg2\" (UID: \"5afda08d-c04b-46ee-b245-6c884c49c78a\") " pod="openshift-ingress-canary/ingress-canary-tnwg2" Jul 09 22:10:43.851493 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:43.851444 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-tnwg2" Jul 09 22:10:44.245255 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:44.245219 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/e8d76e6f-c1a2-4c70-8c4e-ce0b0480d75a-registry-tls\") pod \"image-registry-5957744766-cmxpg\" (UID: \"e8d76e6f-c1a2-4c70-8c4e-ce0b0480d75a\") " pod="openshift-image-registry/image-registry-5957744766-cmxpg" Jul 09 22:10:44.245255 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:44.245256 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/5a6d8c82-ca6e-49b1-b660-78501b981273-metrics-tls\") pod \"dns-default-vptbn\" (UID: \"5a6d8c82-ca6e-49b1-b660-78501b981273\") " pod="openshift-dns/dns-default-vptbn" Jul 09 22:10:44.245498 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:44.245300 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/2b7da825-82cf-48d9-a4fb-50c6adbc34e0-networking-console-plugin-cert\") pod \"networking-console-plugin-6bffdf9778-kggxk\" (UID: \"2b7da825-82cf-48d9-a4fb-50c6adbc34e0\") " pod="openshift-network-console/networking-console-plugin-6bffdf9778-kggxk" Jul 09 22:10:44.245498 ip-10-0-136-251 kubenswrapper[2579]: E0709 22:10:44.245404 2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 09 22:10:44.245498 ip-10-0-136-251 kubenswrapper[2579]: E0709 22:10:44.245444 2579 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 09 22:10:44.245498 ip-10-0-136-251 kubenswrapper[2579]: E0709 22:10:44.245408 2579 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 09 22:10:44.245498 ip-10-0-136-251 kubenswrapper[2579]: E0709 22:10:44.245486 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5a6d8c82-ca6e-49b1-b660-78501b981273-metrics-tls podName:5a6d8c82-ca6e-49b1-b660-78501b981273 nodeName:}" failed. No retries permitted until 2026-07-09 22:10:45.245467196 +0000 UTC m=+37.156919874 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/5a6d8c82-ca6e-49b1-b660-78501b981273-metrics-tls") pod "dns-default-vptbn" (UID: "5a6d8c82-ca6e-49b1-b660-78501b981273") : secret "dns-default-metrics-tls" not found Jul 09 22:10:44.245498 ip-10-0-136-251 kubenswrapper[2579]: E0709 22:10:44.245489 2579 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5957744766-cmxpg: secret "image-registry-tls" not found Jul 09 22:10:44.245796 ip-10-0-136-251 kubenswrapper[2579]: E0709 22:10:44.245504 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2b7da825-82cf-48d9-a4fb-50c6adbc34e0-networking-console-plugin-cert podName:2b7da825-82cf-48d9-a4fb-50c6adbc34e0 nodeName:}" failed. No retries permitted until 2026-07-09 22:10:45.245494629 +0000 UTC m=+37.156947307 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/2b7da825-82cf-48d9-a4fb-50c6adbc34e0-networking-console-plugin-cert") pod "networking-console-plugin-6bffdf9778-kggxk" (UID: "2b7da825-82cf-48d9-a4fb-50c6adbc34e0") : secret "networking-console-plugin-cert" not found Jul 09 22:10:44.245796 ip-10-0-136-251 kubenswrapper[2579]: E0709 22:10:44.245527 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e8d76e6f-c1a2-4c70-8c4e-ce0b0480d75a-registry-tls podName:e8d76e6f-c1a2-4c70-8c4e-ce0b0480d75a nodeName:}" failed. No retries permitted until 2026-07-09 22:10:45.245511118 +0000 UTC m=+37.156963797 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/e8d76e6f-c1a2-4c70-8c4e-ce0b0480d75a-registry-tls") pod "image-registry-5957744766-cmxpg" (UID: "e8d76e6f-c1a2-4c70-8c4e-ce0b0480d75a") : secret "image-registry-tls" not found Jul 09 22:10:44.558858 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:44.558781 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8pj95" Jul 09 22:10:44.559544 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:44.558794 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bwdgc" Jul 09 22:10:44.562716 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:44.562592 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-jtrvn\"" Jul 09 22:10:44.562716 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:44.562598 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 09 22:10:44.563015 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:44.562980 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 09 22:10:45.253889 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:45.253847 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/e8d76e6f-c1a2-4c70-8c4e-ce0b0480d75a-registry-tls\") pod \"image-registry-5957744766-cmxpg\" (UID: \"e8d76e6f-c1a2-4c70-8c4e-ce0b0480d75a\") " pod="openshift-image-registry/image-registry-5957744766-cmxpg" Jul 09 22:10:45.253889 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:45.253888 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/5a6d8c82-ca6e-49b1-b660-78501b981273-metrics-tls\") pod \"dns-default-vptbn\" (UID: \"5a6d8c82-ca6e-49b1-b660-78501b981273\") " pod="openshift-dns/dns-default-vptbn" Jul 09 22:10:45.254160 ip-10-0-136-251 kubenswrapper[2579]: E0709 22:10:45.254007 2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 09 22:10:45.254160 ip-10-0-136-251 kubenswrapper[2579]: E0709 22:10:45.254015 2579 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 09 22:10:45.254160 ip-10-0-136-251 kubenswrapper[2579]: E0709 22:10:45.254038 2579 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5957744766-cmxpg: secret "image-registry-tls" not found Jul 09 22:10:45.254160 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:45.254036 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/2b7da825-82cf-48d9-a4fb-50c6adbc34e0-networking-console-plugin-cert\") pod \"networking-console-plugin-6bffdf9778-kggxk\" (UID: \"2b7da825-82cf-48d9-a4fb-50c6adbc34e0\") " pod="openshift-network-console/networking-console-plugin-6bffdf9778-kggxk" Jul 09 22:10:45.254160 ip-10-0-136-251 kubenswrapper[2579]: E0709 22:10:45.254083 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5a6d8c82-ca6e-49b1-b660-78501b981273-metrics-tls podName:5a6d8c82-ca6e-49b1-b660-78501b981273 nodeName:}" failed. No retries permitted until 2026-07-09 22:10:47.254058187 +0000 UTC m=+39.165510873 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/5a6d8c82-ca6e-49b1-b660-78501b981273-metrics-tls") pod "dns-default-vptbn" (UID: "5a6d8c82-ca6e-49b1-b660-78501b981273") : secret "dns-default-metrics-tls" not found Jul 09 22:10:45.254160 ip-10-0-136-251 kubenswrapper[2579]: E0709 22:10:45.254124 2579 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 09 22:10:45.254160 ip-10-0-136-251 kubenswrapper[2579]: E0709 22:10:45.254124 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e8d76e6f-c1a2-4c70-8c4e-ce0b0480d75a-registry-tls podName:e8d76e6f-c1a2-4c70-8c4e-ce0b0480d75a nodeName:}" failed. No retries permitted until 2026-07-09 22:10:47.254101834 +0000 UTC m=+39.165554515 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/e8d76e6f-c1a2-4c70-8c4e-ce0b0480d75a-registry-tls") pod "image-registry-5957744766-cmxpg" (UID: "e8d76e6f-c1a2-4c70-8c4e-ce0b0480d75a") : secret "image-registry-tls" not found Jul 09 22:10:45.254470 ip-10-0-136-251 kubenswrapper[2579]: E0709 22:10:45.254186 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2b7da825-82cf-48d9-a4fb-50c6adbc34e0-networking-console-plugin-cert podName:2b7da825-82cf-48d9-a4fb-50c6adbc34e0 nodeName:}" failed. No retries permitted until 2026-07-09 22:10:47.254174249 +0000 UTC m=+39.165626927 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/2b7da825-82cf-48d9-a4fb-50c6adbc34e0-networking-console-plugin-cert") pod "networking-console-plugin-6bffdf9778-kggxk" (UID: "2b7da825-82cf-48d9-a4fb-50c6adbc34e0") : secret "networking-console-plugin-cert" not found Jul 09 22:10:45.374240 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:45.374025 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-tnwg2"] Jul 09 22:10:45.440774 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:10:45.440730 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5afda08d_c04b_46ee_b245_6c884c49c78a.slice/crio-148f4d9459e419dca7f0e0f3cf76b3f42d8e409fef18f5296a5947115af5b6ab WatchSource:0}: Error finding container 148f4d9459e419dca7f0e0f3cf76b3f42d8e409fef18f5296a5947115af5b6ab: Status 404 returned error can't find the container with id 148f4d9459e419dca7f0e0f3cf76b3f42d8e409fef18f5296a5947115af5b6ab Jul 09 22:10:45.768738 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:45.768703 2579 generic.go:356] "Generic (PLEG): container finished" podID="499e80b4-0095-43da-ad29-f143e59a7357" containerID="40d8fbee7eb4a3a73a258ddb34f58cadd64298d906a37e220ba0d8c646e84667" exitCode=0 Jul 09 22:10:45.769145 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:45.768777 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-5fzf2" event={"ID":"499e80b4-0095-43da-ad29-f143e59a7357","Type":"ContainerDied","Data":"40d8fbee7eb4a3a73a258ddb34f58cadd64298d906a37e220ba0d8c646e84667"} Jul 09 22:10:45.770035 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:45.770007 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-tnwg2" event={"ID":"5afda08d-c04b-46ee-b245-6c884c49c78a","Type":"ContainerStarted","Data":"148f4d9459e419dca7f0e0f3cf76b3f42d8e409fef18f5296a5947115af5b6ab"} Jul 09 22:10:46.776827 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:46.776797 2579 generic.go:356] "Generic (PLEG): container finished" podID="499e80b4-0095-43da-ad29-f143e59a7357" containerID="8a46b5c3ba4fb93d213af337d0c0c4ac44dee66dcf067a02effdd3b1505deef3" exitCode=0 Jul 09 22:10:46.777285 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:46.776868 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-5fzf2" event={"ID":"499e80b4-0095-43da-ad29-f143e59a7357","Type":"ContainerDied","Data":"8a46b5c3ba4fb93d213af337d0c0c4ac44dee66dcf067a02effdd3b1505deef3"} Jul 09 22:10:47.268620 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:47.268585 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/e8d76e6f-c1a2-4c70-8c4e-ce0b0480d75a-registry-tls\") pod \"image-registry-5957744766-cmxpg\" (UID: \"e8d76e6f-c1a2-4c70-8c4e-ce0b0480d75a\") " pod="openshift-image-registry/image-registry-5957744766-cmxpg" Jul 09 22:10:47.268620 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:47.268620 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/5a6d8c82-ca6e-49b1-b660-78501b981273-metrics-tls\") pod \"dns-default-vptbn\" (UID: \"5a6d8c82-ca6e-49b1-b660-78501b981273\") " pod="openshift-dns/dns-default-vptbn" Jul 09 22:10:47.268874 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:47.268647 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/2b7da825-82cf-48d9-a4fb-50c6adbc34e0-networking-console-plugin-cert\") pod \"networking-console-plugin-6bffdf9778-kggxk\" (UID: \"2b7da825-82cf-48d9-a4fb-50c6adbc34e0\") " pod="openshift-network-console/networking-console-plugin-6bffdf9778-kggxk" Jul 09 22:10:47.268874 ip-10-0-136-251 kubenswrapper[2579]: E0709 22:10:47.268741 2579 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 09 22:10:47.268874 ip-10-0-136-251 kubenswrapper[2579]: E0709 22:10:47.268749 2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 09 22:10:47.268874 ip-10-0-136-251 kubenswrapper[2579]: E0709 22:10:47.268805 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2b7da825-82cf-48d9-a4fb-50c6adbc34e0-networking-console-plugin-cert podName:2b7da825-82cf-48d9-a4fb-50c6adbc34e0 nodeName:}" failed. No retries permitted until 2026-07-09 22:10:51.268786154 +0000 UTC m=+43.180238853 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/2b7da825-82cf-48d9-a4fb-50c6adbc34e0-networking-console-plugin-cert") pod "networking-console-plugin-6bffdf9778-kggxk" (UID: "2b7da825-82cf-48d9-a4fb-50c6adbc34e0") : secret "networking-console-plugin-cert" not found Jul 09 22:10:47.268874 ip-10-0-136-251 kubenswrapper[2579]: E0709 22:10:47.268812 2579 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 09 22:10:47.268874 ip-10-0-136-251 kubenswrapper[2579]: E0709 22:10:47.268823 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5a6d8c82-ca6e-49b1-b660-78501b981273-metrics-tls podName:5a6d8c82-ca6e-49b1-b660-78501b981273 nodeName:}" failed. No retries permitted until 2026-07-09 22:10:51.268814167 +0000 UTC m=+43.180266852 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/5a6d8c82-ca6e-49b1-b660-78501b981273-metrics-tls") pod "dns-default-vptbn" (UID: "5a6d8c82-ca6e-49b1-b660-78501b981273") : secret "dns-default-metrics-tls" not found Jul 09 22:10:47.268874 ip-10-0-136-251 kubenswrapper[2579]: E0709 22:10:47.268829 2579 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5957744766-cmxpg: secret "image-registry-tls" not found Jul 09 22:10:47.268874 ip-10-0-136-251 kubenswrapper[2579]: E0709 22:10:47.268874 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e8d76e6f-c1a2-4c70-8c4e-ce0b0480d75a-registry-tls podName:e8d76e6f-c1a2-4c70-8c4e-ce0b0480d75a nodeName:}" failed. No retries permitted until 2026-07-09 22:10:51.268863142 +0000 UTC m=+43.180315837 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/e8d76e6f-c1a2-4c70-8c4e-ce0b0480d75a-registry-tls") pod "image-registry-5957744766-cmxpg" (UID: "e8d76e6f-c1a2-4c70-8c4e-ce0b0480d75a") : secret "image-registry-tls" not found Jul 09 22:10:47.783336 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:47.783300 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-5fzf2" event={"ID":"499e80b4-0095-43da-ad29-f143e59a7357","Type":"ContainerStarted","Data":"35f11bac372c70fb89e8618fbc08c934283763f25de8620c26b1c4c04e18c053"} Jul 09 22:10:47.808012 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:47.807956 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-5fzf2" podStartSLOduration=11.811785516 podStartE2EDuration="39.807937153s" totalImagesPullingTime="27.996151637s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 22:10:08 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 22:10:47.806670169 +0000 UTC m=+39.718122900" watchObservedRunningTime="2026-07-09 22:10:47.807937153 +0000 UTC m=+39.719389854" Jul 09 22:10:47.972517 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:47.972483 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gcmz7_5946d6db-f1f5-4fee-827b-1427b1bdd90f/konnectivity-agent/0.log" limit=1024 Jul 09 22:10:48.054482 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:48.054397 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gcmz7_5946d6db-f1f5-4fee-827b-1427b1bdd90f/konnectivity-agent/0.log" limit=1024 Jul 09 22:10:49.789680 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:49.789495 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-tnwg2" event={"ID":"5afda08d-c04b-46ee-b245-6c884c49c78a","Type":"ContainerStarted","Data":"6b7cf1514a19e78ddcb239fc9caa8e1c237f458bc951673fce027085f974afd9"} Jul 09 22:10:49.804880 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:49.804817 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-tnwg2" podStartSLOduration=3.48924067 podStartE2EDuration="6.804802629s" totalImagesPullingTime="3.315561959s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 22:10:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 22:10:49.804027426 +0000 UTC m=+41.715480155" watchObservedRunningTime="2026-07-09 22:10:49.804802629 +0000 UTC m=+41.716255329" Jul 09 22:10:50.610447 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:50.610412 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gcmz7_5946d6db-f1f5-4fee-827b-1427b1bdd90f/konnectivity-agent/0.log" limit=1024 Jul 09 22:10:50.711620 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:50.711588 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gcmz7_5946d6db-f1f5-4fee-827b-1427b1bdd90f/konnectivity-agent/0.log" limit=1024 Jul 09 22:10:51.298068 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:51.298026 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/e8d76e6f-c1a2-4c70-8c4e-ce0b0480d75a-registry-tls\") pod \"image-registry-5957744766-cmxpg\" (UID: \"e8d76e6f-c1a2-4c70-8c4e-ce0b0480d75a\") " pod="openshift-image-registry/image-registry-5957744766-cmxpg" Jul 09 22:10:51.298068 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:51.298074 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/5a6d8c82-ca6e-49b1-b660-78501b981273-metrics-tls\") pod \"dns-default-vptbn\" (UID: \"5a6d8c82-ca6e-49b1-b660-78501b981273\") " pod="openshift-dns/dns-default-vptbn" Jul 09 22:10:51.298587 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:51.298103 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/2b7da825-82cf-48d9-a4fb-50c6adbc34e0-networking-console-plugin-cert\") pod \"networking-console-plugin-6bffdf9778-kggxk\" (UID: \"2b7da825-82cf-48d9-a4fb-50c6adbc34e0\") " pod="openshift-network-console/networking-console-plugin-6bffdf9778-kggxk" Jul 09 22:10:51.298587 ip-10-0-136-251 kubenswrapper[2579]: E0709 22:10:51.298179 2579 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 09 22:10:51.298587 ip-10-0-136-251 kubenswrapper[2579]: E0709 22:10:51.298201 2579 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5957744766-cmxpg: secret "image-registry-tls" not found Jul 09 22:10:51.298587 ip-10-0-136-251 kubenswrapper[2579]: E0709 22:10:51.298216 2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 09 22:10:51.298587 ip-10-0-136-251 kubenswrapper[2579]: E0709 22:10:51.298225 2579 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 09 22:10:51.298587 ip-10-0-136-251 kubenswrapper[2579]: E0709 22:10:51.298253 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e8d76e6f-c1a2-4c70-8c4e-ce0b0480d75a-registry-tls podName:e8d76e6f-c1a2-4c70-8c4e-ce0b0480d75a nodeName:}" failed. No retries permitted until 2026-07-09 22:10:59.298237603 +0000 UTC m=+51.209690281 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/e8d76e6f-c1a2-4c70-8c4e-ce0b0480d75a-registry-tls") pod "image-registry-5957744766-cmxpg" (UID: "e8d76e6f-c1a2-4c70-8c4e-ce0b0480d75a") : secret "image-registry-tls" not found Jul 09 22:10:51.298587 ip-10-0-136-251 kubenswrapper[2579]: E0709 22:10:51.298272 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2b7da825-82cf-48d9-a4fb-50c6adbc34e0-networking-console-plugin-cert podName:2b7da825-82cf-48d9-a4fb-50c6adbc34e0 nodeName:}" failed. No retries permitted until 2026-07-09 22:10:59.298259306 +0000 UTC m=+51.209711985 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/2b7da825-82cf-48d9-a4fb-50c6adbc34e0-networking-console-plugin-cert") pod "networking-console-plugin-6bffdf9778-kggxk" (UID: "2b7da825-82cf-48d9-a4fb-50c6adbc34e0") : secret "networking-console-plugin-cert" not found Jul 09 22:10:51.298587 ip-10-0-136-251 kubenswrapper[2579]: E0709 22:10:51.298286 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5a6d8c82-ca6e-49b1-b660-78501b981273-metrics-tls podName:5a6d8c82-ca6e-49b1-b660-78501b981273 nodeName:}" failed. No retries permitted until 2026-07-09 22:10:59.298279001 +0000 UTC m=+51.209731680 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/5a6d8c82-ca6e-49b1-b660-78501b981273-metrics-tls") pod "dns-default-vptbn" (UID: "5a6d8c82-ca6e-49b1-b660-78501b981273") : secret "dns-default-metrics-tls" not found Jul 09 22:10:54.523428 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:54.523385 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gcmz7_5946d6db-f1f5-4fee-827b-1427b1bdd90f/konnectivity-agent/0.log" limit=1024 Jul 09 22:10:54.613972 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:54.613940 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gcmz7_5946d6db-f1f5-4fee-827b-1427b1bdd90f/konnectivity-agent/0.log" limit=1024 Jul 09 22:10:54.671932 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:54.671890 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gcmz7_5946d6db-f1f5-4fee-827b-1427b1bdd90f/konnectivity-agent/0.log" limit=1024 Jul 09 22:10:54.776638 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:54.776547 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gcmz7_5946d6db-f1f5-4fee-827b-1427b1bdd90f/konnectivity-agent/0.log" limit=1024 Jul 09 22:10:54.830694 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:54.830665 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gcmz7_5946d6db-f1f5-4fee-827b-1427b1bdd90f/konnectivity-agent/0.log" limit=1024 Jul 09 22:10:59.358155 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:59.358118 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/e8d76e6f-c1a2-4c70-8c4e-ce0b0480d75a-registry-tls\") pod \"image-registry-5957744766-cmxpg\" (UID: \"e8d76e6f-c1a2-4c70-8c4e-ce0b0480d75a\") " pod="openshift-image-registry/image-registry-5957744766-cmxpg" Jul 09 22:10:59.358155 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:59.358156 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/5a6d8c82-ca6e-49b1-b660-78501b981273-metrics-tls\") pod \"dns-default-vptbn\" (UID: \"5a6d8c82-ca6e-49b1-b660-78501b981273\") " pod="openshift-dns/dns-default-vptbn" Jul 09 22:10:59.358754 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:10:59.358183 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/2b7da825-82cf-48d9-a4fb-50c6adbc34e0-networking-console-plugin-cert\") pod \"networking-console-plugin-6bffdf9778-kggxk\" (UID: \"2b7da825-82cf-48d9-a4fb-50c6adbc34e0\") " pod="openshift-network-console/networking-console-plugin-6bffdf9778-kggxk" Jul 09 22:10:59.358754 ip-10-0-136-251 kubenswrapper[2579]: E0709 22:10:59.358273 2579 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 09 22:10:59.358754 ip-10-0-136-251 kubenswrapper[2579]: E0709 22:10:59.358297 2579 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5957744766-cmxpg: secret "image-registry-tls" not found Jul 09 22:10:59.358754 ip-10-0-136-251 kubenswrapper[2579]: E0709 22:10:59.358349 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e8d76e6f-c1a2-4c70-8c4e-ce0b0480d75a-registry-tls podName:e8d76e6f-c1a2-4c70-8c4e-ce0b0480d75a nodeName:}" failed. No retries permitted until 2026-07-09 22:11:15.358332415 +0000 UTC m=+67.269785094 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/e8d76e6f-c1a2-4c70-8c4e-ce0b0480d75a-registry-tls") pod "image-registry-5957744766-cmxpg" (UID: "e8d76e6f-c1a2-4c70-8c4e-ce0b0480d75a") : secret "image-registry-tls" not found Jul 09 22:10:59.358754 ip-10-0-136-251 kubenswrapper[2579]: E0709 22:10:59.358285 2579 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 09 22:10:59.358754 ip-10-0-136-251 kubenswrapper[2579]: E0709 22:10:59.358431 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2b7da825-82cf-48d9-a4fb-50c6adbc34e0-networking-console-plugin-cert podName:2b7da825-82cf-48d9-a4fb-50c6adbc34e0 nodeName:}" failed. No retries permitted until 2026-07-09 22:11:15.35841853 +0000 UTC m=+67.269871227 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/2b7da825-82cf-48d9-a4fb-50c6adbc34e0-networking-console-plugin-cert") pod "networking-console-plugin-6bffdf9778-kggxk" (UID: "2b7da825-82cf-48d9-a4fb-50c6adbc34e0") : secret "networking-console-plugin-cert" not found Jul 09 22:10:59.358754 ip-10-0-136-251 kubenswrapper[2579]: E0709 22:10:59.358285 2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 09 22:10:59.358754 ip-10-0-136-251 kubenswrapper[2579]: E0709 22:10:59.358511 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5a6d8c82-ca6e-49b1-b660-78501b981273-metrics-tls podName:5a6d8c82-ca6e-49b1-b660-78501b981273 nodeName:}" failed. No retries permitted until 2026-07-09 22:11:15.358493283 +0000 UTC m=+67.269945961 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/5a6d8c82-ca6e-49b1-b660-78501b981273-metrics-tls") pod "dns-default-vptbn" (UID: "5a6d8c82-ca6e-49b1-b660-78501b981273") : secret "dns-default-metrics-tls" not found Jul 09 22:11:09.762830 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:11:09.762802 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-r8dsp" Jul 09 22:11:14.269159 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:11:14.269118 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b5e569ee-1aea-4eeb-8235-93438e648102-original-pull-secret\") pod \"global-pull-secret-syncer-bwdgc\" (UID: \"b5e569ee-1aea-4eeb-8235-93438e648102\") " pod="kube-system/global-pull-secret-syncer-bwdgc" Jul 09 22:11:14.269159 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:11:14.269166 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/23b023ec-a254-462d-a0b4-5c22fd905e72-metrics-certs\") pod \"network-metrics-daemon-8pj95\" (UID: \"23b023ec-a254-462d-a0b4-5c22fd905e72\") " pod="openshift-multus/network-metrics-daemon-8pj95" Jul 09 22:11:14.272104 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:11:14.272084 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 09 22:11:14.272201 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:11:14.272162 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 09 22:11:14.280423 ip-10-0-136-251 kubenswrapper[2579]: E0709 22:11:14.280402 2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Jul 09 22:11:14.280524 ip-10-0-136-251 kubenswrapper[2579]: E0709 22:11:14.280462 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/23b023ec-a254-462d-a0b4-5c22fd905e72-metrics-certs podName:23b023ec-a254-462d-a0b4-5c22fd905e72 nodeName:}" failed. No retries permitted until 2026-07-09 22:12:18.280444619 +0000 UTC m=+130.191897297 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/23b023ec-a254-462d-a0b4-5c22fd905e72-metrics-certs") pod "network-metrics-daemon-8pj95" (UID: "23b023ec-a254-462d-a0b4-5c22fd905e72") : secret "metrics-daemon-secret" not found Jul 09 22:11:14.284195 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:11:14.284163 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b5e569ee-1aea-4eeb-8235-93438e648102-original-pull-secret\") pod \"global-pull-secret-syncer-bwdgc\" (UID: \"b5e569ee-1aea-4eeb-8235-93438e648102\") " pod="kube-system/global-pull-secret-syncer-bwdgc" Jul 09 22:11:14.370446 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:11:14.370371 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fx7mr\" (UniqueName: \"kubernetes.io/projected/541ce094-3638-4979-bf7b-782f3df1cbf8-kube-api-access-fx7mr\") pod \"network-check-target-9b9zs\" (UID: \"541ce094-3638-4979-bf7b-782f3df1cbf8\") " pod="openshift-network-diagnostics/network-check-target-9b9zs" Jul 09 22:11:14.373007 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:11:14.372984 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 09 22:11:14.383200 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:11:14.383182 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 09 22:11:14.393595 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:11:14.393572 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fx7mr\" (UniqueName: \"kubernetes.io/projected/541ce094-3638-4979-bf7b-782f3df1cbf8-kube-api-access-fx7mr\") pod \"network-check-target-9b9zs\" (UID: \"541ce094-3638-4979-bf7b-782f3df1cbf8\") " pod="openshift-network-diagnostics/network-check-target-9b9zs" Jul 09 22:11:14.470582 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:11:14.470550 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-sjfh2\"" Jul 09 22:11:14.478592 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:11:14.478569 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9b9zs" Jul 09 22:11:14.579217 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:11:14.579186 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bwdgc" Jul 09 22:11:14.635128 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:11:14.635087 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-9b9zs"] Jul 09 22:11:14.637792 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:11:14.637763 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod541ce094_3638_4979_bf7b_782f3df1cbf8.slice/crio-b4eaa604dc33b8689998ea766f8fb2f4610a551412eb050bc918f3e9870e2f3e WatchSource:0}: Error finding container b4eaa604dc33b8689998ea766f8fb2f4610a551412eb050bc918f3e9870e2f3e: Status 404 returned error can't find the container with id b4eaa604dc33b8689998ea766f8fb2f4610a551412eb050bc918f3e9870e2f3e Jul 09 22:11:14.727136 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:11:14.727102 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-bwdgc"] Jul 09 22:11:14.730013 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:11:14.729988 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb5e569ee_1aea_4eeb_8235_93438e648102.slice/crio-85195903be530e098cd8e7617cdf1246e03828bfed7ca6ae5e54efb0b83a37ad WatchSource:0}: Error finding container 85195903be530e098cd8e7617cdf1246e03828bfed7ca6ae5e54efb0b83a37ad: Status 404 returned error can't find the container with id 85195903be530e098cd8e7617cdf1246e03828bfed7ca6ae5e54efb0b83a37ad Jul 09 22:11:14.847134 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:11:14.847051 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-bwdgc" event={"ID":"b5e569ee-1aea-4eeb-8235-93438e648102","Type":"ContainerStarted","Data":"85195903be530e098cd8e7617cdf1246e03828bfed7ca6ae5e54efb0b83a37ad"} Jul 09 22:11:14.848081 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:11:14.848057 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-9b9zs" event={"ID":"541ce094-3638-4979-bf7b-782f3df1cbf8","Type":"ContainerStarted","Data":"b4eaa604dc33b8689998ea766f8fb2f4610a551412eb050bc918f3e9870e2f3e"} Jul 09 22:11:15.377201 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:11:15.377163 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/e8d76e6f-c1a2-4c70-8c4e-ce0b0480d75a-registry-tls\") pod \"image-registry-5957744766-cmxpg\" (UID: \"e8d76e6f-c1a2-4c70-8c4e-ce0b0480d75a\") " pod="openshift-image-registry/image-registry-5957744766-cmxpg" Jul 09 22:11:15.377201 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:11:15.377201 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/5a6d8c82-ca6e-49b1-b660-78501b981273-metrics-tls\") pod \"dns-default-vptbn\" (UID: \"5a6d8c82-ca6e-49b1-b660-78501b981273\") " pod="openshift-dns/dns-default-vptbn" Jul 09 22:11:15.377684 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:11:15.377231 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/2b7da825-82cf-48d9-a4fb-50c6adbc34e0-networking-console-plugin-cert\") pod \"networking-console-plugin-6bffdf9778-kggxk\" (UID: \"2b7da825-82cf-48d9-a4fb-50c6adbc34e0\") " pod="openshift-network-console/networking-console-plugin-6bffdf9778-kggxk" Jul 09 22:11:15.377684 ip-10-0-136-251 kubenswrapper[2579]: E0709 22:11:15.377350 2579 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 09 22:11:15.377684 ip-10-0-136-251 kubenswrapper[2579]: E0709 22:11:15.377357 2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 09 22:11:15.377684 ip-10-0-136-251 kubenswrapper[2579]: E0709 22:11:15.377411 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2b7da825-82cf-48d9-a4fb-50c6adbc34e0-networking-console-plugin-cert podName:2b7da825-82cf-48d9-a4fb-50c6adbc34e0 nodeName:}" failed. No retries permitted until 2026-07-09 22:11:47.377391717 +0000 UTC m=+99.288844401 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/2b7da825-82cf-48d9-a4fb-50c6adbc34e0-networking-console-plugin-cert") pod "networking-console-plugin-6bffdf9778-kggxk" (UID: "2b7da825-82cf-48d9-a4fb-50c6adbc34e0") : secret "networking-console-plugin-cert" not found Jul 09 22:11:15.377684 ip-10-0-136-251 kubenswrapper[2579]: E0709 22:11:15.377430 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5a6d8c82-ca6e-49b1-b660-78501b981273-metrics-tls podName:5a6d8c82-ca6e-49b1-b660-78501b981273 nodeName:}" failed. No retries permitted until 2026-07-09 22:11:47.37741999 +0000 UTC m=+99.288872680 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/5a6d8c82-ca6e-49b1-b660-78501b981273-metrics-tls") pod "dns-default-vptbn" (UID: "5a6d8c82-ca6e-49b1-b660-78501b981273") : secret "dns-default-metrics-tls" not found Jul 09 22:11:15.377684 ip-10-0-136-251 kubenswrapper[2579]: E0709 22:11:15.377450 2579 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 09 22:11:15.377684 ip-10-0-136-251 kubenswrapper[2579]: E0709 22:11:15.377463 2579 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5957744766-cmxpg: secret "image-registry-tls" not found Jul 09 22:11:15.377684 ip-10-0-136-251 kubenswrapper[2579]: E0709 22:11:15.377515 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e8d76e6f-c1a2-4c70-8c4e-ce0b0480d75a-registry-tls podName:e8d76e6f-c1a2-4c70-8c4e-ce0b0480d75a nodeName:}" failed. No retries permitted until 2026-07-09 22:11:47.37750377 +0000 UTC m=+99.288956461 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/e8d76e6f-c1a2-4c70-8c4e-ce0b0480d75a-registry-tls") pod "image-registry-5957744766-cmxpg" (UID: "e8d76e6f-c1a2-4c70-8c4e-ce0b0480d75a") : secret "image-registry-tls" not found Jul 09 22:11:19.863380 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:11:19.863283 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-9b9zs" event={"ID":"541ce094-3638-4979-bf7b-782f3df1cbf8","Type":"ContainerStarted","Data":"7703ecf8e85a8de465e41d054c097d53287079fc9ff0d35bde18fd82b65b9b9f"} Jul 09 22:11:19.863380 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:11:19.863374 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-9b9zs" Jul 09 22:11:19.864831 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:11:19.864808 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-bwdgc" event={"ID":"b5e569ee-1aea-4eeb-8235-93438e648102","Type":"ContainerStarted","Data":"84973cb300a217046dcec72a877b5bc1c9eff65899c5f4a54f3b29b1046012d4"} Jul 09 22:11:19.878465 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:11:19.878409 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-9b9zs" podStartSLOduration=66.973375753 podStartE2EDuration="1m11.878393606s" totalImagesPullingTime="4.905017853s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 22:10:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 22:11:19.877406376 +0000 UTC m=+71.788859076" watchObservedRunningTime="2026-07-09 22:11:19.878393606 +0000 UTC m=+71.789846307" Jul 09 22:11:19.894525 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:11:19.894480 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-bwdgc" podStartSLOduration=67.078099001 podStartE2EDuration="1m11.89446585s" totalImagesPullingTime="4.816366849s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 22:10:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 22:11:19.894370189 +0000 UTC m=+71.805822888" watchObservedRunningTime="2026-07-09 22:11:19.89446585 +0000 UTC m=+71.805918550" Jul 09 22:11:47.420199 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:11:47.420045 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/e8d76e6f-c1a2-4c70-8c4e-ce0b0480d75a-registry-tls\") pod \"image-registry-5957744766-cmxpg\" (UID: \"e8d76e6f-c1a2-4c70-8c4e-ce0b0480d75a\") " pod="openshift-image-registry/image-registry-5957744766-cmxpg" Jul 09 22:11:47.420199 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:11:47.420091 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/5a6d8c82-ca6e-49b1-b660-78501b981273-metrics-tls\") pod \"dns-default-vptbn\" (UID: \"5a6d8c82-ca6e-49b1-b660-78501b981273\") " pod="openshift-dns/dns-default-vptbn" Jul 09 22:11:47.420199 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:11:47.420121 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/2b7da825-82cf-48d9-a4fb-50c6adbc34e0-networking-console-plugin-cert\") pod \"networking-console-plugin-6bffdf9778-kggxk\" (UID: \"2b7da825-82cf-48d9-a4fb-50c6adbc34e0\") " pod="openshift-network-console/networking-console-plugin-6bffdf9778-kggxk" Jul 09 22:11:47.420771 ip-10-0-136-251 kubenswrapper[2579]: E0709 22:11:47.420204 2579 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 09 22:11:47.420771 ip-10-0-136-251 kubenswrapper[2579]: E0709 22:11:47.420221 2579 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5957744766-cmxpg: secret "image-registry-tls" not found Jul 09 22:11:47.420771 ip-10-0-136-251 kubenswrapper[2579]: E0709 22:11:47.420224 2579 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 09 22:11:47.420771 ip-10-0-136-251 kubenswrapper[2579]: E0709 22:11:47.420229 2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 09 22:11:47.420771 ip-10-0-136-251 kubenswrapper[2579]: E0709 22:11:47.420305 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e8d76e6f-c1a2-4c70-8c4e-ce0b0480d75a-registry-tls podName:e8d76e6f-c1a2-4c70-8c4e-ce0b0480d75a nodeName:}" failed. No retries permitted until 2026-07-09 22:12:51.420290055 +0000 UTC m=+163.331742746 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/e8d76e6f-c1a2-4c70-8c4e-ce0b0480d75a-registry-tls") pod "image-registry-5957744766-cmxpg" (UID: "e8d76e6f-c1a2-4c70-8c4e-ce0b0480d75a") : secret "image-registry-tls" not found Jul 09 22:11:47.420771 ip-10-0-136-251 kubenswrapper[2579]: E0709 22:11:47.420320 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5a6d8c82-ca6e-49b1-b660-78501b981273-metrics-tls podName:5a6d8c82-ca6e-49b1-b660-78501b981273 nodeName:}" failed. No retries permitted until 2026-07-09 22:12:51.420314119 +0000 UTC m=+163.331766797 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/5a6d8c82-ca6e-49b1-b660-78501b981273-metrics-tls") pod "dns-default-vptbn" (UID: "5a6d8c82-ca6e-49b1-b660-78501b981273") : secret "dns-default-metrics-tls" not found Jul 09 22:11:47.420771 ip-10-0-136-251 kubenswrapper[2579]: E0709 22:11:47.420330 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2b7da825-82cf-48d9-a4fb-50c6adbc34e0-networking-console-plugin-cert podName:2b7da825-82cf-48d9-a4fb-50c6adbc34e0 nodeName:}" failed. No retries permitted until 2026-07-09 22:12:51.420324973 +0000 UTC m=+163.331777651 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/2b7da825-82cf-48d9-a4fb-50c6adbc34e0-networking-console-plugin-cert") pod "networking-console-plugin-6bffdf9778-kggxk" (UID: "2b7da825-82cf-48d9-a4fb-50c6adbc34e0") : secret "networking-console-plugin-cert" not found Jul 09 22:11:50.869594 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:11:50.869565 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-9b9zs" Jul 09 22:12:18.341189 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:12:18.341150 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/23b023ec-a254-462d-a0b4-5c22fd905e72-metrics-certs\") pod \"network-metrics-daemon-8pj95\" (UID: \"23b023ec-a254-462d-a0b4-5c22fd905e72\") " pod="openshift-multus/network-metrics-daemon-8pj95" Jul 09 22:12:18.341762 ip-10-0-136-251 kubenswrapper[2579]: E0709 22:12:18.341269 2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Jul 09 22:12:18.341762 ip-10-0-136-251 kubenswrapper[2579]: E0709 22:12:18.341330 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/23b023ec-a254-462d-a0b4-5c22fd905e72-metrics-certs podName:23b023ec-a254-462d-a0b4-5c22fd905e72 nodeName:}" failed. No retries permitted until 2026-07-09 22:14:20.341312526 +0000 UTC m=+252.252765215 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/23b023ec-a254-462d-a0b4-5c22fd905e72-metrics-certs") pod "network-metrics-daemon-8pj95" (UID: "23b023ec-a254-462d-a0b4-5c22fd905e72") : secret "metrics-daemon-secret" not found Jul 09 22:12:46.481442 ip-10-0-136-251 kubenswrapper[2579]: E0709 22:12:46.481386 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[registry-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-image-registry/image-registry-5957744766-cmxpg" podUID="e8d76e6f-c1a2-4c70-8c4e-ce0b0480d75a" Jul 09 22:12:46.498531 ip-10-0-136-251 kubenswrapper[2579]: E0709 22:12:46.498497 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[networking-console-plugin-cert], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-network-console/networking-console-plugin-6bffdf9778-kggxk" podUID="2b7da825-82cf-48d9-a4fb-50c6adbc34e0" Jul 09 22:12:46.526831 ip-10-0-136-251 kubenswrapper[2579]: E0709 22:12:46.526803 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-vptbn" podUID="5a6d8c82-ca6e-49b1-b660-78501b981273" Jul 09 22:12:47.071241 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:12:47.071211 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5957744766-cmxpg" Jul 09 22:12:47.071241 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:12:47.071247 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-vptbn" Jul 09 22:12:47.573300 ip-10-0-136-251 kubenswrapper[2579]: E0709 22:12:47.573259 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-certs], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-multus/network-metrics-daemon-8pj95" podUID="23b023ec-a254-462d-a0b4-5c22fd905e72" Jul 09 22:12:51.480465 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:12:51.480414 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/2b7da825-82cf-48d9-a4fb-50c6adbc34e0-networking-console-plugin-cert\") pod \"networking-console-plugin-6bffdf9778-kggxk\" (UID: \"2b7da825-82cf-48d9-a4fb-50c6adbc34e0\") " pod="openshift-network-console/networking-console-plugin-6bffdf9778-kggxk" Jul 09 22:12:51.480971 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:12:51.480512 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/e8d76e6f-c1a2-4c70-8c4e-ce0b0480d75a-registry-tls\") pod \"image-registry-5957744766-cmxpg\" (UID: \"e8d76e6f-c1a2-4c70-8c4e-ce0b0480d75a\") " pod="openshift-image-registry/image-registry-5957744766-cmxpg" Jul 09 22:12:51.480971 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:12:51.480532 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/5a6d8c82-ca6e-49b1-b660-78501b981273-metrics-tls\") pod \"dns-default-vptbn\" (UID: \"5a6d8c82-ca6e-49b1-b660-78501b981273\") " pod="openshift-dns/dns-default-vptbn" Jul 09 22:12:51.480971 ip-10-0-136-251 kubenswrapper[2579]: E0709 22:12:51.480588 2579 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 09 22:12:51.480971 ip-10-0-136-251 kubenswrapper[2579]: E0709 22:12:51.480618 2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 09 22:12:51.480971 ip-10-0-136-251 kubenswrapper[2579]: E0709 22:12:51.480639 2579 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 09 22:12:51.480971 ip-10-0-136-251 kubenswrapper[2579]: E0709 22:12:51.480658 2579 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5957744766-cmxpg: secret "image-registry-tls" not found Jul 09 22:12:51.480971 ip-10-0-136-251 kubenswrapper[2579]: E0709 22:12:51.480674 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5a6d8c82-ca6e-49b1-b660-78501b981273-metrics-tls podName:5a6d8c82-ca6e-49b1-b660-78501b981273 nodeName:}" failed. No retries permitted until 2026-07-09 22:14:53.480660958 +0000 UTC m=+285.392113636 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/5a6d8c82-ca6e-49b1-b660-78501b981273-metrics-tls") pod "dns-default-vptbn" (UID: "5a6d8c82-ca6e-49b1-b660-78501b981273") : secret "dns-default-metrics-tls" not found Jul 09 22:12:51.480971 ip-10-0-136-251 kubenswrapper[2579]: E0709 22:12:51.480689 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2b7da825-82cf-48d9-a4fb-50c6adbc34e0-networking-console-plugin-cert podName:2b7da825-82cf-48d9-a4fb-50c6adbc34e0 nodeName:}" failed. No retries permitted until 2026-07-09 22:14:53.480682446 +0000 UTC m=+285.392135124 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/2b7da825-82cf-48d9-a4fb-50c6adbc34e0-networking-console-plugin-cert") pod "networking-console-plugin-6bffdf9778-kggxk" (UID: "2b7da825-82cf-48d9-a4fb-50c6adbc34e0") : secret "networking-console-plugin-cert" not found Jul 09 22:12:51.480971 ip-10-0-136-251 kubenswrapper[2579]: E0709 22:12:51.480698 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e8d76e6f-c1a2-4c70-8c4e-ce0b0480d75a-registry-tls podName:e8d76e6f-c1a2-4c70-8c4e-ce0b0480d75a nodeName:}" failed. No retries permitted until 2026-07-09 22:14:53.480693356 +0000 UTC m=+285.392146034 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/e8d76e6f-c1a2-4c70-8c4e-ce0b0480d75a-registry-tls") pod "image-registry-5957744766-cmxpg" (UID: "e8d76e6f-c1a2-4c70-8c4e-ce0b0480d75a") : secret "image-registry-tls" not found Jul 09 22:12:59.558733 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:12:59.558692 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8pj95" Jul 09 22:12:59.559320 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:12:59.558901 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-6bffdf9778-kggxk" Jul 09 22:14:20.413411 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:14:20.413356 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/23b023ec-a254-462d-a0b4-5c22fd905e72-metrics-certs\") pod \"network-metrics-daemon-8pj95\" (UID: \"23b023ec-a254-462d-a0b4-5c22fd905e72\") " pod="openshift-multus/network-metrics-daemon-8pj95" Jul 09 22:14:20.413950 ip-10-0-136-251 kubenswrapper[2579]: E0709 22:14:20.413508 2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Jul 09 22:14:20.413950 ip-10-0-136-251 kubenswrapper[2579]: E0709 22:14:20.413584 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/23b023ec-a254-462d-a0b4-5c22fd905e72-metrics-certs podName:23b023ec-a254-462d-a0b4-5c22fd905e72 nodeName:}" failed. No retries permitted until 2026-07-09 22:16:22.413569571 +0000 UTC m=+374.325022249 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/23b023ec-a254-462d-a0b4-5c22fd905e72-metrics-certs") pod "network-metrics-daemon-8pj95" (UID: "23b023ec-a254-462d-a0b4-5c22fd905e72") : secret "metrics-daemon-secret" not found Jul 09 22:14:30.942089 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:14:30.941998 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-lvwg4"] Jul 09 22:14:30.943763 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:14:30.943743 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-lvwg4" Jul 09 22:14:30.947172 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:14:30.947147 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\"" Jul 09 22:14:30.947797 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:14:30.947776 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-r5t5q"] Jul 09 22:14:30.948194 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:14:30.948172 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\"" Jul 09 22:14:30.948301 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:14:30.948197 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\"" Jul 09 22:14:30.948301 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:14:30.948266 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-d6ctf\"" Jul 09 22:14:30.948414 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:14:30.948336 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\"" Jul 09 22:14:30.949463 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:14:30.949447 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-854859444c-bhmbj"] Jul 09 22:14:30.949599 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:14:30.949583 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-r5t5q" Jul 09 22:14:30.951525 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:14:30.951509 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-854859444c-bhmbj" Jul 09 22:14:30.973078 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:14:30.973054 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\"" Jul 09 22:14:30.974783 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:14:30.974760 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\"" Jul 09 22:14:30.974891 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:14:30.974830 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\"" Jul 09 22:14:30.974891 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:14:30.974767 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Jul 09 22:14:30.974891 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:14:30.974766 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\"" Jul 09 22:14:30.976550 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:14:30.976533 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\"" Jul 09 22:14:30.976745 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:14:30.976728 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\"" Jul 09 22:14:30.976745 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:14:30.976737 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Jul 09 22:14:30.980878 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:14:30.980855 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-lvwg4"] Jul 09 22:14:30.983735 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:14:30.983721 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-hvjcs\"" Jul 09 22:14:30.984171 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:14:30.984157 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-tv5hr\"" Jul 09 22:14:30.986663 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:14:30.986644 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t5s4b\" (UniqueName: \"kubernetes.io/projected/dc1ba5a9-cd3c-46dc-8320-124980722f22-kube-api-access-t5s4b\") pod \"service-ca-operator-5d46d9ffd4-lvwg4\" (UID: \"dc1ba5a9-cd3c-46dc-8320-124980722f22\") " pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-lvwg4" Jul 09 22:14:30.986737 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:14:30.986680 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/dc1ba5a9-cd3c-46dc-8320-124980722f22-config\") pod \"service-ca-operator-5d46d9ffd4-lvwg4\" (UID: \"dc1ba5a9-cd3c-46dc-8320-124980722f22\") " pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-lvwg4" Jul 09 22:14:30.986737 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:14:30.986697 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b7lwp\" (UniqueName: \"kubernetes.io/projected/7144bb6f-5266-4f4e-9d78-5ad5aabbd2d4-kube-api-access-b7lwp\") pod \"cluster-samples-operator-56fcf86bf-r5t5q\" (UID: \"7144bb6f-5266-4f4e-9d78-5ad5aabbd2d4\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-r5t5q" Jul 09 22:14:30.986737 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:14:30.986716 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/7144bb6f-5266-4f4e-9d78-5ad5aabbd2d4-samples-operator-tls\") pod \"cluster-samples-operator-56fcf86bf-r5t5q\" (UID: \"7144bb6f-5266-4f4e-9d78-5ad5aabbd2d4\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-r5t5q" Jul 09 22:14:30.986885 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:14:30.986791 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/dc1ba5a9-cd3c-46dc-8320-124980722f22-serving-cert\") pod \"service-ca-operator-5d46d9ffd4-lvwg4\" (UID: \"dc1ba5a9-cd3c-46dc-8320-124980722f22\") " pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-lvwg4" Jul 09 22:14:30.987805 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:14:30.987791 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\"" Jul 09 22:14:30.999533 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:14:30.999514 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-854859444c-bhmbj"] Jul 09 22:14:31.004225 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:14:31.004203 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-r5t5q"] Jul 09 22:14:31.087960 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:14:31.087715 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/dc1ba5a9-cd3c-46dc-8320-124980722f22-serving-cert\") pod \"service-ca-operator-5d46d9ffd4-lvwg4\" (UID: \"dc1ba5a9-cd3c-46dc-8320-124980722f22\") " pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-lvwg4" Jul 09 22:14:31.087960 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:14:31.087779 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/31587075-3ee2-43c7-9d0e-5340de9912a1-service-ca-bundle\") pod \"router-default-854859444c-bhmbj\" (UID: \"31587075-3ee2-43c7-9d0e-5340de9912a1\") " pod="openshift-ingress/router-default-854859444c-bhmbj" Jul 09 22:14:31.087960 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:14:31.087867 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/31587075-3ee2-43c7-9d0e-5340de9912a1-default-certificate\") pod \"router-default-854859444c-bhmbj\" (UID: \"31587075-3ee2-43c7-9d0e-5340de9912a1\") " pod="openshift-ingress/router-default-854859444c-bhmbj" Jul 09 22:14:31.087960 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:14:31.087897 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t5s4b\" (UniqueName: \"kubernetes.io/projected/dc1ba5a9-cd3c-46dc-8320-124980722f22-kube-api-access-t5s4b\") pod \"service-ca-operator-5d46d9ffd4-lvwg4\" (UID: \"dc1ba5a9-cd3c-46dc-8320-124980722f22\") " pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-lvwg4" Jul 09 22:14:31.087960 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:14:31.087958 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/31587075-3ee2-43c7-9d0e-5340de9912a1-metrics-certs\") pod \"router-default-854859444c-bhmbj\" (UID: \"31587075-3ee2-43c7-9d0e-5340de9912a1\") " pod="openshift-ingress/router-default-854859444c-bhmbj" Jul 09 22:14:31.088424 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:14:31.088000 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/dc1ba5a9-cd3c-46dc-8320-124980722f22-config\") pod \"service-ca-operator-5d46d9ffd4-lvwg4\" (UID: \"dc1ba5a9-cd3c-46dc-8320-124980722f22\") " pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-lvwg4" Jul 09 22:14:31.088424 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:14:31.088027 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b7lwp\" (UniqueName: \"kubernetes.io/projected/7144bb6f-5266-4f4e-9d78-5ad5aabbd2d4-kube-api-access-b7lwp\") pod \"cluster-samples-operator-56fcf86bf-r5t5q\" (UID: \"7144bb6f-5266-4f4e-9d78-5ad5aabbd2d4\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-r5t5q" Jul 09 22:14:31.088424 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:14:31.088056 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/7144bb6f-5266-4f4e-9d78-5ad5aabbd2d4-samples-operator-tls\") pod \"cluster-samples-operator-56fcf86bf-r5t5q\" (UID: \"7144bb6f-5266-4f4e-9d78-5ad5aabbd2d4\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-r5t5q" Jul 09 22:14:31.088424 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:14:31.088080 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/31587075-3ee2-43c7-9d0e-5340de9912a1-stats-auth\") pod \"router-default-854859444c-bhmbj\" (UID: \"31587075-3ee2-43c7-9d0e-5340de9912a1\") " pod="openshift-ingress/router-default-854859444c-bhmbj" Jul 09 22:14:31.088424 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:14:31.088105 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c4kjg\" (UniqueName: \"kubernetes.io/projected/31587075-3ee2-43c7-9d0e-5340de9912a1-kube-api-access-c4kjg\") pod \"router-default-854859444c-bhmbj\" (UID: \"31587075-3ee2-43c7-9d0e-5340de9912a1\") " pod="openshift-ingress/router-default-854859444c-bhmbj" Jul 09 22:14:31.088424 ip-10-0-136-251 kubenswrapper[2579]: E0709 22:14:31.088306 2579 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 09 22:14:31.088424 ip-10-0-136-251 kubenswrapper[2579]: E0709 22:14:31.088389 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7144bb6f-5266-4f4e-9d78-5ad5aabbd2d4-samples-operator-tls podName:7144bb6f-5266-4f4e-9d78-5ad5aabbd2d4 nodeName:}" failed. No retries permitted until 2026-07-09 22:14:31.588370254 +0000 UTC m=+263.499822941 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/7144bb6f-5266-4f4e-9d78-5ad5aabbd2d4-samples-operator-tls") pod "cluster-samples-operator-56fcf86bf-r5t5q" (UID: "7144bb6f-5266-4f4e-9d78-5ad5aabbd2d4") : secret "samples-operator-tls" not found Jul 09 22:14:31.088689 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:14:31.088587 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/dc1ba5a9-cd3c-46dc-8320-124980722f22-config\") pod \"service-ca-operator-5d46d9ffd4-lvwg4\" (UID: \"dc1ba5a9-cd3c-46dc-8320-124980722f22\") " pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-lvwg4" Jul 09 22:14:31.088975 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:14:31.088954 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-xnjc7"] Jul 09 22:14:31.090278 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:14:31.090260 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/dc1ba5a9-cd3c-46dc-8320-124980722f22-serving-cert\") pod \"service-ca-operator-5d46d9ffd4-lvwg4\" (UID: \"dc1ba5a9-cd3c-46dc-8320-124980722f22\") " pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-lvwg4" Jul 09 22:14:31.090755 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:14:31.090739 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-xnjc7" Jul 09 22:14:31.093732 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:14:31.093702 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\"" Jul 09 22:14:31.093858 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:14:31.093796 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\"" Jul 09 22:14:31.094432 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:14:31.094399 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\"" Jul 09 22:14:31.094593 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:14:31.094485 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-5s8w5\"" Jul 09 22:14:31.094668 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:14:31.094494 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\"" Jul 09 22:14:31.115501 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:14:31.115474 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b7lwp\" (UniqueName: \"kubernetes.io/projected/7144bb6f-5266-4f4e-9d78-5ad5aabbd2d4-kube-api-access-b7lwp\") pod \"cluster-samples-operator-56fcf86bf-r5t5q\" (UID: \"7144bb6f-5266-4f4e-9d78-5ad5aabbd2d4\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-r5t5q" Jul 09 22:14:31.116897 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:14:31.116876 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-xnjc7"] Jul 09 22:14:31.132059 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:14:31.132036 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t5s4b\" (UniqueName: \"kubernetes.io/projected/dc1ba5a9-cd3c-46dc-8320-124980722f22-kube-api-access-t5s4b\") pod \"service-ca-operator-5d46d9ffd4-lvwg4\" (UID: \"dc1ba5a9-cd3c-46dc-8320-124980722f22\") " pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-lvwg4" Jul 09 22:14:31.188878 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:14:31.188852 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/31587075-3ee2-43c7-9d0e-5340de9912a1-default-certificate\") pod \"router-default-854859444c-bhmbj\" (UID: \"31587075-3ee2-43c7-9d0e-5340de9912a1\") " pod="openshift-ingress/router-default-854859444c-bhmbj" Jul 09 22:14:31.189059 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:14:31.188897 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c4kjg\" (UniqueName: \"kubernetes.io/projected/31587075-3ee2-43c7-9d0e-5340de9912a1-kube-api-access-c4kjg\") pod \"router-default-854859444c-bhmbj\" (UID: \"31587075-3ee2-43c7-9d0e-5340de9912a1\") " pod="openshift-ingress/router-default-854859444c-bhmbj" Jul 09 22:14:31.189059 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:14:31.188936 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t7k56\" (UniqueName: \"kubernetes.io/projected/7096ce79-14fb-4803-ab44-3a2eb01adf74-kube-api-access-t7k56\") pod \"kube-storage-version-migrator-operator-846bf668f7-xnjc7\" (UID: \"7096ce79-14fb-4803-ab44-3a2eb01adf74\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-xnjc7" Jul 09 22:14:31.189059 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:14:31.188962 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/7096ce79-14fb-4803-ab44-3a2eb01adf74-config\") pod \"kube-storage-version-migrator-operator-846bf668f7-xnjc7\" (UID: \"7096ce79-14fb-4803-ab44-3a2eb01adf74\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-xnjc7" Jul 09 22:14:31.189059 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:14:31.189005 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/31587075-3ee2-43c7-9d0e-5340de9912a1-metrics-certs\") pod \"router-default-854859444c-bhmbj\" (UID: \"31587075-3ee2-43c7-9d0e-5340de9912a1\") " pod="openshift-ingress/router-default-854859444c-bhmbj" Jul 09 22:14:31.189059 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:14:31.189021 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/7096ce79-14fb-4803-ab44-3a2eb01adf74-serving-cert\") pod \"kube-storage-version-migrator-operator-846bf668f7-xnjc7\" (UID: \"7096ce79-14fb-4803-ab44-3a2eb01adf74\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-xnjc7" Jul 09 22:14:31.189307 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:14:31.189113 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/31587075-3ee2-43c7-9d0e-5340de9912a1-stats-auth\") pod \"router-default-854859444c-bhmbj\" (UID: \"31587075-3ee2-43c7-9d0e-5340de9912a1\") " pod="openshift-ingress/router-default-854859444c-bhmbj" Jul 09 22:14:31.189307 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:14:31.189156 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/31587075-3ee2-43c7-9d0e-5340de9912a1-service-ca-bundle\") pod \"router-default-854859444c-bhmbj\" (UID: \"31587075-3ee2-43c7-9d0e-5340de9912a1\") " pod="openshift-ingress/router-default-854859444c-bhmbj" Jul 09 22:14:31.189307 ip-10-0-136-251 kubenswrapper[2579]: E0709 22:14:31.189165 2579 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 09 22:14:31.189307 ip-10-0-136-251 kubenswrapper[2579]: E0709 22:14:31.189227 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/31587075-3ee2-43c7-9d0e-5340de9912a1-metrics-certs podName:31587075-3ee2-43c7-9d0e-5340de9912a1 nodeName:}" failed. No retries permitted until 2026-07-09 22:14:31.689208119 +0000 UTC m=+263.600660798 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/31587075-3ee2-43c7-9d0e-5340de9912a1-metrics-certs") pod "router-default-854859444c-bhmbj" (UID: "31587075-3ee2-43c7-9d0e-5340de9912a1") : secret "router-metrics-certs-default" not found Jul 09 22:14:31.189307 ip-10-0-136-251 kubenswrapper[2579]: E0709 22:14:31.189298 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/31587075-3ee2-43c7-9d0e-5340de9912a1-service-ca-bundle podName:31587075-3ee2-43c7-9d0e-5340de9912a1 nodeName:}" failed. No retries permitted until 2026-07-09 22:14:31.689284217 +0000 UTC m=+263.600736901 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/31587075-3ee2-43c7-9d0e-5340de9912a1-service-ca-bundle") pod "router-default-854859444c-bhmbj" (UID: "31587075-3ee2-43c7-9d0e-5340de9912a1") : configmap references non-existent config key: service-ca.crt Jul 09 22:14:31.191316 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:14:31.191295 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/31587075-3ee2-43c7-9d0e-5340de9912a1-default-certificate\") pod \"router-default-854859444c-bhmbj\" (UID: \"31587075-3ee2-43c7-9d0e-5340de9912a1\") " pod="openshift-ingress/router-default-854859444c-bhmbj" Jul 09 22:14:31.191815 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:14:31.191794 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/31587075-3ee2-43c7-9d0e-5340de9912a1-stats-auth\") pod \"router-default-854859444c-bhmbj\" (UID: \"31587075-3ee2-43c7-9d0e-5340de9912a1\") " pod="openshift-ingress/router-default-854859444c-bhmbj" Jul 09 22:14:31.219084 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:14:31.219028 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c4kjg\" (UniqueName: \"kubernetes.io/projected/31587075-3ee2-43c7-9d0e-5340de9912a1-kube-api-access-c4kjg\") pod \"router-default-854859444c-bhmbj\" (UID: \"31587075-3ee2-43c7-9d0e-5340de9912a1\") " pod="openshift-ingress/router-default-854859444c-bhmbj" Jul 09 22:14:31.253987 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:14:31.253954 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-lvwg4" Jul 09 22:14:31.290560 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:14:31.290307 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t7k56\" (UniqueName: \"kubernetes.io/projected/7096ce79-14fb-4803-ab44-3a2eb01adf74-kube-api-access-t7k56\") pod \"kube-storage-version-migrator-operator-846bf668f7-xnjc7\" (UID: \"7096ce79-14fb-4803-ab44-3a2eb01adf74\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-xnjc7" Jul 09 22:14:31.290560 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:14:31.290365 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/7096ce79-14fb-4803-ab44-3a2eb01adf74-config\") pod \"kube-storage-version-migrator-operator-846bf668f7-xnjc7\" (UID: \"7096ce79-14fb-4803-ab44-3a2eb01adf74\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-xnjc7" Jul 09 22:14:31.290560 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:14:31.290435 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/7096ce79-14fb-4803-ab44-3a2eb01adf74-serving-cert\") pod \"kube-storage-version-migrator-operator-846bf668f7-xnjc7\" (UID: \"7096ce79-14fb-4803-ab44-3a2eb01adf74\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-xnjc7" Jul 09 22:14:31.290971 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:14:31.290949 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/7096ce79-14fb-4803-ab44-3a2eb01adf74-config\") pod \"kube-storage-version-migrator-operator-846bf668f7-xnjc7\" (UID: \"7096ce79-14fb-4803-ab44-3a2eb01adf74\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-xnjc7" Jul 09 22:14:31.292733 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:14:31.292703 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/7096ce79-14fb-4803-ab44-3a2eb01adf74-serving-cert\") pod \"kube-storage-version-migrator-operator-846bf668f7-xnjc7\" (UID: \"7096ce79-14fb-4803-ab44-3a2eb01adf74\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-xnjc7" Jul 09 22:14:31.306951 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:14:31.306908 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t7k56\" (UniqueName: \"kubernetes.io/projected/7096ce79-14fb-4803-ab44-3a2eb01adf74-kube-api-access-t7k56\") pod \"kube-storage-version-migrator-operator-846bf668f7-xnjc7\" (UID: \"7096ce79-14fb-4803-ab44-3a2eb01adf74\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-xnjc7" Jul 09 22:14:31.393229 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:14:31.393197 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-lvwg4"] Jul 09 22:14:31.396432 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:14:31.396410 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddc1ba5a9_cd3c_46dc_8320_124980722f22.slice/crio-5d74545762ba922d5b9bf2c228d5d994129b2617acb8c897a5d0f3c5e495ffa7 WatchSource:0}: Error finding container 5d74545762ba922d5b9bf2c228d5d994129b2617acb8c897a5d0f3c5e495ffa7: Status 404 returned error can't find the container with id 5d74545762ba922d5b9bf2c228d5d994129b2617acb8c897a5d0f3c5e495ffa7 Jul 09 22:14:31.402714 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:14:31.402691 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-xnjc7" Jul 09 22:14:31.568196 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:14:31.568173 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-xnjc7"] Jul 09 22:14:31.568944 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:14:31.568909 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7096ce79_14fb_4803_ab44_3a2eb01adf74.slice/crio-5302105519d54f44a2a2f99f1db1134f01efcd497cbf7f3fe8e3ef4596ac9520 WatchSource:0}: Error finding container 5302105519d54f44a2a2f99f1db1134f01efcd497cbf7f3fe8e3ef4596ac9520: Status 404 returned error can't find the container with id 5302105519d54f44a2a2f99f1db1134f01efcd497cbf7f3fe8e3ef4596ac9520 Jul 09 22:14:31.592643 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:14:31.592614 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/7144bb6f-5266-4f4e-9d78-5ad5aabbd2d4-samples-operator-tls\") pod \"cluster-samples-operator-56fcf86bf-r5t5q\" (UID: \"7144bb6f-5266-4f4e-9d78-5ad5aabbd2d4\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-r5t5q" Jul 09 22:14:31.592757 ip-10-0-136-251 kubenswrapper[2579]: E0709 22:14:31.592720 2579 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 09 22:14:31.592804 ip-10-0-136-251 kubenswrapper[2579]: E0709 22:14:31.592766 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7144bb6f-5266-4f4e-9d78-5ad5aabbd2d4-samples-operator-tls podName:7144bb6f-5266-4f4e-9d78-5ad5aabbd2d4 nodeName:}" failed. No retries permitted until 2026-07-09 22:14:32.592753696 +0000 UTC m=+264.504206374 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/7144bb6f-5266-4f4e-9d78-5ad5aabbd2d4-samples-operator-tls") pod "cluster-samples-operator-56fcf86bf-r5t5q" (UID: "7144bb6f-5266-4f4e-9d78-5ad5aabbd2d4") : secret "samples-operator-tls" not found Jul 09 22:14:31.694015 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:14:31.693971 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/31587075-3ee2-43c7-9d0e-5340de9912a1-metrics-certs\") pod \"router-default-854859444c-bhmbj\" (UID: \"31587075-3ee2-43c7-9d0e-5340de9912a1\") " pod="openshift-ingress/router-default-854859444c-bhmbj" Jul 09 22:14:31.694184 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:14:31.694040 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/31587075-3ee2-43c7-9d0e-5340de9912a1-service-ca-bundle\") pod \"router-default-854859444c-bhmbj\" (UID: \"31587075-3ee2-43c7-9d0e-5340de9912a1\") " pod="openshift-ingress/router-default-854859444c-bhmbj" Jul 09 22:14:31.694184 ip-10-0-136-251 kubenswrapper[2579]: E0709 22:14:31.694112 2579 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 09 22:14:31.694278 ip-10-0-136-251 kubenswrapper[2579]: E0709 22:14:31.694205 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/31587075-3ee2-43c7-9d0e-5340de9912a1-metrics-certs podName:31587075-3ee2-43c7-9d0e-5340de9912a1 nodeName:}" failed. No retries permitted until 2026-07-09 22:14:32.694166325 +0000 UTC m=+264.605619003 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/31587075-3ee2-43c7-9d0e-5340de9912a1-metrics-certs") pod "router-default-854859444c-bhmbj" (UID: "31587075-3ee2-43c7-9d0e-5340de9912a1") : secret "router-metrics-certs-default" not found Jul 09 22:14:31.694278 ip-10-0-136-251 kubenswrapper[2579]: E0709 22:14:31.694220 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/31587075-3ee2-43c7-9d0e-5340de9912a1-service-ca-bundle podName:31587075-3ee2-43c7-9d0e-5340de9912a1 nodeName:}" failed. No retries permitted until 2026-07-09 22:14:32.694213926 +0000 UTC m=+264.605666604 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/31587075-3ee2-43c7-9d0e-5340de9912a1-service-ca-bundle") pod "router-default-854859444c-bhmbj" (UID: "31587075-3ee2-43c7-9d0e-5340de9912a1") : configmap references non-existent config key: service-ca.crt Jul 09 22:14:32.317124 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:14:32.317084 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-xnjc7" event={"ID":"7096ce79-14fb-4803-ab44-3a2eb01adf74","Type":"ContainerStarted","Data":"5302105519d54f44a2a2f99f1db1134f01efcd497cbf7f3fe8e3ef4596ac9520"} Jul 09 22:14:32.317970 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:14:32.317945 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-lvwg4" event={"ID":"dc1ba5a9-cd3c-46dc-8320-124980722f22","Type":"ContainerStarted","Data":"5d74545762ba922d5b9bf2c228d5d994129b2617acb8c897a5d0f3c5e495ffa7"} Jul 09 22:14:32.605414 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:14:32.605313 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/7144bb6f-5266-4f4e-9d78-5ad5aabbd2d4-samples-operator-tls\") pod \"cluster-samples-operator-56fcf86bf-r5t5q\" (UID: \"7144bb6f-5266-4f4e-9d78-5ad5aabbd2d4\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-r5t5q" Jul 09 22:14:32.605587 ip-10-0-136-251 kubenswrapper[2579]: E0709 22:14:32.605547 2579 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 09 22:14:32.605649 ip-10-0-136-251 kubenswrapper[2579]: E0709 22:14:32.605616 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7144bb6f-5266-4f4e-9d78-5ad5aabbd2d4-samples-operator-tls podName:7144bb6f-5266-4f4e-9d78-5ad5aabbd2d4 nodeName:}" failed. No retries permitted until 2026-07-09 22:14:34.605595714 +0000 UTC m=+266.517048408 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/7144bb6f-5266-4f4e-9d78-5ad5aabbd2d4-samples-operator-tls") pod "cluster-samples-operator-56fcf86bf-r5t5q" (UID: "7144bb6f-5266-4f4e-9d78-5ad5aabbd2d4") : secret "samples-operator-tls" not found Jul 09 22:14:32.706830 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:14:32.706791 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/31587075-3ee2-43c7-9d0e-5340de9912a1-metrics-certs\") pod \"router-default-854859444c-bhmbj\" (UID: \"31587075-3ee2-43c7-9d0e-5340de9912a1\") " pod="openshift-ingress/router-default-854859444c-bhmbj" Jul 09 22:14:32.707025 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:14:32.706861 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/31587075-3ee2-43c7-9d0e-5340de9912a1-service-ca-bundle\") pod \"router-default-854859444c-bhmbj\" (UID: \"31587075-3ee2-43c7-9d0e-5340de9912a1\") " pod="openshift-ingress/router-default-854859444c-bhmbj" Jul 09 22:14:32.707025 ip-10-0-136-251 kubenswrapper[2579]: E0709 22:14:32.706983 2579 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 09 22:14:32.707025 ip-10-0-136-251 kubenswrapper[2579]: E0709 22:14:32.707014 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/31587075-3ee2-43c7-9d0e-5340de9912a1-service-ca-bundle podName:31587075-3ee2-43c7-9d0e-5340de9912a1 nodeName:}" failed. No retries permitted until 2026-07-09 22:14:34.706992403 +0000 UTC m=+266.618445083 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/31587075-3ee2-43c7-9d0e-5340de9912a1-service-ca-bundle") pod "router-default-854859444c-bhmbj" (UID: "31587075-3ee2-43c7-9d0e-5340de9912a1") : configmap references non-existent config key: service-ca.crt Jul 09 22:14:32.707231 ip-10-0-136-251 kubenswrapper[2579]: E0709 22:14:32.707049 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/31587075-3ee2-43c7-9d0e-5340de9912a1-metrics-certs podName:31587075-3ee2-43c7-9d0e-5340de9912a1 nodeName:}" failed. No retries permitted until 2026-07-09 22:14:34.70702943 +0000 UTC m=+266.618482114 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/31587075-3ee2-43c7-9d0e-5340de9912a1-metrics-certs") pod "router-default-854859444c-bhmbj" (UID: "31587075-3ee2-43c7-9d0e-5340de9912a1") : secret "router-metrics-certs-default" not found Jul 09 22:14:34.325126 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:14:34.325100 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-lvwg4" event={"ID":"dc1ba5a9-cd3c-46dc-8320-124980722f22","Type":"ContainerStarted","Data":"6d69df8d08a4c097f20715963fe964dfeef2df54a886e6907674f9de447b6c74"} Jul 09 22:14:34.623819 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:14:34.623719 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/7144bb6f-5266-4f4e-9d78-5ad5aabbd2d4-samples-operator-tls\") pod \"cluster-samples-operator-56fcf86bf-r5t5q\" (UID: \"7144bb6f-5266-4f4e-9d78-5ad5aabbd2d4\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-r5t5q" Jul 09 22:14:34.624006 ip-10-0-136-251 kubenswrapper[2579]: E0709 22:14:34.623838 2579 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 09 22:14:34.624006 ip-10-0-136-251 kubenswrapper[2579]: E0709 22:14:34.623891 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7144bb6f-5266-4f4e-9d78-5ad5aabbd2d4-samples-operator-tls podName:7144bb6f-5266-4f4e-9d78-5ad5aabbd2d4 nodeName:}" failed. No retries permitted until 2026-07-09 22:14:38.623877752 +0000 UTC m=+270.535330429 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/7144bb6f-5266-4f4e-9d78-5ad5aabbd2d4-samples-operator-tls") pod "cluster-samples-operator-56fcf86bf-r5t5q" (UID: "7144bb6f-5266-4f4e-9d78-5ad5aabbd2d4") : secret "samples-operator-tls" not found Jul 09 22:14:34.724934 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:14:34.724894 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/31587075-3ee2-43c7-9d0e-5340de9912a1-metrics-certs\") pod \"router-default-854859444c-bhmbj\" (UID: \"31587075-3ee2-43c7-9d0e-5340de9912a1\") " pod="openshift-ingress/router-default-854859444c-bhmbj" Jul 09 22:14:34.725113 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:14:34.724979 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/31587075-3ee2-43c7-9d0e-5340de9912a1-service-ca-bundle\") pod \"router-default-854859444c-bhmbj\" (UID: \"31587075-3ee2-43c7-9d0e-5340de9912a1\") " pod="openshift-ingress/router-default-854859444c-bhmbj" Jul 09 22:14:34.725113 ip-10-0-136-251 kubenswrapper[2579]: E0709 22:14:34.725054 2579 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 09 22:14:34.725196 ip-10-0-136-251 kubenswrapper[2579]: E0709 22:14:34.725130 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/31587075-3ee2-43c7-9d0e-5340de9912a1-metrics-certs podName:31587075-3ee2-43c7-9d0e-5340de9912a1 nodeName:}" failed. No retries permitted until 2026-07-09 22:14:38.725110049 +0000 UTC m=+270.636562742 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/31587075-3ee2-43c7-9d0e-5340de9912a1-metrics-certs") pod "router-default-854859444c-bhmbj" (UID: "31587075-3ee2-43c7-9d0e-5340de9912a1") : secret "router-metrics-certs-default" not found Jul 09 22:14:34.725196 ip-10-0-136-251 kubenswrapper[2579]: E0709 22:14:34.725164 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/31587075-3ee2-43c7-9d0e-5340de9912a1-service-ca-bundle podName:31587075-3ee2-43c7-9d0e-5340de9912a1 nodeName:}" failed. No retries permitted until 2026-07-09 22:14:38.725145509 +0000 UTC m=+270.636598192 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/31587075-3ee2-43c7-9d0e-5340de9912a1-service-ca-bundle") pod "router-default-854859444c-bhmbj" (UID: "31587075-3ee2-43c7-9d0e-5340de9912a1") : configmap references non-existent config key: service-ca.crt Jul 09 22:14:35.181017 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:14:35.180961 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-lvwg4" podStartSLOduration=2.939824877 podStartE2EDuration="5.180938017s" totalImagesPullingTime="2.24111314s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 22:14:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 22:14:34.363658674 +0000 UTC m=+266.275111373" watchObservedRunningTime="2026-07-09 22:14:35.180938017 +0000 UTC m=+267.092390707" Jul 09 22:14:35.181249 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:14:35.181237 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-7fv2q"] Jul 09 22:14:35.183228 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:14:35.183214 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-7fv2q" Jul 09 22:14:35.189715 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:14:35.189692 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Jul 09 22:14:35.197138 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:14:35.197119 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Jul 09 22:14:35.200174 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:14:35.200156 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Jul 09 22:14:35.200546 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:14:35.200527 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-rqrfq\"" Jul 09 22:14:35.200900 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:14:35.200887 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Jul 09 22:14:35.224090 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:14:35.224059 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-7fv2q"] Jul 09 22:14:35.329838 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:14:35.329801 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-xnjc7" event={"ID":"7096ce79-14fb-4803-ab44-3a2eb01adf74","Type":"ContainerStarted","Data":"79bb1cdd0541f9e7edc8a9f42de4ea9f4a77e41c3dec5fe3285ac40ff9e82479"} Jul 09 22:14:35.330333 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:14:35.330190 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n6m2w\" (UniqueName: \"kubernetes.io/projected/c1c63bd0-6360-4f30-9a66-f4255a633916-kube-api-access-n6m2w\") pod \"insights-runtime-extractor-7fv2q\" (UID: \"c1c63bd0-6360-4f30-9a66-f4255a633916\") " pod="openshift-insights/insights-runtime-extractor-7fv2q" Jul 09 22:14:35.330333 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:14:35.330310 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c1c63bd0-6360-4f30-9a66-f4255a633916-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-7fv2q\" (UID: \"c1c63bd0-6360-4f30-9a66-f4255a633916\") " pod="openshift-insights/insights-runtime-extractor-7fv2q" Jul 09 22:14:35.330454 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:14:35.330347 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/c1c63bd0-6360-4f30-9a66-f4255a633916-crio-socket\") pod \"insights-runtime-extractor-7fv2q\" (UID: \"c1c63bd0-6360-4f30-9a66-f4255a633916\") " pod="openshift-insights/insights-runtime-extractor-7fv2q" Jul 09 22:14:35.330454 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:14:35.330376 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/c1c63bd0-6360-4f30-9a66-f4255a633916-data-volume\") pod \"insights-runtime-extractor-7fv2q\" (UID: \"c1c63bd0-6360-4f30-9a66-f4255a633916\") " pod="openshift-insights/insights-runtime-extractor-7fv2q" Jul 09 22:14:35.330564 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:14:35.330490 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/c1c63bd0-6360-4f30-9a66-f4255a633916-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-7fv2q\" (UID: \"c1c63bd0-6360-4f30-9a66-f4255a633916\") " pod="openshift-insights/insights-runtime-extractor-7fv2q" Jul 09 22:14:35.361102 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:14:35.361060 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-xnjc7" podStartSLOduration=1.628444981 podStartE2EDuration="4.361031032s" totalImagesPullingTime="2.732586051s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 22:14:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 22:14:35.360113477 +0000 UTC m=+267.271566205" watchObservedRunningTime="2026-07-09 22:14:35.361031032 +0000 UTC m=+267.272483731" Jul 09 22:14:35.431673 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:14:35.431580 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n6m2w\" (UniqueName: \"kubernetes.io/projected/c1c63bd0-6360-4f30-9a66-f4255a633916-kube-api-access-n6m2w\") pod \"insights-runtime-extractor-7fv2q\" (UID: \"c1c63bd0-6360-4f30-9a66-f4255a633916\") " pod="openshift-insights/insights-runtime-extractor-7fv2q" Jul 09 22:14:35.431829 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:14:35.431748 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c1c63bd0-6360-4f30-9a66-f4255a633916-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-7fv2q\" (UID: \"c1c63bd0-6360-4f30-9a66-f4255a633916\") " pod="openshift-insights/insights-runtime-extractor-7fv2q" Jul 09 22:14:35.431829 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:14:35.431782 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/c1c63bd0-6360-4f30-9a66-f4255a633916-crio-socket\") pod \"insights-runtime-extractor-7fv2q\" (UID: \"c1c63bd0-6360-4f30-9a66-f4255a633916\") " pod="openshift-insights/insights-runtime-extractor-7fv2q" Jul 09 22:14:35.431829 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:14:35.431803 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/c1c63bd0-6360-4f30-9a66-f4255a633916-data-volume\") pod \"insights-runtime-extractor-7fv2q\" (UID: \"c1c63bd0-6360-4f30-9a66-f4255a633916\") " pod="openshift-insights/insights-runtime-extractor-7fv2q" Jul 09 22:14:35.432004 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:14:35.431851 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/c1c63bd0-6360-4f30-9a66-f4255a633916-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-7fv2q\" (UID: \"c1c63bd0-6360-4f30-9a66-f4255a633916\") " pod="openshift-insights/insights-runtime-extractor-7fv2q" Jul 09 22:14:35.432004 ip-10-0-136-251 kubenswrapper[2579]: E0709 22:14:35.431950 2579 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 09 22:14:35.432004 ip-10-0-136-251 kubenswrapper[2579]: E0709 22:14:35.431978 2579 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 09 22:14:35.432166 ip-10-0-136-251 kubenswrapper[2579]: E0709 22:14:35.432005 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/c1c63bd0-6360-4f30-9a66-f4255a633916-kube-rbac-proxy-cm podName:c1c63bd0-6360-4f30-9a66-f4255a633916 nodeName:}" failed. No retries permitted until 2026-07-09 22:14:35.931986959 +0000 UTC m=+267.843439663 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/c1c63bd0-6360-4f30-9a66-f4255a633916-kube-rbac-proxy-cm") pod "insights-runtime-extractor-7fv2q" (UID: "c1c63bd0-6360-4f30-9a66-f4255a633916") : configmap "kube-rbac-proxy" not found Jul 09 22:14:35.432166 ip-10-0-136-251 kubenswrapper[2579]: E0709 22:14:35.432050 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c1c63bd0-6360-4f30-9a66-f4255a633916-insights-runtime-extractor-tls podName:c1c63bd0-6360-4f30-9a66-f4255a633916 nodeName:}" failed. No retries permitted until 2026-07-09 22:14:35.932028526 +0000 UTC m=+267.843481237 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/c1c63bd0-6360-4f30-9a66-f4255a633916-insights-runtime-extractor-tls") pod "insights-runtime-extractor-7fv2q" (UID: "c1c63bd0-6360-4f30-9a66-f4255a633916") : secret "insights-runtime-extractor-tls" not found Jul 09 22:14:35.432287 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:14:35.432182 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/c1c63bd0-6360-4f30-9a66-f4255a633916-crio-socket\") pod \"insights-runtime-extractor-7fv2q\" (UID: \"c1c63bd0-6360-4f30-9a66-f4255a633916\") " pod="openshift-insights/insights-runtime-extractor-7fv2q" Jul 09 22:14:35.432287 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:14:35.432227 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/c1c63bd0-6360-4f30-9a66-f4255a633916-data-volume\") pod \"insights-runtime-extractor-7fv2q\" (UID: \"c1c63bd0-6360-4f30-9a66-f4255a633916\") " pod="openshift-insights/insights-runtime-extractor-7fv2q" Jul 09 22:14:35.453346 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:14:35.453323 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n6m2w\" (UniqueName: \"kubernetes.io/projected/c1c63bd0-6360-4f30-9a66-f4255a633916-kube-api-access-n6m2w\") pod \"insights-runtime-extractor-7fv2q\" (UID: \"c1c63bd0-6360-4f30-9a66-f4255a633916\") " pod="openshift-insights/insights-runtime-extractor-7fv2q" Jul 09 22:14:35.884886 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:14:35.884850 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-d57d58cc9-zlbgp"] Jul 09 22:14:35.886733 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:14:35.886718 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-d57d58cc9-zlbgp" Jul 09 22:14:35.894827 ip-10-0-136-251 kubenswrapper[2579]: E0709 22:14:35.894797 2579 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"kube-storage-version-migrator-sa-dockercfg-ccbwv\" is forbidden: User \"system:node:ip-10-0-136-251.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"openshift-kube-storage-version-migrator\": no relationship found between node 'ip-10-0-136-251.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-ccbwv\"" type="*v1.Secret" Jul 09 22:14:35.894827 ip-10-0-136-251 kubenswrapper[2579]: E0709 22:14:35.894813 2579 reflector.go:205] "Failed to watch" err="failed to list *v1.ConfigMap: configmaps \"kube-root-ca.crt\" is forbidden: User \"system:node:ip-10-0-136-251.ec2.internal\" cannot list resource \"configmaps\" in API group \"\" in the namespace \"openshift-kube-storage-version-migrator\": no relationship found between node 'ip-10-0-136-251.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" Jul 09 22:14:35.895011 ip-10-0-136-251 kubenswrapper[2579]: E0709 22:14:35.894847 2579 reflector.go:205] "Failed to watch" err="failed to list *v1.ConfigMap: configmaps \"openshift-service-ca.crt\" is forbidden: User \"system:node:ip-10-0-136-251.ec2.internal\" cannot list resource \"configmaps\" in API group \"\" in the namespace \"openshift-kube-storage-version-migrator\": no relationship found between node 'ip-10-0-136-251.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" Jul 09 22:14:35.895011 ip-10-0-136-251 kubenswrapper[2579]: E0709 22:14:35.894928 2579 status_manager.go:1018] "Failed to get status for pod" err="pods \"migrator-d57d58cc9-zlbgp\" is forbidden: User \"system:node:ip-10-0-136-251.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-kube-storage-version-migrator\": no relationship found between node 'ip-10-0-136-251.ec2.internal' and this object" podUID="b3c6c71f-f332-4b65-94a5-62502bded97b" pod="openshift-kube-storage-version-migrator/migrator-d57d58cc9-zlbgp" Jul 09 22:14:35.919475 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:14:35.919449 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-d57d58cc9-zlbgp"] Jul 09 22:14:35.935723 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:14:35.935696 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c1c63bd0-6360-4f30-9a66-f4255a633916-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-7fv2q\" (UID: \"c1c63bd0-6360-4f30-9a66-f4255a633916\") " pod="openshift-insights/insights-runtime-extractor-7fv2q" Jul 09 22:14:35.935881 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:14:35.935734 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/c1c63bd0-6360-4f30-9a66-f4255a633916-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-7fv2q\" (UID: \"c1c63bd0-6360-4f30-9a66-f4255a633916\") " pod="openshift-insights/insights-runtime-extractor-7fv2q" Jul 09 22:14:35.935881 ip-10-0-136-251 kubenswrapper[2579]: E0709 22:14:35.935819 2579 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 09 22:14:35.935881 ip-10-0-136-251 kubenswrapper[2579]: E0709 22:14:35.935860 2579 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 09 22:14:35.936036 ip-10-0-136-251 kubenswrapper[2579]: E0709 22:14:35.935869 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/c1c63bd0-6360-4f30-9a66-f4255a633916-kube-rbac-proxy-cm podName:c1c63bd0-6360-4f30-9a66-f4255a633916 nodeName:}" failed. No retries permitted until 2026-07-09 22:14:36.935855888 +0000 UTC m=+268.847308566 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/c1c63bd0-6360-4f30-9a66-f4255a633916-kube-rbac-proxy-cm") pod "insights-runtime-extractor-7fv2q" (UID: "c1c63bd0-6360-4f30-9a66-f4255a633916") : configmap "kube-rbac-proxy" not found Jul 09 22:14:35.936078 ip-10-0-136-251 kubenswrapper[2579]: E0709 22:14:35.936041 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c1c63bd0-6360-4f30-9a66-f4255a633916-insights-runtime-extractor-tls podName:c1c63bd0-6360-4f30-9a66-f4255a633916 nodeName:}" failed. No retries permitted until 2026-07-09 22:14:36.93602672 +0000 UTC m=+268.847479404 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/c1c63bd0-6360-4f30-9a66-f4255a633916-insights-runtime-extractor-tls") pod "insights-runtime-extractor-7fv2q" (UID: "c1c63bd0-6360-4f30-9a66-f4255a633916") : secret "insights-runtime-extractor-tls" not found Jul 09 22:14:36.037229 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:14:36.037196 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5dvvm\" (UniqueName: \"kubernetes.io/projected/b3c6c71f-f332-4b65-94a5-62502bded97b-kube-api-access-5dvvm\") pod \"migrator-d57d58cc9-zlbgp\" (UID: \"b3c6c71f-f332-4b65-94a5-62502bded97b\") " pod="openshift-kube-storage-version-migrator/migrator-d57d58cc9-zlbgp" Jul 09 22:14:36.138024 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:14:36.137914 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5dvvm\" (UniqueName: \"kubernetes.io/projected/b3c6c71f-f332-4b65-94a5-62502bded97b-kube-api-access-5dvvm\") pod \"migrator-d57d58cc9-zlbgp\" (UID: \"b3c6c71f-f332-4b65-94a5-62502bded97b\") " pod="openshift-kube-storage-version-migrator/migrator-d57d58cc9-zlbgp" Jul 09 22:14:36.837099 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:14:36.837060 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-ccbwv\"" Jul 09 22:14:36.944527 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:14:36.944493 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c1c63bd0-6360-4f30-9a66-f4255a633916-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-7fv2q\" (UID: \"c1c63bd0-6360-4f30-9a66-f4255a633916\") " pod="openshift-insights/insights-runtime-extractor-7fv2q" Jul 09 22:14:36.944698 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:14:36.944536 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/c1c63bd0-6360-4f30-9a66-f4255a633916-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-7fv2q\" (UID: \"c1c63bd0-6360-4f30-9a66-f4255a633916\") " pod="openshift-insights/insights-runtime-extractor-7fv2q" Jul 09 22:14:36.944698 ip-10-0-136-251 kubenswrapper[2579]: E0709 22:14:36.944656 2579 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 09 22:14:36.944698 ip-10-0-136-251 kubenswrapper[2579]: E0709 22:14:36.944664 2579 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 09 22:14:36.944842 ip-10-0-136-251 kubenswrapper[2579]: E0709 22:14:36.944709 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/c1c63bd0-6360-4f30-9a66-f4255a633916-kube-rbac-proxy-cm podName:c1c63bd0-6360-4f30-9a66-f4255a633916 nodeName:}" failed. No retries permitted until 2026-07-09 22:14:38.944695127 +0000 UTC m=+270.856147808 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/c1c63bd0-6360-4f30-9a66-f4255a633916-kube-rbac-proxy-cm") pod "insights-runtime-extractor-7fv2q" (UID: "c1c63bd0-6360-4f30-9a66-f4255a633916") : configmap "kube-rbac-proxy" not found Jul 09 22:14:36.944842 ip-10-0-136-251 kubenswrapper[2579]: E0709 22:14:36.944722 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c1c63bd0-6360-4f30-9a66-f4255a633916-insights-runtime-extractor-tls podName:c1c63bd0-6360-4f30-9a66-f4255a633916 nodeName:}" failed. No retries permitted until 2026-07-09 22:14:38.944715892 +0000 UTC m=+270.856168570 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/c1c63bd0-6360-4f30-9a66-f4255a633916-insights-runtime-extractor-tls") pod "insights-runtime-extractor-7fv2q" (UID: "c1c63bd0-6360-4f30-9a66-f4255a633916") : secret "insights-runtime-extractor-tls" not found Jul 09 22:14:36.953275 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:14:36.953247 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\"" Jul 09 22:14:37.013284 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:14:37.013253 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\"" Jul 09 22:14:37.018522 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:14:37.018498 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5dvvm\" (UniqueName: \"kubernetes.io/projected/b3c6c71f-f332-4b65-94a5-62502bded97b-kube-api-access-5dvvm\") pod \"migrator-d57d58cc9-zlbgp\" (UID: \"b3c6c71f-f332-4b65-94a5-62502bded97b\") " pod="openshift-kube-storage-version-migrator/migrator-d57d58cc9-zlbgp" Jul 09 22:14:37.095636 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:14:37.095550 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-d57d58cc9-zlbgp" Jul 09 22:14:37.169486 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:14:37.169462 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-nr4cq_6807e2e6-d9a4-475a-9fb4-4b958127840f/dns-node-resolver/0.log" Jul 09 22:14:37.226949 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:14:37.226902 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-d57d58cc9-zlbgp"] Jul 09 22:14:37.229818 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:14:37.229792 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb3c6c71f_f332_4b65_94a5_62502bded97b.slice/crio-8588b75a3410c54620c6680eb2ce838137bea9832a2d7e31df70a9982eaba05b WatchSource:0}: Error finding container 8588b75a3410c54620c6680eb2ce838137bea9832a2d7e31df70a9982eaba05b: Status 404 returned error can't find the container with id 8588b75a3410c54620c6680eb2ce838137bea9832a2d7e31df70a9982eaba05b Jul 09 22:14:37.335542 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:14:37.335507 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-d57d58cc9-zlbgp" event={"ID":"b3c6c71f-f332-4b65-94a5-62502bded97b","Type":"ContainerStarted","Data":"8588b75a3410c54620c6680eb2ce838137bea9832a2d7e31df70a9982eaba05b"} Jul 09 22:14:38.174988 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:14:38.174957 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-l578c_af0be020-5b8b-4828-bf98-531265c795b2/node-ca/0.log" Jul 09 22:14:38.660352 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:14:38.660317 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/7144bb6f-5266-4f4e-9d78-5ad5aabbd2d4-samples-operator-tls\") pod \"cluster-samples-operator-56fcf86bf-r5t5q\" (UID: \"7144bb6f-5266-4f4e-9d78-5ad5aabbd2d4\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-r5t5q" Jul 09 22:14:38.660558 ip-10-0-136-251 kubenswrapper[2579]: E0709 22:14:38.660490 2579 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 09 22:14:38.660649 ip-10-0-136-251 kubenswrapper[2579]: E0709 22:14:38.660573 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7144bb6f-5266-4f4e-9d78-5ad5aabbd2d4-samples-operator-tls podName:7144bb6f-5266-4f4e-9d78-5ad5aabbd2d4 nodeName:}" failed. No retries permitted until 2026-07-09 22:14:46.660549363 +0000 UTC m=+278.572002046 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/7144bb6f-5266-4f4e-9d78-5ad5aabbd2d4-samples-operator-tls") pod "cluster-samples-operator-56fcf86bf-r5t5q" (UID: "7144bb6f-5266-4f4e-9d78-5ad5aabbd2d4") : secret "samples-operator-tls" not found Jul 09 22:14:38.761502 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:14:38.761479 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/31587075-3ee2-43c7-9d0e-5340de9912a1-metrics-certs\") pod \"router-default-854859444c-bhmbj\" (UID: \"31587075-3ee2-43c7-9d0e-5340de9912a1\") " pod="openshift-ingress/router-default-854859444c-bhmbj" Jul 09 22:14:38.761600 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:14:38.761540 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/31587075-3ee2-43c7-9d0e-5340de9912a1-service-ca-bundle\") pod \"router-default-854859444c-bhmbj\" (UID: \"31587075-3ee2-43c7-9d0e-5340de9912a1\") " pod="openshift-ingress/router-default-854859444c-bhmbj" Jul 09 22:14:38.761663 ip-10-0-136-251 kubenswrapper[2579]: E0709 22:14:38.761616 2579 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 09 22:14:38.761709 ip-10-0-136-251 kubenswrapper[2579]: E0709 22:14:38.761691 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/31587075-3ee2-43c7-9d0e-5340de9912a1-service-ca-bundle podName:31587075-3ee2-43c7-9d0e-5340de9912a1 nodeName:}" failed. No retries permitted until 2026-07-09 22:14:46.761676058 +0000 UTC m=+278.673128737 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/31587075-3ee2-43c7-9d0e-5340de9912a1-service-ca-bundle") pod "router-default-854859444c-bhmbj" (UID: "31587075-3ee2-43c7-9d0e-5340de9912a1") : configmap references non-existent config key: service-ca.crt Jul 09 22:14:38.761748 ip-10-0-136-251 kubenswrapper[2579]: E0709 22:14:38.761715 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/31587075-3ee2-43c7-9d0e-5340de9912a1-metrics-certs podName:31587075-3ee2-43c7-9d0e-5340de9912a1 nodeName:}" failed. No retries permitted until 2026-07-09 22:14:46.761706851 +0000 UTC m=+278.673159529 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/31587075-3ee2-43c7-9d0e-5340de9912a1-metrics-certs") pod "router-default-854859444c-bhmbj" (UID: "31587075-3ee2-43c7-9d0e-5340de9912a1") : secret "router-metrics-certs-default" not found Jul 09 22:14:38.963318 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:14:38.963233 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c1c63bd0-6360-4f30-9a66-f4255a633916-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-7fv2q\" (UID: \"c1c63bd0-6360-4f30-9a66-f4255a633916\") " pod="openshift-insights/insights-runtime-extractor-7fv2q" Jul 09 22:14:38.963318 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:14:38.963275 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/c1c63bd0-6360-4f30-9a66-f4255a633916-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-7fv2q\" (UID: \"c1c63bd0-6360-4f30-9a66-f4255a633916\") " pod="openshift-insights/insights-runtime-extractor-7fv2q" Jul 09 22:14:38.963558 ip-10-0-136-251 kubenswrapper[2579]: E0709 22:14:38.963360 2579 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 09 22:14:38.963558 ip-10-0-136-251 kubenswrapper[2579]: E0709 22:14:38.963404 2579 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 09 22:14:38.963558 ip-10-0-136-251 kubenswrapper[2579]: E0709 22:14:38.963410 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/c1c63bd0-6360-4f30-9a66-f4255a633916-kube-rbac-proxy-cm podName:c1c63bd0-6360-4f30-9a66-f4255a633916 nodeName:}" failed. No retries permitted until 2026-07-09 22:14:42.963392779 +0000 UTC m=+274.874845457 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/c1c63bd0-6360-4f30-9a66-f4255a633916-kube-rbac-proxy-cm") pod "insights-runtime-extractor-7fv2q" (UID: "c1c63bd0-6360-4f30-9a66-f4255a633916") : configmap "kube-rbac-proxy" not found Jul 09 22:14:38.963666 ip-10-0-136-251 kubenswrapper[2579]: E0709 22:14:38.963563 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c1c63bd0-6360-4f30-9a66-f4255a633916-insights-runtime-extractor-tls podName:c1c63bd0-6360-4f30-9a66-f4255a633916 nodeName:}" failed. No retries permitted until 2026-07-09 22:14:42.963544796 +0000 UTC m=+274.874997479 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/c1c63bd0-6360-4f30-9a66-f4255a633916-insights-runtime-extractor-tls") pod "insights-runtime-extractor-7fv2q" (UID: "c1c63bd0-6360-4f30-9a66-f4255a633916") : secret "insights-runtime-extractor-tls" not found Jul 09 22:14:38.976977 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:14:38.976953 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-tnwg2_5afda08d-c04b-46ee-b245-6c884c49c78a/serve-healthcheck-canary/0.log" Jul 09 22:14:39.346592 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:14:39.346559 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-d57d58cc9-zlbgp" event={"ID":"b3c6c71f-f332-4b65-94a5-62502bded97b","Type":"ContainerStarted","Data":"198fc016c9d8ac406a81b22dbb8db12724ed64156c5601dcc7aa47e0cb219497"} Jul 09 22:14:39.347004 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:14:39.346610 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-d57d58cc9-zlbgp" event={"ID":"b3c6c71f-f332-4b65-94a5-62502bded97b","Type":"ContainerStarted","Data":"9194051c2fd9eccf0dd83129dee0b27c213dddd07301d191ff734e40860af68e"} Jul 09 22:14:39.404478 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:14:39.404428 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-d57d58cc9-zlbgp" podStartSLOduration=2.936283404 podStartE2EDuration="4.40440954s" totalImagesPullingTime="1.468126136s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 22:14:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 22:14:39.402509191 +0000 UTC m=+271.313961908" watchObservedRunningTime="2026-07-09 22:14:39.40440954 +0000 UTC m=+271.315862283" Jul 09 22:14:42.998692 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:14:42.998655 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c1c63bd0-6360-4f30-9a66-f4255a633916-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-7fv2q\" (UID: \"c1c63bd0-6360-4f30-9a66-f4255a633916\") " pod="openshift-insights/insights-runtime-extractor-7fv2q" Jul 09 22:14:42.999106 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:14:42.998708 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/c1c63bd0-6360-4f30-9a66-f4255a633916-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-7fv2q\" (UID: \"c1c63bd0-6360-4f30-9a66-f4255a633916\") " pod="openshift-insights/insights-runtime-extractor-7fv2q" Jul 09 22:14:42.999106 ip-10-0-136-251 kubenswrapper[2579]: E0709 22:14:42.998772 2579 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 09 22:14:42.999106 ip-10-0-136-251 kubenswrapper[2579]: E0709 22:14:42.998826 2579 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 09 22:14:42.999106 ip-10-0-136-251 kubenswrapper[2579]: E0709 22:14:42.998878 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/c1c63bd0-6360-4f30-9a66-f4255a633916-kube-rbac-proxy-cm podName:c1c63bd0-6360-4f30-9a66-f4255a633916 nodeName:}" failed. No retries permitted until 2026-07-09 22:14:50.998857447 +0000 UTC m=+282.910310134 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/c1c63bd0-6360-4f30-9a66-f4255a633916-kube-rbac-proxy-cm") pod "insights-runtime-extractor-7fv2q" (UID: "c1c63bd0-6360-4f30-9a66-f4255a633916") : configmap "kube-rbac-proxy" not found Jul 09 22:14:42.999106 ip-10-0-136-251 kubenswrapper[2579]: E0709 22:14:42.998907 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c1c63bd0-6360-4f30-9a66-f4255a633916-insights-runtime-extractor-tls podName:c1c63bd0-6360-4f30-9a66-f4255a633916 nodeName:}" failed. No retries permitted until 2026-07-09 22:14:50.998894525 +0000 UTC m=+282.910347209 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/c1c63bd0-6360-4f30-9a66-f4255a633916-insights-runtime-extractor-tls") pod "insights-runtime-extractor-7fv2q" (UID: "c1c63bd0-6360-4f30-9a66-f4255a633916") : secret "insights-runtime-extractor-tls" not found Jul 09 22:14:46.731939 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:14:46.731883 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/7144bb6f-5266-4f4e-9d78-5ad5aabbd2d4-samples-operator-tls\") pod \"cluster-samples-operator-56fcf86bf-r5t5q\" (UID: \"7144bb6f-5266-4f4e-9d78-5ad5aabbd2d4\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-r5t5q" Jul 09 22:14:46.734253 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:14:46.734229 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/7144bb6f-5266-4f4e-9d78-5ad5aabbd2d4-samples-operator-tls\") pod \"cluster-samples-operator-56fcf86bf-r5t5q\" (UID: \"7144bb6f-5266-4f4e-9d78-5ad5aabbd2d4\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-r5t5q" Jul 09 22:14:46.833274 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:14:46.833241 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/31587075-3ee2-43c7-9d0e-5340de9912a1-metrics-certs\") pod \"router-default-854859444c-bhmbj\" (UID: \"31587075-3ee2-43c7-9d0e-5340de9912a1\") " pod="openshift-ingress/router-default-854859444c-bhmbj" Jul 09 22:14:46.833491 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:14:46.833305 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/31587075-3ee2-43c7-9d0e-5340de9912a1-service-ca-bundle\") pod \"router-default-854859444c-bhmbj\" (UID: \"31587075-3ee2-43c7-9d0e-5340de9912a1\") " pod="openshift-ingress/router-default-854859444c-bhmbj" Jul 09 22:14:46.833556 ip-10-0-136-251 kubenswrapper[2579]: E0709 22:14:46.833502 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/31587075-3ee2-43c7-9d0e-5340de9912a1-service-ca-bundle podName:31587075-3ee2-43c7-9d0e-5340de9912a1 nodeName:}" failed. No retries permitted until 2026-07-09 22:15:02.833482939 +0000 UTC m=+294.744935620 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/31587075-3ee2-43c7-9d0e-5340de9912a1-service-ca-bundle") pod "router-default-854859444c-bhmbj" (UID: "31587075-3ee2-43c7-9d0e-5340de9912a1") : configmap references non-existent config key: service-ca.crt Jul 09 22:14:46.835576 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:14:46.835559 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/31587075-3ee2-43c7-9d0e-5340de9912a1-metrics-certs\") pod \"router-default-854859444c-bhmbj\" (UID: \"31587075-3ee2-43c7-9d0e-5340de9912a1\") " pod="openshift-ingress/router-default-854859444c-bhmbj" Jul 09 22:14:46.859550 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:14:46.859525 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-r5t5q" Jul 09 22:14:46.982634 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:14:46.982610 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-r5t5q"] Jul 09 22:14:47.373012 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:14:47.372980 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-r5t5q" event={"ID":"7144bb6f-5266-4f4e-9d78-5ad5aabbd2d4","Type":"ContainerStarted","Data":"f2c3788bf907065cce1faf4ad888b12adf6c6b08d3cc100889dfeeb37a2dafa9"} Jul 09 22:14:49.382172 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:14:49.382129 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-r5t5q" event={"ID":"7144bb6f-5266-4f4e-9d78-5ad5aabbd2d4","Type":"ContainerStarted","Data":"b6f82442f89ea628edb38b0a9b4b6d0c5d902f48a03c2f8667072e18e48b1038"} Jul 09 22:14:49.382528 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:14:49.382195 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-r5t5q" event={"ID":"7144bb6f-5266-4f4e-9d78-5ad5aabbd2d4","Type":"ContainerStarted","Data":"3cbeb8d11fdf68e2cd9b0f821a31d8063d593a1d13738ba0dbccbef7ec03d5ec"} Jul 09 22:14:49.399260 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:14:49.399214 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-r5t5q" podStartSLOduration=17.709788007 podStartE2EDuration="19.399201692s" totalImagesPullingTime="1.689413685s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 22:14:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 22:14:49.398284574 +0000 UTC m=+281.309737273" watchObservedRunningTime="2026-07-09 22:14:49.399201692 +0000 UTC m=+281.310654392" Jul 09 22:14:50.072362 ip-10-0-136-251 kubenswrapper[2579]: E0709 22:14:50.072311 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[registry-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-image-registry/image-registry-5957744766-cmxpg" podUID="e8d76e6f-c1a2-4c70-8c4e-ce0b0480d75a" Jul 09 22:14:50.072362 ip-10-0-136-251 kubenswrapper[2579]: E0709 22:14:50.072337 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-vptbn" podUID="5a6d8c82-ca6e-49b1-b660-78501b981273" Jul 09 22:14:50.385205 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:14:50.385115 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-vptbn" Jul 09 22:14:50.385205 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:14:50.385121 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5957744766-cmxpg" Jul 09 22:14:51.070394 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:14:51.070353 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/c1c63bd0-6360-4f30-9a66-f4255a633916-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-7fv2q\" (UID: \"c1c63bd0-6360-4f30-9a66-f4255a633916\") " pod="openshift-insights/insights-runtime-extractor-7fv2q" Jul 09 22:14:51.070585 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:14:51.070459 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c1c63bd0-6360-4f30-9a66-f4255a633916-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-7fv2q\" (UID: \"c1c63bd0-6360-4f30-9a66-f4255a633916\") " pod="openshift-insights/insights-runtime-extractor-7fv2q" Jul 09 22:14:51.070585 ip-10-0-136-251 kubenswrapper[2579]: E0709 22:14:51.070546 2579 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 09 22:14:51.070667 ip-10-0-136-251 kubenswrapper[2579]: E0709 22:14:51.070607 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c1c63bd0-6360-4f30-9a66-f4255a633916-insights-runtime-extractor-tls podName:c1c63bd0-6360-4f30-9a66-f4255a633916 nodeName:}" failed. No retries permitted until 2026-07-09 22:15:07.070593762 +0000 UTC m=+298.982046441 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/c1c63bd0-6360-4f30-9a66-f4255a633916-insights-runtime-extractor-tls") pod "insights-runtime-extractor-7fv2q" (UID: "c1c63bd0-6360-4f30-9a66-f4255a633916") : secret "insights-runtime-extractor-tls" not found Jul 09 22:14:51.070876 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:14:51.070854 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/c1c63bd0-6360-4f30-9a66-f4255a633916-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-7fv2q\" (UID: \"c1c63bd0-6360-4f30-9a66-f4255a633916\") " pod="openshift-insights/insights-runtime-extractor-7fv2q" Jul 09 22:14:53.489327 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:14:53.489284 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/5a6d8c82-ca6e-49b1-b660-78501b981273-metrics-tls\") pod \"dns-default-vptbn\" (UID: \"5a6d8c82-ca6e-49b1-b660-78501b981273\") " pod="openshift-dns/dns-default-vptbn" Jul 09 22:14:53.489770 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:14:53.489336 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/2b7da825-82cf-48d9-a4fb-50c6adbc34e0-networking-console-plugin-cert\") pod \"networking-console-plugin-6bffdf9778-kggxk\" (UID: \"2b7da825-82cf-48d9-a4fb-50c6adbc34e0\") " pod="openshift-network-console/networking-console-plugin-6bffdf9778-kggxk" Jul 09 22:14:53.489770 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:14:53.489408 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/e8d76e6f-c1a2-4c70-8c4e-ce0b0480d75a-registry-tls\") pod \"image-registry-5957744766-cmxpg\" (UID: \"e8d76e6f-c1a2-4c70-8c4e-ce0b0480d75a\") " pod="openshift-image-registry/image-registry-5957744766-cmxpg" Jul 09 22:14:53.491716 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:14:53.491689 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/5a6d8c82-ca6e-49b1-b660-78501b981273-metrics-tls\") pod \"dns-default-vptbn\" (UID: \"5a6d8c82-ca6e-49b1-b660-78501b981273\") " pod="openshift-dns/dns-default-vptbn" Jul 09 22:14:53.491848 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:14:53.491789 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/2b7da825-82cf-48d9-a4fb-50c6adbc34e0-networking-console-plugin-cert\") pod \"networking-console-plugin-6bffdf9778-kggxk\" (UID: \"2b7da825-82cf-48d9-a4fb-50c6adbc34e0\") " pod="openshift-network-console/networking-console-plugin-6bffdf9778-kggxk" Jul 09 22:14:53.491848 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:14:53.491809 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/e8d76e6f-c1a2-4c70-8c4e-ce0b0480d75a-registry-tls\") pod \"image-registry-5957744766-cmxpg\" (UID: \"e8d76e6f-c1a2-4c70-8c4e-ce0b0480d75a\") " pod="openshift-image-registry/image-registry-5957744766-cmxpg" Jul 09 22:14:53.563159 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:14:53.563128 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-br9j6\"" Jul 09 22:14:53.570563 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:14:53.570524 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-6bffdf9778-kggxk" Jul 09 22:14:53.689073 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:14:53.689044 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-r6x6z\"" Jul 09 22:14:53.689073 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:14:53.689075 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-jn276\"" Jul 09 22:14:53.697472 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:14:53.697447 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-vptbn" Jul 09 22:14:53.697617 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:14:53.697538 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5957744766-cmxpg" Jul 09 22:14:53.716067 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:14:53.716039 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-6bffdf9778-kggxk"] Jul 09 22:14:53.719396 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:14:53.719367 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2b7da825_82cf_48d9_a4fb_50c6adbc34e0.slice/crio-3c03a99fc8be94552724e9949599edb8f9794d148378c8a55f2276c0fd5625f9 WatchSource:0}: Error finding container 3c03a99fc8be94552724e9949599edb8f9794d148378c8a55f2276c0fd5625f9: Status 404 returned error can't find the container with id 3c03a99fc8be94552724e9949599edb8f9794d148378c8a55f2276c0fd5625f9 Jul 09 22:14:53.872716 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:14:53.872691 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-5957744766-cmxpg"] Jul 09 22:14:53.874349 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:14:53.874323 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode8d76e6f_c1a2_4c70_8c4e_ce0b0480d75a.slice/crio-579f930798f03de904629c2b7689c5fc3535d2e4cafc9743e8de6a739fe897ed WatchSource:0}: Error finding container 579f930798f03de904629c2b7689c5fc3535d2e4cafc9743e8de6a739fe897ed: Status 404 returned error can't find the container with id 579f930798f03de904629c2b7689c5fc3535d2e4cafc9743e8de6a739fe897ed Jul 09 22:14:53.874720 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:14:53.874700 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-vptbn"] Jul 09 22:14:53.877579 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:14:53.877549 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5a6d8c82_ca6e_49b1_b660_78501b981273.slice/crio-a678e6585d44e5df2fbd0b76eae56c97952b14ede7bd0b7a767c6a3726fae250 WatchSource:0}: Error finding container a678e6585d44e5df2fbd0b76eae56c97952b14ede7bd0b7a767c6a3726fae250: Status 404 returned error can't find the container with id a678e6585d44e5df2fbd0b76eae56c97952b14ede7bd0b7a767c6a3726fae250 Jul 09 22:14:54.399547 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:14:54.399506 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-vptbn" event={"ID":"5a6d8c82-ca6e-49b1-b660-78501b981273","Type":"ContainerStarted","Data":"a678e6585d44e5df2fbd0b76eae56c97952b14ede7bd0b7a767c6a3726fae250"} Jul 09 22:14:54.400811 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:14:54.400785 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5957744766-cmxpg" event={"ID":"e8d76e6f-c1a2-4c70-8c4e-ce0b0480d75a","Type":"ContainerStarted","Data":"1dddb73c32c9d749193d8fada82d30f394177f3ac6365bf38c231f926bc543d4"} Jul 09 22:14:54.400908 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:14:54.400827 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5957744766-cmxpg" event={"ID":"e8d76e6f-c1a2-4c70-8c4e-ce0b0480d75a","Type":"ContainerStarted","Data":"579f930798f03de904629c2b7689c5fc3535d2e4cafc9743e8de6a739fe897ed"} Jul 09 22:14:54.400908 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:14:54.400872 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-5957744766-cmxpg" Jul 09 22:14:54.401882 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:14:54.401863 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-6bffdf9778-kggxk" event={"ID":"2b7da825-82cf-48d9-a4fb-50c6adbc34e0","Type":"ContainerStarted","Data":"3c03a99fc8be94552724e9949599edb8f9794d148378c8a55f2276c0fd5625f9"} Jul 09 22:14:54.431772 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:14:54.431720 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-5957744766-cmxpg" podStartSLOduration=271.431706144 podStartE2EDuration="4m31.431706144s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 22:10:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 22:14:54.431684385 +0000 UTC m=+286.343137085" watchObservedRunningTime="2026-07-09 22:14:54.431706144 +0000 UTC m=+286.343158841" Jul 09 22:14:56.411005 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:14:56.410969 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-6bffdf9778-kggxk" event={"ID":"2b7da825-82cf-48d9-a4fb-50c6adbc34e0","Type":"ContainerStarted","Data":"7d5c7e38b5003bb8033f2666856738c24386e02f2e783269c5185f8aa090fd5f"} Jul 09 22:14:56.439513 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:14:56.439437 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-console/networking-console-plugin-6bffdf9778-kggxk" podStartSLOduration=269.468311285 podStartE2EDuration="4m31.439413923s" totalImagesPullingTime="1.971102638s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 22:10:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 22:14:56.437902777 +0000 UTC m=+288.349355477" watchObservedRunningTime="2026-07-09 22:14:56.439413923 +0000 UTC m=+288.350866624" Jul 09 22:14:57.415514 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:14:57.415470 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-vptbn" event={"ID":"5a6d8c82-ca6e-49b1-b660-78501b981273","Type":"ContainerStarted","Data":"8eb0b0b4dca94d344be04665db39c0865489087573d3900c84ddd09f14c06ae5"} Jul 09 22:14:57.415514 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:14:57.415524 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-vptbn" event={"ID":"5a6d8c82-ca6e-49b1-b660-78501b981273","Type":"ContainerStarted","Data":"e8e355253c7006084b9a901d4978b6efc08cd3795e2e6f0e47d8655009b80bff"} Jul 09 22:14:57.416058 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:14:57.415610 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-vptbn" Jul 09 22:14:57.440365 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:14:57.440311 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-vptbn" podStartSLOduration=251.752050405 podStartE2EDuration="4m14.440296133s" totalImagesPullingTime="2.688245728s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 22:10:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 22:14:57.437319741 +0000 UTC m=+289.348772441" watchObservedRunningTime="2026-07-09 22:14:57.440296133 +0000 UTC m=+289.351748832" Jul 09 22:15:02.130132 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:15:02.130090 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-685bfb5f64-4jqjt"] Jul 09 22:15:02.133317 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:15:02.133292 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-685bfb5f64-4jqjt" Jul 09 22:15:02.136439 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:15:02.136420 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Jul 09 22:15:02.136576 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:15:02.136470 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"default-dockercfg-pq6qf\"" Jul 09 22:15:02.136961 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:15:02.136945 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Jul 09 22:15:02.152156 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:15:02.152128 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-685bfb5f64-4jqjt"] Jul 09 22:15:02.154991 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:15:02.154959 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6csmv\" (UniqueName: \"kubernetes.io/projected/4d3d7a7e-833c-4557-9833-f03aeec52a95-kube-api-access-6csmv\") pod \"downloads-685bfb5f64-4jqjt\" (UID: \"4d3d7a7e-833c-4557-9833-f03aeec52a95\") " pod="openshift-console/downloads-685bfb5f64-4jqjt" Jul 09 22:15:02.256163 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:15:02.256127 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6csmv\" (UniqueName: \"kubernetes.io/projected/4d3d7a7e-833c-4557-9833-f03aeec52a95-kube-api-access-6csmv\") pod \"downloads-685bfb5f64-4jqjt\" (UID: \"4d3d7a7e-833c-4557-9833-f03aeec52a95\") " pod="openshift-console/downloads-685bfb5f64-4jqjt" Jul 09 22:15:02.288359 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:15:02.288325 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6csmv\" (UniqueName: \"kubernetes.io/projected/4d3d7a7e-833c-4557-9833-f03aeec52a95-kube-api-access-6csmv\") pod \"downloads-685bfb5f64-4jqjt\" (UID: \"4d3d7a7e-833c-4557-9833-f03aeec52a95\") " pod="openshift-console/downloads-685bfb5f64-4jqjt" Jul 09 22:15:02.443075 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:15:02.442981 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-685bfb5f64-4jqjt" Jul 09 22:15:02.559310 ip-10-0-136-251 kubenswrapper[2579]: E0709 22:15:02.559267 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-certs], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-multus/network-metrics-daemon-8pj95" podUID="23b023ec-a254-462d-a0b4-5c22fd905e72" Jul 09 22:15:02.579507 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:15:02.579471 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-685bfb5f64-4jqjt"] Jul 09 22:15:02.861584 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:15:02.861543 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/31587075-3ee2-43c7-9d0e-5340de9912a1-service-ca-bundle\") pod \"router-default-854859444c-bhmbj\" (UID: \"31587075-3ee2-43c7-9d0e-5340de9912a1\") " pod="openshift-ingress/router-default-854859444c-bhmbj" Jul 09 22:15:02.862231 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:15:02.862213 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/31587075-3ee2-43c7-9d0e-5340de9912a1-service-ca-bundle\") pod \"router-default-854859444c-bhmbj\" (UID: \"31587075-3ee2-43c7-9d0e-5340de9912a1\") " pod="openshift-ingress/router-default-854859444c-bhmbj" Jul 09 22:15:03.064909 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:15:03.064869 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-854859444c-bhmbj" Jul 09 22:15:03.238707 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:15:03.238682 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-854859444c-bhmbj"] Jul 09 22:15:03.240392 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:15:03.240362 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod31587075_3ee2_43c7_9d0e_5340de9912a1.slice/crio-d79db1580f50ac398f5d6b364acc3c56a298df9942e46f8c3059ee38577c50e9 WatchSource:0}: Error finding container d79db1580f50ac398f5d6b364acc3c56a298df9942e46f8c3059ee38577c50e9: Status 404 returned error can't find the container with id d79db1580f50ac398f5d6b364acc3c56a298df9942e46f8c3059ee38577c50e9 Jul 09 22:15:03.435822 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:15:03.435732 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-854859444c-bhmbj" event={"ID":"31587075-3ee2-43c7-9d0e-5340de9912a1","Type":"ContainerStarted","Data":"6b61e4b21b2094aa70d1e5711707018ba034842a2b60e68affedde1a678204cd"} Jul 09 22:15:03.436011 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:15:03.435834 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-854859444c-bhmbj" event={"ID":"31587075-3ee2-43c7-9d0e-5340de9912a1","Type":"ContainerStarted","Data":"d79db1580f50ac398f5d6b364acc3c56a298df9942e46f8c3059ee38577c50e9"} Jul 09 22:15:03.437311 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:15:03.437279 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-685bfb5f64-4jqjt" event={"ID":"4d3d7a7e-833c-4557-9833-f03aeec52a95","Type":"ContainerStarted","Data":"5dbcf4d0ee7d703a3e34911292d5e2eac18e601a884f0b5124e43e11a44bebc9"} Jul 09 22:15:03.479430 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:15:03.479367 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/router-default-854859444c-bhmbj" podStartSLOduration=33.479347117 podStartE2EDuration="33.479347117s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 22:14:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 22:15:03.479150672 +0000 UTC m=+295.390603372" watchObservedRunningTime="2026-07-09 22:15:03.479347117 +0000 UTC m=+295.390799821" Jul 09 22:15:04.065101 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:15:04.065060 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-854859444c-bhmbj" Jul 09 22:15:04.067957 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:15:04.067935 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-854859444c-bhmbj" Jul 09 22:15:04.441270 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:15:04.441189 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-854859444c-bhmbj" Jul 09 22:15:04.442465 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:15:04.442446 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-854859444c-bhmbj" Jul 09 22:15:07.092759 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:15:07.092724 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c1c63bd0-6360-4f30-9a66-f4255a633916-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-7fv2q\" (UID: \"c1c63bd0-6360-4f30-9a66-f4255a633916\") " pod="openshift-insights/insights-runtime-extractor-7fv2q" Jul 09 22:15:07.095171 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:15:07.095146 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c1c63bd0-6360-4f30-9a66-f4255a633916-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-7fv2q\" (UID: \"c1c63bd0-6360-4f30-9a66-f4255a633916\") " pod="openshift-insights/insights-runtime-extractor-7fv2q" Jul 09 22:15:07.293440 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:15:07.293406 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-7fv2q" Jul 09 22:15:07.420914 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:15:07.420881 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-vptbn" Jul 09 22:15:07.461227 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:15:07.461202 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-7fv2q"] Jul 09 22:15:07.463506 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:15:07.463475 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc1c63bd0_6360_4f30_9a66_f4255a633916.slice/crio-e172761ec140a06541a4b6cbacbf57bf3a3c4fd839e2bcb44c99c17f537389ad WatchSource:0}: Error finding container e172761ec140a06541a4b6cbacbf57bf3a3c4fd839e2bcb44c99c17f537389ad: Status 404 returned error can't find the container with id e172761ec140a06541a4b6cbacbf57bf3a3c4fd839e2bcb44c99c17f537389ad Jul 09 22:15:08.397001 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:15:08.396967 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-7445556d44-f87lt"] Jul 09 22:15:08.403956 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:15:08.400899 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7445556d44-f87lt" Jul 09 22:15:08.405252 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:15:08.405224 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Jul 09 22:15:08.405375 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:15:08.405286 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Jul 09 22:15:08.406046 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:15:08.406024 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Jul 09 22:15:08.406366 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:15:08.406320 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Jul 09 22:15:08.406645 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:15:08.406604 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-l6k92\"" Jul 09 22:15:08.406907 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:15:08.406892 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Jul 09 22:15:08.414426 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:15:08.414400 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7445556d44-f87lt"] Jul 09 22:15:08.465176 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:15:08.464816 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-7fv2q" event={"ID":"c1c63bd0-6360-4f30-9a66-f4255a633916","Type":"ContainerStarted","Data":"e172761ec140a06541a4b6cbacbf57bf3a3c4fd839e2bcb44c99c17f537389ad"} Jul 09 22:15:08.495158 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:15:08.495108 2579 kubelet.go:1648] "Image garbage collection succeeded" Jul 09 22:15:08.507001 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:15:08.506844 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/3a476bd5-1f8c-47dd-8fac-c44d2dcea62d-service-ca\") pod \"console-7445556d44-f87lt\" (UID: \"3a476bd5-1f8c-47dd-8fac-c44d2dcea62d\") " pod="openshift-console/console-7445556d44-f87lt" Jul 09 22:15:08.507001 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:15:08.506892 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/3a476bd5-1f8c-47dd-8fac-c44d2dcea62d-oauth-serving-cert\") pod \"console-7445556d44-f87lt\" (UID: \"3a476bd5-1f8c-47dd-8fac-c44d2dcea62d\") " pod="openshift-console/console-7445556d44-f87lt" Jul 09 22:15:08.507001 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:15:08.506954 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/3a476bd5-1f8c-47dd-8fac-c44d2dcea62d-console-serving-cert\") pod \"console-7445556d44-f87lt\" (UID: \"3a476bd5-1f8c-47dd-8fac-c44d2dcea62d\") " pod="openshift-console/console-7445556d44-f87lt" Jul 09 22:15:08.507276 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:15:08.507077 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bsdjz\" (UniqueName: \"kubernetes.io/projected/3a476bd5-1f8c-47dd-8fac-c44d2dcea62d-kube-api-access-bsdjz\") pod \"console-7445556d44-f87lt\" (UID: \"3a476bd5-1f8c-47dd-8fac-c44d2dcea62d\") " pod="openshift-console/console-7445556d44-f87lt" Jul 09 22:15:08.507276 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:15:08.507177 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/3a476bd5-1f8c-47dd-8fac-c44d2dcea62d-console-config\") pod \"console-7445556d44-f87lt\" (UID: \"3a476bd5-1f8c-47dd-8fac-c44d2dcea62d\") " pod="openshift-console/console-7445556d44-f87lt" Jul 09 22:15:08.507276 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:15:08.507226 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/3a476bd5-1f8c-47dd-8fac-c44d2dcea62d-console-oauth-config\") pod \"console-7445556d44-f87lt\" (UID: \"3a476bd5-1f8c-47dd-8fac-c44d2dcea62d\") " pod="openshift-console/console-7445556d44-f87lt" Jul 09 22:15:08.608311 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:15:08.608142 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/3a476bd5-1f8c-47dd-8fac-c44d2dcea62d-console-serving-cert\") pod \"console-7445556d44-f87lt\" (UID: \"3a476bd5-1f8c-47dd-8fac-c44d2dcea62d\") " pod="openshift-console/console-7445556d44-f87lt" Jul 09 22:15:08.608311 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:15:08.608203 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bsdjz\" (UniqueName: \"kubernetes.io/projected/3a476bd5-1f8c-47dd-8fac-c44d2dcea62d-kube-api-access-bsdjz\") pod \"console-7445556d44-f87lt\" (UID: \"3a476bd5-1f8c-47dd-8fac-c44d2dcea62d\") " pod="openshift-console/console-7445556d44-f87lt" Jul 09 22:15:08.608311 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:15:08.608269 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/3a476bd5-1f8c-47dd-8fac-c44d2dcea62d-console-config\") pod \"console-7445556d44-f87lt\" (UID: \"3a476bd5-1f8c-47dd-8fac-c44d2dcea62d\") " pod="openshift-console/console-7445556d44-f87lt" Jul 09 22:15:08.608311 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:15:08.608311 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/3a476bd5-1f8c-47dd-8fac-c44d2dcea62d-console-oauth-config\") pod \"console-7445556d44-f87lt\" (UID: \"3a476bd5-1f8c-47dd-8fac-c44d2dcea62d\") " pod="openshift-console/console-7445556d44-f87lt" Jul 09 22:15:08.609225 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:15:08.608366 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/3a476bd5-1f8c-47dd-8fac-c44d2dcea62d-service-ca\") pod \"console-7445556d44-f87lt\" (UID: \"3a476bd5-1f8c-47dd-8fac-c44d2dcea62d\") " pod="openshift-console/console-7445556d44-f87lt" Jul 09 22:15:08.609225 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:15:08.608394 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/3a476bd5-1f8c-47dd-8fac-c44d2dcea62d-oauth-serving-cert\") pod \"console-7445556d44-f87lt\" (UID: \"3a476bd5-1f8c-47dd-8fac-c44d2dcea62d\") " pod="openshift-console/console-7445556d44-f87lt" Jul 09 22:15:08.611684 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:15:08.611655 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Jul 09 22:15:08.611894 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:15:08.611865 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Jul 09 22:15:08.612029 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:15:08.611909 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Jul 09 22:15:08.612569 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:15:08.612204 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Jul 09 22:15:08.612569 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:15:08.612389 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Jul 09 22:15:08.619874 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:15:08.619847 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/3a476bd5-1f8c-47dd-8fac-c44d2dcea62d-console-config\") pod \"console-7445556d44-f87lt\" (UID: \"3a476bd5-1f8c-47dd-8fac-c44d2dcea62d\") " pod="openshift-console/console-7445556d44-f87lt" Jul 09 22:15:08.620001 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:15:08.619847 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/3a476bd5-1f8c-47dd-8fac-c44d2dcea62d-service-ca\") pod \"console-7445556d44-f87lt\" (UID: \"3a476bd5-1f8c-47dd-8fac-c44d2dcea62d\") " pod="openshift-console/console-7445556d44-f87lt" Jul 09 22:15:08.620001 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:15:08.619936 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/3a476bd5-1f8c-47dd-8fac-c44d2dcea62d-oauth-serving-cert\") pod \"console-7445556d44-f87lt\" (UID: \"3a476bd5-1f8c-47dd-8fac-c44d2dcea62d\") " pod="openshift-console/console-7445556d44-f87lt" Jul 09 22:15:08.626593 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:15:08.622721 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/3a476bd5-1f8c-47dd-8fac-c44d2dcea62d-console-oauth-config\") pod \"console-7445556d44-f87lt\" (UID: \"3a476bd5-1f8c-47dd-8fac-c44d2dcea62d\") " pod="openshift-console/console-7445556d44-f87lt" Jul 09 22:15:08.628631 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:15:08.628585 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bsdjz\" (UniqueName: \"kubernetes.io/projected/3a476bd5-1f8c-47dd-8fac-c44d2dcea62d-kube-api-access-bsdjz\") pod \"console-7445556d44-f87lt\" (UID: \"3a476bd5-1f8c-47dd-8fac-c44d2dcea62d\") " pod="openshift-console/console-7445556d44-f87lt" Jul 09 22:15:08.629204 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:15:08.629076 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/3a476bd5-1f8c-47dd-8fac-c44d2dcea62d-console-serving-cert\") pod \"console-7445556d44-f87lt\" (UID: \"3a476bd5-1f8c-47dd-8fac-c44d2dcea62d\") " pod="openshift-console/console-7445556d44-f87lt" Jul 09 22:15:08.735933 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:15:08.735825 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-l6k92\"" Jul 09 22:15:08.743660 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:15:08.743576 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7445556d44-f87lt" Jul 09 22:15:08.908422 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:15:08.908344 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7445556d44-f87lt"] Jul 09 22:15:08.912359 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:15:08.912328 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3a476bd5_1f8c_47dd_8fac_c44d2dcea62d.slice/crio-de832b196adf67e28d021a488d729ab031951364ca6b675e8b8a47683ba64ffa WatchSource:0}: Error finding container de832b196adf67e28d021a488d729ab031951364ca6b675e8b8a47683ba64ffa: Status 404 returned error can't find the container with id de832b196adf67e28d021a488d729ab031951364ca6b675e8b8a47683ba64ffa Jul 09 22:15:09.056167 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:15:09.056130 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-567f7b8b7c-d8kq7"] Jul 09 22:15:09.060129 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:15:09.060104 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-567f7b8b7c-d8kq7" Jul 09 22:15:09.063238 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:15:09.063072 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Jul 09 22:15:09.063238 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:15:09.063118 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Jul 09 22:15:09.063437 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:15:09.063252 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\"" Jul 09 22:15:09.063437 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:15:09.063314 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\"" Jul 09 22:15:09.063437 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:15:09.063345 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-64sfw\"" Jul 09 22:15:09.063594 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:15:09.063520 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Jul 09 22:15:09.071593 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:15:09.071570 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-567f7b8b7c-d8kq7"] Jul 09 22:15:09.112818 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:15:09.112783 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/04192941-26ac-4e9e-a52a-381d544238d2-metrics-client-ca\") pod \"prometheus-operator-567f7b8b7c-d8kq7\" (UID: \"04192941-26ac-4e9e-a52a-381d544238d2\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-d8kq7" Jul 09 22:15:09.112818 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:15:09.112821 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/04192941-26ac-4e9e-a52a-381d544238d2-prometheus-operator-tls\") pod \"prometheus-operator-567f7b8b7c-d8kq7\" (UID: \"04192941-26ac-4e9e-a52a-381d544238d2\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-d8kq7" Jul 09 22:15:09.113148 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:15:09.112866 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/04192941-26ac-4e9e-a52a-381d544238d2-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-567f7b8b7c-d8kq7\" (UID: \"04192941-26ac-4e9e-a52a-381d544238d2\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-d8kq7" Jul 09 22:15:09.113148 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:15:09.112914 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4bkpx\" (UniqueName: \"kubernetes.io/projected/04192941-26ac-4e9e-a52a-381d544238d2-kube-api-access-4bkpx\") pod \"prometheus-operator-567f7b8b7c-d8kq7\" (UID: \"04192941-26ac-4e9e-a52a-381d544238d2\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-d8kq7" Jul 09 22:15:09.213477 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:15:09.213442 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/04192941-26ac-4e9e-a52a-381d544238d2-metrics-client-ca\") pod \"prometheus-operator-567f7b8b7c-d8kq7\" (UID: \"04192941-26ac-4e9e-a52a-381d544238d2\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-d8kq7" Jul 09 22:15:09.213477 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:15:09.213482 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/04192941-26ac-4e9e-a52a-381d544238d2-prometheus-operator-tls\") pod \"prometheus-operator-567f7b8b7c-d8kq7\" (UID: \"04192941-26ac-4e9e-a52a-381d544238d2\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-d8kq7" Jul 09 22:15:09.213742 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:15:09.213535 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/04192941-26ac-4e9e-a52a-381d544238d2-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-567f7b8b7c-d8kq7\" (UID: \"04192941-26ac-4e9e-a52a-381d544238d2\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-d8kq7" Jul 09 22:15:09.213742 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:15:09.213568 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4bkpx\" (UniqueName: \"kubernetes.io/projected/04192941-26ac-4e9e-a52a-381d544238d2-kube-api-access-4bkpx\") pod \"prometheus-operator-567f7b8b7c-d8kq7\" (UID: \"04192941-26ac-4e9e-a52a-381d544238d2\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-d8kq7" Jul 09 22:15:09.213742 ip-10-0-136-251 kubenswrapper[2579]: E0709 22:15:09.213668 2579 secret.go:189] Couldn't get secret openshift-monitoring/prometheus-operator-tls: secret "prometheus-operator-tls" not found Jul 09 22:15:09.213742 ip-10-0-136-251 kubenswrapper[2579]: E0709 22:15:09.213741 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/04192941-26ac-4e9e-a52a-381d544238d2-prometheus-operator-tls podName:04192941-26ac-4e9e-a52a-381d544238d2 nodeName:}" failed. No retries permitted until 2026-07-09 22:15:09.71371953 +0000 UTC m=+301.625172229 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "prometheus-operator-tls" (UniqueName: "kubernetes.io/secret/04192941-26ac-4e9e-a52a-381d544238d2-prometheus-operator-tls") pod "prometheus-operator-567f7b8b7c-d8kq7" (UID: "04192941-26ac-4e9e-a52a-381d544238d2") : secret "prometheus-operator-tls" not found Jul 09 22:15:09.214310 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:15:09.214283 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/04192941-26ac-4e9e-a52a-381d544238d2-metrics-client-ca\") pod \"prometheus-operator-567f7b8b7c-d8kq7\" (UID: \"04192941-26ac-4e9e-a52a-381d544238d2\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-d8kq7" Jul 09 22:15:09.216522 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:15:09.216498 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/04192941-26ac-4e9e-a52a-381d544238d2-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-567f7b8b7c-d8kq7\" (UID: \"04192941-26ac-4e9e-a52a-381d544238d2\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-d8kq7" Jul 09 22:15:09.223340 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:15:09.223316 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4bkpx\" (UniqueName: \"kubernetes.io/projected/04192941-26ac-4e9e-a52a-381d544238d2-kube-api-access-4bkpx\") pod \"prometheus-operator-567f7b8b7c-d8kq7\" (UID: \"04192941-26ac-4e9e-a52a-381d544238d2\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-d8kq7" Jul 09 22:15:09.471558 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:15:09.471461 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7445556d44-f87lt" event={"ID":"3a476bd5-1f8c-47dd-8fac-c44d2dcea62d","Type":"ContainerStarted","Data":"de832b196adf67e28d021a488d729ab031951364ca6b675e8b8a47683ba64ffa"} Jul 09 22:15:09.717911 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:15:09.717870 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/04192941-26ac-4e9e-a52a-381d544238d2-prometheus-operator-tls\") pod \"prometheus-operator-567f7b8b7c-d8kq7\" (UID: \"04192941-26ac-4e9e-a52a-381d544238d2\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-d8kq7" Jul 09 22:15:09.720793 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:15:09.720752 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/04192941-26ac-4e9e-a52a-381d544238d2-prometheus-operator-tls\") pod \"prometheus-operator-567f7b8b7c-d8kq7\" (UID: \"04192941-26ac-4e9e-a52a-381d544238d2\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-d8kq7" Jul 09 22:15:09.974093 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:15:09.974049 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-567f7b8b7c-d8kq7" Jul 09 22:15:12.086176 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:15:12.086150 2579 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 09 22:15:12.105251 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:15:12.104645 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-567f7b8b7c-d8kq7"] Jul 09 22:15:12.108129 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:15:12.108093 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod04192941_26ac_4e9e_a52a_381d544238d2.slice/crio-1c60b67831e3d6ba6ce2140b1090183707135281454ef6d06fc3c47adfff4371 WatchSource:0}: Error finding container 1c60b67831e3d6ba6ce2140b1090183707135281454ef6d06fc3c47adfff4371: Status 404 returned error can't find the container with id 1c60b67831e3d6ba6ce2140b1090183707135281454ef6d06fc3c47adfff4371 Jul 09 22:15:12.484583 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:15:12.484541 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-7fv2q" event={"ID":"c1c63bd0-6360-4f30-9a66-f4255a633916","Type":"ContainerStarted","Data":"3633518b27ed75e8a36b8debd2bfa3708394e13217798ba2608a9eeea4ebb8bb"} Jul 09 22:15:12.486117 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:15:12.486075 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-567f7b8b7c-d8kq7" event={"ID":"04192941-26ac-4e9e-a52a-381d544238d2","Type":"ContainerStarted","Data":"1c60b67831e3d6ba6ce2140b1090183707135281454ef6d06fc3c47adfff4371"} Jul 09 22:15:13.493747 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:15:13.493706 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-7fv2q" event={"ID":"c1c63bd0-6360-4f30-9a66-f4255a633916","Type":"ContainerStarted","Data":"f5a5cb4933872be2494cc34617291cd3a9eb59cbd546894dc3de1ffb17d74e7e"} Jul 09 22:15:13.704126 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:15:13.704072 2579 patch_prober.go:28] interesting pod/image-registry-5957744766-cmxpg container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]} Jul 09 22:15:13.704311 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:15:13.704162 2579 prober.go:120] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-5957744766-cmxpg" podUID="e8d76e6f-c1a2-4c70-8c4e-ce0b0480d75a" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 09 22:15:15.411179 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:15:15.411145 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-5957744766-cmxpg" Jul 09 22:15:17.559137 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:15:17.559098 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8pj95" Jul 09 22:15:23.533283 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:15:23.533242 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-567f7b8b7c-d8kq7" event={"ID":"04192941-26ac-4e9e-a52a-381d544238d2","Type":"ContainerStarted","Data":"60b43137c7db827d3cca3fa16522d8ac34e352edbe10d282e4db52aa888299ba"} Jul 09 22:15:23.533812 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:15:23.533350 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-567f7b8b7c-d8kq7" event={"ID":"04192941-26ac-4e9e-a52a-381d544238d2","Type":"ContainerStarted","Data":"36c1eb3a9fbbb1c54b688a7812fb10d3d8ce887fd9f72104d5a0706a3da516e1"} Jul 09 22:15:23.535061 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:15:23.535033 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-685bfb5f64-4jqjt" event={"ID":"4d3d7a7e-833c-4557-9833-f03aeec52a95","Type":"ContainerStarted","Data":"2c5a7c971463522c1f4dff6bc3d8a9185d85d2714473dbae2757ca24349f9aba"} Jul 09 22:15:23.535187 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:15:23.535168 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-685bfb5f64-4jqjt" Jul 09 22:15:23.537042 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:15:23.537012 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7445556d44-f87lt" event={"ID":"3a476bd5-1f8c-47dd-8fac-c44d2dcea62d","Type":"ContainerStarted","Data":"36b512474158f14a2bf4998812aebb0410cdda0c68dcf790b5f27641d4c158b8"} Jul 09 22:15:23.539482 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:15:23.539460 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-7fv2q" event={"ID":"c1c63bd0-6360-4f30-9a66-f4255a633916","Type":"ContainerStarted","Data":"52ee2c66dd4a443076376d5e88d11a00072991f9997dccdb3dd1e78beda737e6"} Jul 09 22:15:23.554415 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:15:23.554350 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-567f7b8b7c-d8kq7" podStartSLOduration=3.841764542 podStartE2EDuration="14.554326835s" totalImagesPullingTime="10.712562293s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 22:15:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 22:15:23.552946803 +0000 UTC m=+315.464399528" watchObservedRunningTime="2026-07-09 22:15:23.554326835 +0000 UTC m=+315.465779537" Jul 09 22:15:23.556905 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:15:23.556881 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-685bfb5f64-4jqjt" Jul 09 22:15:23.576050 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:15:23.576002 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-7fv2q" podStartSLOduration=33.293356987 podStartE2EDuration="48.575985529s" totalImagesPullingTime="15.282628542s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 22:14:35 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 22:15:23.574690558 +0000 UTC m=+315.486143281" watchObservedRunningTime="2026-07-09 22:15:23.575985529 +0000 UTC m=+315.487438230" Jul 09 22:15:23.595677 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:15:23.595625 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/downloads-685bfb5f64-4jqjt" podStartSLOduration=1.358310333 podStartE2EDuration="21.595604085s" totalImagesPullingTime="20.237293752s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 22:15:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 22:15:23.593775072 +0000 UTC m=+315.505227777" watchObservedRunningTime="2026-07-09 22:15:23.595604085 +0000 UTC m=+315.507056788" Jul 09 22:15:23.615277 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:15:23.615218 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-7445556d44-f87lt" podStartSLOduration=1.707540996 podStartE2EDuration="15.615196068s" totalImagesPullingTime="13.907655072s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 22:15:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 22:15:23.614159891 +0000 UTC m=+315.525612615" watchObservedRunningTime="2026-07-09 22:15:23.615196068 +0000 UTC m=+315.526648770" Jul 09 22:15:24.179183 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:15:24.179142 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-5957744766-cmxpg"] Jul 09 22:15:26.476956 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:15:26.476906 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7445556d44-f87lt"] Jul 09 22:15:28.744790 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:15:28.744742 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-7445556d44-f87lt" Jul 09 22:15:28.950127 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:15:28.950091 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-qcsd5"] Jul 09 22:15:29.007545 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:15:29.007464 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-qcsd5" Jul 09 22:15:29.017890 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:15:29.017865 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Jul 09 22:15:29.018062 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:15:29.018001 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Jul 09 22:15:29.018062 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:15:29.018038 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-tq6cx\"" Jul 09 22:15:29.018402 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:15:29.018381 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Jul 09 22:15:29.093462 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:15:29.093393 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/f9a132fa-2995-418a-9f53-19ecdf6ec341-node-exporter-tls\") pod \"node-exporter-qcsd5\" (UID: \"f9a132fa-2995-418a-9f53-19ecdf6ec341\") " pod="openshift-monitoring/node-exporter-qcsd5" Jul 09 22:15:29.093462 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:15:29.093467 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/f9a132fa-2995-418a-9f53-19ecdf6ec341-node-exporter-wtmp\") pod \"node-exporter-qcsd5\" (UID: \"f9a132fa-2995-418a-9f53-19ecdf6ec341\") " pod="openshift-monitoring/node-exporter-qcsd5" Jul 09 22:15:29.093726 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:15:29.093497 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f9a132fa-2995-418a-9f53-19ecdf6ec341-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-qcsd5\" (UID: \"f9a132fa-2995-418a-9f53-19ecdf6ec341\") " pod="openshift-monitoring/node-exporter-qcsd5" Jul 09 22:15:29.093726 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:15:29.093592 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nt7c5\" (UniqueName: \"kubernetes.io/projected/f9a132fa-2995-418a-9f53-19ecdf6ec341-kube-api-access-nt7c5\") pod \"node-exporter-qcsd5\" (UID: \"f9a132fa-2995-418a-9f53-19ecdf6ec341\") " pod="openshift-monitoring/node-exporter-qcsd5" Jul 09 22:15:29.093726 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:15:29.093629 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f9a132fa-2995-418a-9f53-19ecdf6ec341-metrics-client-ca\") pod \"node-exporter-qcsd5\" (UID: \"f9a132fa-2995-418a-9f53-19ecdf6ec341\") " pod="openshift-monitoring/node-exporter-qcsd5" Jul 09 22:15:29.093726 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:15:29.093716 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/f9a132fa-2995-418a-9f53-19ecdf6ec341-node-exporter-textfile\") pod \"node-exporter-qcsd5\" (UID: \"f9a132fa-2995-418a-9f53-19ecdf6ec341\") " pod="openshift-monitoring/node-exporter-qcsd5" Jul 09 22:15:29.093900 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:15:29.093749 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f9a132fa-2995-418a-9f53-19ecdf6ec341-sys\") pod \"node-exporter-qcsd5\" (UID: \"f9a132fa-2995-418a-9f53-19ecdf6ec341\") " pod="openshift-monitoring/node-exporter-qcsd5" Jul 09 22:15:29.093900 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:15:29.093831 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/f9a132fa-2995-418a-9f53-19ecdf6ec341-node-exporter-accelerators-collector-config\") pod \"node-exporter-qcsd5\" (UID: \"f9a132fa-2995-418a-9f53-19ecdf6ec341\") " pod="openshift-monitoring/node-exporter-qcsd5" Jul 09 22:15:29.093900 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:15:29.093859 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/f9a132fa-2995-418a-9f53-19ecdf6ec341-root\") pod \"node-exporter-qcsd5\" (UID: \"f9a132fa-2995-418a-9f53-19ecdf6ec341\") " pod="openshift-monitoring/node-exporter-qcsd5" Jul 09 22:15:29.194346 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:15:29.194310 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/f9a132fa-2995-418a-9f53-19ecdf6ec341-node-exporter-tls\") pod \"node-exporter-qcsd5\" (UID: \"f9a132fa-2995-418a-9f53-19ecdf6ec341\") " pod="openshift-monitoring/node-exporter-qcsd5" Jul 09 22:15:29.194523 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:15:29.194353 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/f9a132fa-2995-418a-9f53-19ecdf6ec341-node-exporter-wtmp\") pod \"node-exporter-qcsd5\" (UID: \"f9a132fa-2995-418a-9f53-19ecdf6ec341\") " pod="openshift-monitoring/node-exporter-qcsd5" Jul 09 22:15:29.194523 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:15:29.194379 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f9a132fa-2995-418a-9f53-19ecdf6ec341-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-qcsd5\" (UID: \"f9a132fa-2995-418a-9f53-19ecdf6ec341\") " pod="openshift-monitoring/node-exporter-qcsd5" Jul 09 22:15:29.194523 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:15:29.194407 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nt7c5\" (UniqueName: \"kubernetes.io/projected/f9a132fa-2995-418a-9f53-19ecdf6ec341-kube-api-access-nt7c5\") pod \"node-exporter-qcsd5\" (UID: \"f9a132fa-2995-418a-9f53-19ecdf6ec341\") " pod="openshift-monitoring/node-exporter-qcsd5" Jul 09 22:15:29.194523 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:15:29.194437 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f9a132fa-2995-418a-9f53-19ecdf6ec341-metrics-client-ca\") pod \"node-exporter-qcsd5\" (UID: \"f9a132fa-2995-418a-9f53-19ecdf6ec341\") " pod="openshift-monitoring/node-exporter-qcsd5" Jul 09 22:15:29.194523 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:15:29.194489 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/f9a132fa-2995-418a-9f53-19ecdf6ec341-node-exporter-textfile\") pod \"node-exporter-qcsd5\" (UID: \"f9a132fa-2995-418a-9f53-19ecdf6ec341\") " pod="openshift-monitoring/node-exporter-qcsd5" Jul 09 22:15:29.194523 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:15:29.194512 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f9a132fa-2995-418a-9f53-19ecdf6ec341-sys\") pod \"node-exporter-qcsd5\" (UID: \"f9a132fa-2995-418a-9f53-19ecdf6ec341\") " pod="openshift-monitoring/node-exporter-qcsd5" Jul 09 22:15:29.194841 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:15:29.194543 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/f9a132fa-2995-418a-9f53-19ecdf6ec341-node-exporter-accelerators-collector-config\") pod \"node-exporter-qcsd5\" (UID: \"f9a132fa-2995-418a-9f53-19ecdf6ec341\") " pod="openshift-monitoring/node-exporter-qcsd5" Jul 09 22:15:29.194841 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:15:29.194568 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/f9a132fa-2995-418a-9f53-19ecdf6ec341-node-exporter-wtmp\") pod \"node-exporter-qcsd5\" (UID: \"f9a132fa-2995-418a-9f53-19ecdf6ec341\") " pod="openshift-monitoring/node-exporter-qcsd5" Jul 09 22:15:29.194841 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:15:29.194575 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/f9a132fa-2995-418a-9f53-19ecdf6ec341-root\") pod \"node-exporter-qcsd5\" (UID: \"f9a132fa-2995-418a-9f53-19ecdf6ec341\") " pod="openshift-monitoring/node-exporter-qcsd5" Jul 09 22:15:29.194841 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:15:29.194612 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/f9a132fa-2995-418a-9f53-19ecdf6ec341-root\") pod \"node-exporter-qcsd5\" (UID: \"f9a132fa-2995-418a-9f53-19ecdf6ec341\") " pod="openshift-monitoring/node-exporter-qcsd5" Jul 09 22:15:29.195081 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:15:29.194940 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/f9a132fa-2995-418a-9f53-19ecdf6ec341-node-exporter-textfile\") pod \"node-exporter-qcsd5\" (UID: \"f9a132fa-2995-418a-9f53-19ecdf6ec341\") " pod="openshift-monitoring/node-exporter-qcsd5" Jul 09 22:15:29.195081 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:15:29.194991 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f9a132fa-2995-418a-9f53-19ecdf6ec341-sys\") pod \"node-exporter-qcsd5\" (UID: \"f9a132fa-2995-418a-9f53-19ecdf6ec341\") " pod="openshift-monitoring/node-exporter-qcsd5" Jul 09 22:15:29.195081 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:15:29.195069 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f9a132fa-2995-418a-9f53-19ecdf6ec341-metrics-client-ca\") pod \"node-exporter-qcsd5\" (UID: \"f9a132fa-2995-418a-9f53-19ecdf6ec341\") " pod="openshift-monitoring/node-exporter-qcsd5" Jul 09 22:15:29.195231 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:15:29.195157 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/f9a132fa-2995-418a-9f53-19ecdf6ec341-node-exporter-accelerators-collector-config\") pod \"node-exporter-qcsd5\" (UID: \"f9a132fa-2995-418a-9f53-19ecdf6ec341\") " pod="openshift-monitoring/node-exporter-qcsd5" Jul 09 22:15:29.196998 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:15:29.196974 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f9a132fa-2995-418a-9f53-19ecdf6ec341-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-qcsd5\" (UID: \"f9a132fa-2995-418a-9f53-19ecdf6ec341\") " pod="openshift-monitoring/node-exporter-qcsd5" Jul 09 22:15:29.197194 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:15:29.197174 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/f9a132fa-2995-418a-9f53-19ecdf6ec341-node-exporter-tls\") pod \"node-exporter-qcsd5\" (UID: \"f9a132fa-2995-418a-9f53-19ecdf6ec341\") " pod="openshift-monitoring/node-exporter-qcsd5" Jul 09 22:15:29.204800 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:15:29.204775 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nt7c5\" (UniqueName: \"kubernetes.io/projected/f9a132fa-2995-418a-9f53-19ecdf6ec341-kube-api-access-nt7c5\") pod \"node-exporter-qcsd5\" (UID: \"f9a132fa-2995-418a-9f53-19ecdf6ec341\") " pod="openshift-monitoring/node-exporter-qcsd5" Jul 09 22:15:29.318499 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:15:29.318422 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-qcsd5" Jul 09 22:15:29.370660 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:15:29.370619 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf9a132fa_2995_418a_9f53_19ecdf6ec341.slice/crio-bcfd1506052b835277a2fe954ea71fffca34daec8524949dead57a0429700ac5 WatchSource:0}: Error finding container bcfd1506052b835277a2fe954ea71fffca34daec8524949dead57a0429700ac5: Status 404 returned error can't find the container with id bcfd1506052b835277a2fe954ea71fffca34daec8524949dead57a0429700ac5 Jul 09 22:15:29.565175 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:15:29.565136 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-qcsd5" event={"ID":"f9a132fa-2995-418a-9f53-19ecdf6ec341","Type":"ContainerStarted","Data":"bcfd1506052b835277a2fe954ea71fffca34daec8524949dead57a0429700ac5"} Jul 09 22:15:31.574943 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:15:31.574832 2579 generic.go:356] "Generic (PLEG): container finished" podID="f9a132fa-2995-418a-9f53-19ecdf6ec341" containerID="13c41a87949fe658a37a2212a55a178b338195b7f5175b7980c39a4ece0bf07d" exitCode=0 Jul 09 22:15:31.574943 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:15:31.574909 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-qcsd5" event={"ID":"f9a132fa-2995-418a-9f53-19ecdf6ec341","Type":"ContainerDied","Data":"13c41a87949fe658a37a2212a55a178b338195b7f5175b7980c39a4ece0bf07d"} Jul 09 22:15:31.912806 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:15:31.912715 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-74f4c8dc89-jclzs"] Jul 09 22:15:31.930384 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:15:31.930349 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-74f4c8dc89-jclzs"] Jul 09 22:15:31.930570 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:15:31.930475 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-74f4c8dc89-jclzs" Jul 09 22:15:31.933251 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:15:31.933227 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\"" Jul 09 22:15:31.933411 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:15:31.933287 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\"" Jul 09 22:15:31.933411 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:15:31.933288 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\"" Jul 09 22:15:31.933411 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:15:31.933292 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-aftdil3b6gf2l\"" Jul 09 22:15:31.933621 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:15:31.933605 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\"" Jul 09 22:15:31.933679 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:15:31.933606 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\"" Jul 09 22:15:31.933679 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:15:31.933637 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-jk4dz\"" Jul 09 22:15:32.020247 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:15:32.020205 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/f27c5db7-84bc-49ca-8654-64590044faa0-secret-thanos-querier-tls\") pod \"thanos-querier-74f4c8dc89-jclzs\" (UID: \"f27c5db7-84bc-49ca-8654-64590044faa0\") " pod="openshift-monitoring/thanos-querier-74f4c8dc89-jclzs" Jul 09 22:15:32.020247 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:15:32.020248 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6srsd\" (UniqueName: \"kubernetes.io/projected/f27c5db7-84bc-49ca-8654-64590044faa0-kube-api-access-6srsd\") pod \"thanos-querier-74f4c8dc89-jclzs\" (UID: \"f27c5db7-84bc-49ca-8654-64590044faa0\") " pod="openshift-monitoring/thanos-querier-74f4c8dc89-jclzs" Jul 09 22:15:32.020464 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:15:32.020282 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/f27c5db7-84bc-49ca-8654-64590044faa0-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-74f4c8dc89-jclzs\" (UID: \"f27c5db7-84bc-49ca-8654-64590044faa0\") " pod="openshift-monitoring/thanos-querier-74f4c8dc89-jclzs" Jul 09 22:15:32.020464 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:15:32.020302 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/f27c5db7-84bc-49ca-8654-64590044faa0-secret-grpc-tls\") pod \"thanos-querier-74f4c8dc89-jclzs\" (UID: \"f27c5db7-84bc-49ca-8654-64590044faa0\") " pod="openshift-monitoring/thanos-querier-74f4c8dc89-jclzs" Jul 09 22:15:32.020464 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:15:32.020332 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/f27c5db7-84bc-49ca-8654-64590044faa0-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-74f4c8dc89-jclzs\" (UID: \"f27c5db7-84bc-49ca-8654-64590044faa0\") " pod="openshift-monitoring/thanos-querier-74f4c8dc89-jclzs" Jul 09 22:15:32.020464 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:15:32.020348 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f27c5db7-84bc-49ca-8654-64590044faa0-metrics-client-ca\") pod \"thanos-querier-74f4c8dc89-jclzs\" (UID: \"f27c5db7-84bc-49ca-8654-64590044faa0\") " pod="openshift-monitoring/thanos-querier-74f4c8dc89-jclzs" Jul 09 22:15:32.020464 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:15:32.020365 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/f27c5db7-84bc-49ca-8654-64590044faa0-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-74f4c8dc89-jclzs\" (UID: \"f27c5db7-84bc-49ca-8654-64590044faa0\") " pod="openshift-monitoring/thanos-querier-74f4c8dc89-jclzs" Jul 09 22:15:32.020464 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:15:32.020423 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/f27c5db7-84bc-49ca-8654-64590044faa0-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-74f4c8dc89-jclzs\" (UID: \"f27c5db7-84bc-49ca-8654-64590044faa0\") " pod="openshift-monitoring/thanos-querier-74f4c8dc89-jclzs" Jul 09 22:15:32.121704 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:15:32.121671 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/f27c5db7-84bc-49ca-8654-64590044faa0-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-74f4c8dc89-jclzs\" (UID: \"f27c5db7-84bc-49ca-8654-64590044faa0\") " pod="openshift-monitoring/thanos-querier-74f4c8dc89-jclzs" Jul 09 22:15:32.121913 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:15:32.121750 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/f27c5db7-84bc-49ca-8654-64590044faa0-secret-thanos-querier-tls\") pod \"thanos-querier-74f4c8dc89-jclzs\" (UID: \"f27c5db7-84bc-49ca-8654-64590044faa0\") " pod="openshift-monitoring/thanos-querier-74f4c8dc89-jclzs" Jul 09 22:15:32.121913 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:15:32.121788 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6srsd\" (UniqueName: \"kubernetes.io/projected/f27c5db7-84bc-49ca-8654-64590044faa0-kube-api-access-6srsd\") pod \"thanos-querier-74f4c8dc89-jclzs\" (UID: \"f27c5db7-84bc-49ca-8654-64590044faa0\") " pod="openshift-monitoring/thanos-querier-74f4c8dc89-jclzs" Jul 09 22:15:32.121913 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:15:32.121843 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/f27c5db7-84bc-49ca-8654-64590044faa0-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-74f4c8dc89-jclzs\" (UID: \"f27c5db7-84bc-49ca-8654-64590044faa0\") " pod="openshift-monitoring/thanos-querier-74f4c8dc89-jclzs" Jul 09 22:15:32.121913 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:15:32.121872 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/f27c5db7-84bc-49ca-8654-64590044faa0-secret-grpc-tls\") pod \"thanos-querier-74f4c8dc89-jclzs\" (UID: \"f27c5db7-84bc-49ca-8654-64590044faa0\") " pod="openshift-monitoring/thanos-querier-74f4c8dc89-jclzs" Jul 09 22:15:32.121913 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:15:32.121905 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/f27c5db7-84bc-49ca-8654-64590044faa0-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-74f4c8dc89-jclzs\" (UID: \"f27c5db7-84bc-49ca-8654-64590044faa0\") " pod="openshift-monitoring/thanos-querier-74f4c8dc89-jclzs" Jul 09 22:15:32.122229 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:15:32.122068 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f27c5db7-84bc-49ca-8654-64590044faa0-metrics-client-ca\") pod \"thanos-querier-74f4c8dc89-jclzs\" (UID: \"f27c5db7-84bc-49ca-8654-64590044faa0\") " pod="openshift-monitoring/thanos-querier-74f4c8dc89-jclzs" Jul 09 22:15:32.122229 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:15:32.122116 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/f27c5db7-84bc-49ca-8654-64590044faa0-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-74f4c8dc89-jclzs\" (UID: \"f27c5db7-84bc-49ca-8654-64590044faa0\") " pod="openshift-monitoring/thanos-querier-74f4c8dc89-jclzs" Jul 09 22:15:32.122863 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:15:32.122830 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f27c5db7-84bc-49ca-8654-64590044faa0-metrics-client-ca\") pod \"thanos-querier-74f4c8dc89-jclzs\" (UID: \"f27c5db7-84bc-49ca-8654-64590044faa0\") " pod="openshift-monitoring/thanos-querier-74f4c8dc89-jclzs" Jul 09 22:15:32.124877 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:15:32.124850 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/f27c5db7-84bc-49ca-8654-64590044faa0-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-74f4c8dc89-jclzs\" (UID: \"f27c5db7-84bc-49ca-8654-64590044faa0\") " pod="openshift-monitoring/thanos-querier-74f4c8dc89-jclzs" Jul 09 22:15:32.125195 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:15:32.125026 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/f27c5db7-84bc-49ca-8654-64590044faa0-secret-thanos-querier-tls\") pod \"thanos-querier-74f4c8dc89-jclzs\" (UID: \"f27c5db7-84bc-49ca-8654-64590044faa0\") " pod="openshift-monitoring/thanos-querier-74f4c8dc89-jclzs" Jul 09 22:15:32.125195 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:15:32.125041 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/f27c5db7-84bc-49ca-8654-64590044faa0-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-74f4c8dc89-jclzs\" (UID: \"f27c5db7-84bc-49ca-8654-64590044faa0\") " pod="openshift-monitoring/thanos-querier-74f4c8dc89-jclzs" Jul 09 22:15:32.125195 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:15:32.125073 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/f27c5db7-84bc-49ca-8654-64590044faa0-secret-grpc-tls\") pod \"thanos-querier-74f4c8dc89-jclzs\" (UID: \"f27c5db7-84bc-49ca-8654-64590044faa0\") " pod="openshift-monitoring/thanos-querier-74f4c8dc89-jclzs" Jul 09 22:15:32.125195 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:15:32.125108 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/f27c5db7-84bc-49ca-8654-64590044faa0-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-74f4c8dc89-jclzs\" (UID: \"f27c5db7-84bc-49ca-8654-64590044faa0\") " pod="openshift-monitoring/thanos-querier-74f4c8dc89-jclzs" Jul 09 22:15:32.125195 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:15:32.125132 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/f27c5db7-84bc-49ca-8654-64590044faa0-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-74f4c8dc89-jclzs\" (UID: \"f27c5db7-84bc-49ca-8654-64590044faa0\") " pod="openshift-monitoring/thanos-querier-74f4c8dc89-jclzs" Jul 09 22:15:32.130623 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:15:32.130601 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6srsd\" (UniqueName: \"kubernetes.io/projected/f27c5db7-84bc-49ca-8654-64590044faa0-kube-api-access-6srsd\") pod \"thanos-querier-74f4c8dc89-jclzs\" (UID: \"f27c5db7-84bc-49ca-8654-64590044faa0\") " pod="openshift-monitoring/thanos-querier-74f4c8dc89-jclzs" Jul 09 22:15:32.240176 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:15:32.240143 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-74f4c8dc89-jclzs" Jul 09 22:15:32.399501 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:15:32.399477 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-74f4c8dc89-jclzs"] Jul 09 22:15:32.401849 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:15:32.401822 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf27c5db7_84bc_49ca_8654_64590044faa0.slice/crio-77423379d2a288bfde673a203939dd857aa145bce5d16f8e4e8038a2d8acffa5 WatchSource:0}: Error finding container 77423379d2a288bfde673a203939dd857aa145bce5d16f8e4e8038a2d8acffa5: Status 404 returned error can't find the container with id 77423379d2a288bfde673a203939dd857aa145bce5d16f8e4e8038a2d8acffa5 Jul 09 22:15:32.580604 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:15:32.580520 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-qcsd5" event={"ID":"f9a132fa-2995-418a-9f53-19ecdf6ec341","Type":"ContainerStarted","Data":"b534769957d4deb0a926e199abfe05ebf8d46d8aabebfbdf92218494315e2156"} Jul 09 22:15:32.580604 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:15:32.580569 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-qcsd5" event={"ID":"f9a132fa-2995-418a-9f53-19ecdf6ec341","Type":"ContainerStarted","Data":"8455723418c932ef811fe37afa4c6b8b45955f4516e2c1b75cd0576669dca25f"} Jul 09 22:15:32.581812 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:15:32.581788 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-74f4c8dc89-jclzs" event={"ID":"f27c5db7-84bc-49ca-8654-64590044faa0","Type":"ContainerStarted","Data":"77423379d2a288bfde673a203939dd857aa145bce5d16f8e4e8038a2d8acffa5"} Jul 09 22:15:32.606908 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:15:32.606858 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-qcsd5" podStartSLOduration=2.602564067 podStartE2EDuration="4.606841097s" totalImagesPullingTime="1.911666679s" totalInitContainerRuntime="92.610351ms" isStatefulPod=true podCreationTimestamp="2026-07-09 22:15:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 22:15:32.606185251 +0000 UTC m=+324.517637950" watchObservedRunningTime="2026-07-09 22:15:32.606841097 +0000 UTC m=+324.518293797" Jul 09 22:15:33.721870 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:15:33.721830 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-d5dcff7cf-d6q56"] Jul 09 22:15:33.742550 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:15:33.742497 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-d5dcff7cf-d6q56"] Jul 09 22:15:33.742725 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:15:33.742625 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-d6q56" Jul 09 22:15:33.745330 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:15:33.745118 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\"" Jul 09 22:15:33.745330 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:15:33.745214 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-dj2qk\"" Jul 09 22:15:33.840719 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:15:33.840684 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/91037a42-3ca0-4174-9ce2-ade5caadea9f-monitoring-plugin-cert\") pod \"monitoring-plugin-d5dcff7cf-d6q56\" (UID: \"91037a42-3ca0-4174-9ce2-ade5caadea9f\") " pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-d6q56" Jul 09 22:15:33.840719 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:15:33.840723 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ft7gb\" (UniqueName: \"kubernetes.io/projected/91037a42-3ca0-4174-9ce2-ade5caadea9f-kube-api-access-ft7gb\") pod \"monitoring-plugin-d5dcff7cf-d6q56\" (UID: \"91037a42-3ca0-4174-9ce2-ade5caadea9f\") " pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-d6q56" Jul 09 22:15:33.941884 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:15:33.941850 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/91037a42-3ca0-4174-9ce2-ade5caadea9f-monitoring-plugin-cert\") pod \"monitoring-plugin-d5dcff7cf-d6q56\" (UID: \"91037a42-3ca0-4174-9ce2-ade5caadea9f\") " pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-d6q56" Jul 09 22:15:33.941884 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:15:33.941891 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ft7gb\" (UniqueName: \"kubernetes.io/projected/91037a42-3ca0-4174-9ce2-ade5caadea9f-kube-api-access-ft7gb\") pod \"monitoring-plugin-d5dcff7cf-d6q56\" (UID: \"91037a42-3ca0-4174-9ce2-ade5caadea9f\") " pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-d6q56" Jul 09 22:15:33.944458 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:15:33.944432 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/91037a42-3ca0-4174-9ce2-ade5caadea9f-monitoring-plugin-cert\") pod \"monitoring-plugin-d5dcff7cf-d6q56\" (UID: \"91037a42-3ca0-4174-9ce2-ade5caadea9f\") " pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-d6q56" Jul 09 22:15:33.951648 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:15:33.951616 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ft7gb\" (UniqueName: \"kubernetes.io/projected/91037a42-3ca0-4174-9ce2-ade5caadea9f-kube-api-access-ft7gb\") pod \"monitoring-plugin-d5dcff7cf-d6q56\" (UID: \"91037a42-3ca0-4174-9ce2-ade5caadea9f\") " pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-d6q56" Jul 09 22:15:34.053889 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:15:34.053855 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-d6q56" Jul 09 22:15:34.219547 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:15:34.219527 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-d5dcff7cf-d6q56"] Jul 09 22:15:34.221651 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:15:34.221624 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod91037a42_3ca0_4174_9ce2_ade5caadea9f.slice/crio-010601fd02c754c475dd707ff791ef466358691ec09933f8e4d2830d5bf94c4f WatchSource:0}: Error finding container 010601fd02c754c475dd707ff791ef466358691ec09933f8e4d2830d5bf94c4f: Status 404 returned error can't find the container with id 010601fd02c754c475dd707ff791ef466358691ec09933f8e4d2830d5bf94c4f Jul 09 22:15:34.590511 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:15:34.590469 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-d6q56" event={"ID":"91037a42-3ca0-4174-9ce2-ade5caadea9f","Type":"ContainerStarted","Data":"010601fd02c754c475dd707ff791ef466358691ec09933f8e4d2830d5bf94c4f"} Jul 09 22:15:35.213195 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:15:35.212907 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 09 22:15:35.254897 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:15:35.254816 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 09 22:15:35.255089 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:15:35.254989 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 09 22:15:35.266504 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:15:35.264310 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Jul 09 22:15:35.266504 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:15:35.264594 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Jul 09 22:15:35.266504 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:15:35.264772 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Jul 09 22:15:35.266504 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:15:35.265202 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Jul 09 22:15:35.266504 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:15:35.265229 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Jul 09 22:15:35.266504 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:15:35.265397 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Jul 09 22:15:35.266504 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:15:35.265461 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Jul 09 22:15:35.266504 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:15:35.265952 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-tbrt6\"" Jul 09 22:15:35.266504 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:15:35.266163 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Jul 09 22:15:35.266504 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:15:35.266344 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Jul 09 22:15:35.267973 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:15:35.267543 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Jul 09 22:15:35.267973 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:15:35.267800 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Jul 09 22:15:35.268139 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:15:35.268060 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Jul 09 22:15:35.268598 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:15:35.268294 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Jul 09 22:15:35.268598 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:15:35.268419 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-bbi1m9cvt2s77\"" Jul 09 22:15:35.268598 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:15:35.268502 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Jul 09 22:15:35.269047 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:15:35.269025 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Jul 09 22:15:35.355172 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:15:35.355138 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/c3770ca5-9562-4939-bf80-3ff488746902-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"c3770ca5-9562-4939-bf80-3ff488746902\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 22:15:35.355172 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:15:35.355176 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/c3770ca5-9562-4939-bf80-3ff488746902-config\") pod \"prometheus-k8s-0\" (UID: \"c3770ca5-9562-4939-bf80-3ff488746902\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 22:15:35.355414 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:15:35.355203 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/c3770ca5-9562-4939-bf80-3ff488746902-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"c3770ca5-9562-4939-bf80-3ff488746902\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 22:15:35.355414 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:15:35.355267 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c3770ca5-9562-4939-bf80-3ff488746902-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"c3770ca5-9562-4939-bf80-3ff488746902\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 22:15:35.355414 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:15:35.355313 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/c3770ca5-9562-4939-bf80-3ff488746902-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"c3770ca5-9562-4939-bf80-3ff488746902\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 22:15:35.355414 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:15:35.355338 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/c3770ca5-9562-4939-bf80-3ff488746902-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"c3770ca5-9562-4939-bf80-3ff488746902\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 22:15:35.355414 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:15:35.355362 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/c3770ca5-9562-4939-bf80-3ff488746902-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"c3770ca5-9562-4939-bf80-3ff488746902\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 22:15:35.355414 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:15:35.355404 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c3770ca5-9562-4939-bf80-3ff488746902-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"c3770ca5-9562-4939-bf80-3ff488746902\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 22:15:35.355719 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:15:35.355442 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/c3770ca5-9562-4939-bf80-3ff488746902-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"c3770ca5-9562-4939-bf80-3ff488746902\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 22:15:35.355719 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:15:35.355464 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/c3770ca5-9562-4939-bf80-3ff488746902-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"c3770ca5-9562-4939-bf80-3ff488746902\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 22:15:35.355719 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:15:35.355492 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/c3770ca5-9562-4939-bf80-3ff488746902-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"c3770ca5-9562-4939-bf80-3ff488746902\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 22:15:35.355719 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:15:35.355507 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/c3770ca5-9562-4939-bf80-3ff488746902-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"c3770ca5-9562-4939-bf80-3ff488746902\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 22:15:35.355719 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:15:35.355572 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c3770ca5-9562-4939-bf80-3ff488746902-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"c3770ca5-9562-4939-bf80-3ff488746902\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 22:15:35.355719 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:15:35.355605 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c3770ca5-9562-4939-bf80-3ff488746902-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"c3770ca5-9562-4939-bf80-3ff488746902\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 22:15:35.355719 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:15:35.355632 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/c3770ca5-9562-4939-bf80-3ff488746902-web-config\") pod \"prometheus-k8s-0\" (UID: \"c3770ca5-9562-4939-bf80-3ff488746902\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 22:15:35.355719 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:15:35.355694 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/c3770ca5-9562-4939-bf80-3ff488746902-config-out\") pod \"prometheus-k8s-0\" (UID: \"c3770ca5-9562-4939-bf80-3ff488746902\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 22:15:35.356149 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:15:35.355721 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c9q8m\" (UniqueName: \"kubernetes.io/projected/c3770ca5-9562-4939-bf80-3ff488746902-kube-api-access-c9q8m\") pod \"prometheus-k8s-0\" (UID: \"c3770ca5-9562-4939-bf80-3ff488746902\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 22:15:35.356149 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:15:35.355761 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/c3770ca5-9562-4939-bf80-3ff488746902-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"c3770ca5-9562-4939-bf80-3ff488746902\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 22:15:35.356149 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:15:35.355777 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/c3770ca5-9562-4939-bf80-3ff488746902-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"c3770ca5-9562-4939-bf80-3ff488746902\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 22:15:35.356149 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:15:35.355796 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/c3770ca5-9562-4939-bf80-3ff488746902-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"c3770ca5-9562-4939-bf80-3ff488746902\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 22:15:35.456733 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:15:35.456688 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/c3770ca5-9562-4939-bf80-3ff488746902-config\") pod \"prometheus-k8s-0\" (UID: \"c3770ca5-9562-4939-bf80-3ff488746902\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 22:15:35.456945 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:15:35.456761 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/c3770ca5-9562-4939-bf80-3ff488746902-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"c3770ca5-9562-4939-bf80-3ff488746902\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 22:15:35.456945 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:15:35.456788 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c3770ca5-9562-4939-bf80-3ff488746902-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"c3770ca5-9562-4939-bf80-3ff488746902\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 22:15:35.456945 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:15:35.456821 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/c3770ca5-9562-4939-bf80-3ff488746902-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"c3770ca5-9562-4939-bf80-3ff488746902\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 22:15:35.456945 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:15:35.456850 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/c3770ca5-9562-4939-bf80-3ff488746902-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"c3770ca5-9562-4939-bf80-3ff488746902\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 22:15:35.456945 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:15:35.456885 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/c3770ca5-9562-4939-bf80-3ff488746902-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"c3770ca5-9562-4939-bf80-3ff488746902\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 22:15:35.456945 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:15:35.456912 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c3770ca5-9562-4939-bf80-3ff488746902-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"c3770ca5-9562-4939-bf80-3ff488746902\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 22:15:35.457263 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:15:35.456972 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/c3770ca5-9562-4939-bf80-3ff488746902-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"c3770ca5-9562-4939-bf80-3ff488746902\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 22:15:35.457263 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:15:35.456996 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/c3770ca5-9562-4939-bf80-3ff488746902-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"c3770ca5-9562-4939-bf80-3ff488746902\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 22:15:35.457263 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:15:35.457027 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/c3770ca5-9562-4939-bf80-3ff488746902-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"c3770ca5-9562-4939-bf80-3ff488746902\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 22:15:35.457263 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:15:35.457053 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/c3770ca5-9562-4939-bf80-3ff488746902-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"c3770ca5-9562-4939-bf80-3ff488746902\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 22:15:35.457263 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:15:35.457107 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c3770ca5-9562-4939-bf80-3ff488746902-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"c3770ca5-9562-4939-bf80-3ff488746902\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 22:15:35.457263 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:15:35.457131 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c3770ca5-9562-4939-bf80-3ff488746902-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"c3770ca5-9562-4939-bf80-3ff488746902\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 22:15:35.457263 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:15:35.457160 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/c3770ca5-9562-4939-bf80-3ff488746902-web-config\") pod \"prometheus-k8s-0\" (UID: \"c3770ca5-9562-4939-bf80-3ff488746902\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 22:15:35.457263 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:15:35.457200 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/c3770ca5-9562-4939-bf80-3ff488746902-config-out\") pod \"prometheus-k8s-0\" (UID: \"c3770ca5-9562-4939-bf80-3ff488746902\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 22:15:35.457263 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:15:35.457231 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c9q8m\" (UniqueName: \"kubernetes.io/projected/c3770ca5-9562-4939-bf80-3ff488746902-kube-api-access-c9q8m\") pod \"prometheus-k8s-0\" (UID: \"c3770ca5-9562-4939-bf80-3ff488746902\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 22:15:35.457704 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:15:35.457278 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/c3770ca5-9562-4939-bf80-3ff488746902-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"c3770ca5-9562-4939-bf80-3ff488746902\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 22:15:35.457704 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:15:35.457310 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/c3770ca5-9562-4939-bf80-3ff488746902-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"c3770ca5-9562-4939-bf80-3ff488746902\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 22:15:35.457704 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:15:35.457333 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/c3770ca5-9562-4939-bf80-3ff488746902-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"c3770ca5-9562-4939-bf80-3ff488746902\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 22:15:35.457704 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:15:35.457358 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/c3770ca5-9562-4939-bf80-3ff488746902-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"c3770ca5-9562-4939-bf80-3ff488746902\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 22:15:35.458719 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:15:35.458687 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c3770ca5-9562-4939-bf80-3ff488746902-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"c3770ca5-9562-4939-bf80-3ff488746902\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 22:15:35.459508 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:15:35.459219 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/c3770ca5-9562-4939-bf80-3ff488746902-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"c3770ca5-9562-4939-bf80-3ff488746902\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 22:15:35.459737 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:15:35.458691 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c3770ca5-9562-4939-bf80-3ff488746902-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"c3770ca5-9562-4939-bf80-3ff488746902\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 22:15:35.461462 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:15:35.461154 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/c3770ca5-9562-4939-bf80-3ff488746902-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"c3770ca5-9562-4939-bf80-3ff488746902\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 22:15:35.463501 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:15:35.462294 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/c3770ca5-9562-4939-bf80-3ff488746902-web-config\") pod \"prometheus-k8s-0\" (UID: \"c3770ca5-9562-4939-bf80-3ff488746902\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 22:15:35.463501 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:15:35.462435 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/c3770ca5-9562-4939-bf80-3ff488746902-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"c3770ca5-9562-4939-bf80-3ff488746902\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 22:15:35.463501 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:15:35.463054 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c3770ca5-9562-4939-bf80-3ff488746902-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"c3770ca5-9562-4939-bf80-3ff488746902\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 22:15:35.463501 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:15:35.463080 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/c3770ca5-9562-4939-bf80-3ff488746902-config-out\") pod \"prometheus-k8s-0\" (UID: \"c3770ca5-9562-4939-bf80-3ff488746902\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 22:15:35.463501 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:15:35.463081 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c3770ca5-9562-4939-bf80-3ff488746902-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"c3770ca5-9562-4939-bf80-3ff488746902\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 22:15:35.463501 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:15:35.463239 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/c3770ca5-9562-4939-bf80-3ff488746902-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"c3770ca5-9562-4939-bf80-3ff488746902\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 22:15:35.463501 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:15:35.463437 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/c3770ca5-9562-4939-bf80-3ff488746902-config\") pod \"prometheus-k8s-0\" (UID: \"c3770ca5-9562-4939-bf80-3ff488746902\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 22:15:35.466638 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:15:35.463571 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/c3770ca5-9562-4939-bf80-3ff488746902-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"c3770ca5-9562-4939-bf80-3ff488746902\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 22:15:35.466638 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:15:35.463601 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/c3770ca5-9562-4939-bf80-3ff488746902-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"c3770ca5-9562-4939-bf80-3ff488746902\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 22:15:35.466638 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:15:35.464380 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/c3770ca5-9562-4939-bf80-3ff488746902-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"c3770ca5-9562-4939-bf80-3ff488746902\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 22:15:35.466638 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:15:35.464883 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/c3770ca5-9562-4939-bf80-3ff488746902-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"c3770ca5-9562-4939-bf80-3ff488746902\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 22:15:35.466638 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:15:35.465220 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/c3770ca5-9562-4939-bf80-3ff488746902-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"c3770ca5-9562-4939-bf80-3ff488746902\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 22:15:35.466971 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:15:35.466946 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/c3770ca5-9562-4939-bf80-3ff488746902-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"c3770ca5-9562-4939-bf80-3ff488746902\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 22:15:35.467735 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:15:35.467324 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/c3770ca5-9562-4939-bf80-3ff488746902-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"c3770ca5-9562-4939-bf80-3ff488746902\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 22:15:35.467735 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:15:35.467359 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/c3770ca5-9562-4939-bf80-3ff488746902-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"c3770ca5-9562-4939-bf80-3ff488746902\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 22:15:35.469136 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:15:35.469115 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c9q8m\" (UniqueName: \"kubernetes.io/projected/c3770ca5-9562-4939-bf80-3ff488746902-kube-api-access-c9q8m\") pod \"prometheus-k8s-0\" (UID: \"c3770ca5-9562-4939-bf80-3ff488746902\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 09 22:15:35.570629 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:15:35.570591 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 09 22:15:36.725201 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:15:36.725168 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 09 22:15:37.604327 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:15:37.604286 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-d6q56" event={"ID":"91037a42-3ca0-4174-9ce2-ade5caadea9f","Type":"ContainerStarted","Data":"aca769bd245acdf1094e20b620cc2651635b44fba2147ef3d4aa7e2fcfddff6b"} Jul 09 22:15:37.604555 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:15:37.604492 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-d6q56" Jul 09 22:15:37.606887 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:15:37.606837 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-74f4c8dc89-jclzs" event={"ID":"f27c5db7-84bc-49ca-8654-64590044faa0","Type":"ContainerStarted","Data":"ce979f78625e5a6e6ceda00878268a5d4af0f952f1661bc7365446b6e5ff45d1"} Jul 09 22:15:37.607046 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:15:37.606946 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-74f4c8dc89-jclzs" event={"ID":"f27c5db7-84bc-49ca-8654-64590044faa0","Type":"ContainerStarted","Data":"5d59a86b56721540a0d7c84628737839758f9400fbff9489028e446194686c94"} Jul 09 22:15:37.607046 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:15:37.606994 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-74f4c8dc89-jclzs" event={"ID":"f27c5db7-84bc-49ca-8654-64590044faa0","Type":"ContainerStarted","Data":"162dc3132b15810bd4093ae78a7d3a6bf55a4b06d4670d6f14cdd182eac36d89"} Jul 09 22:15:37.608478 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:15:37.608451 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"c3770ca5-9562-4939-bf80-3ff488746902","Type":"ContainerStarted","Data":"77db87487d13874d9b621dd346c4ea9d2886f9bcbd05554f97ac96996a9c7ae6"} Jul 09 22:15:37.610772 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:15:37.610749 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-d6q56" Jul 09 22:15:37.623312 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:15:37.623251 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-d6q56" podStartSLOduration=2.258528432 podStartE2EDuration="4.623234898s" totalImagesPullingTime="2.364706466s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 22:15:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 22:15:37.622126076 +0000 UTC m=+329.533578796" watchObservedRunningTime="2026-07-09 22:15:37.623234898 +0000 UTC m=+329.534687598" Jul 09 22:15:38.613014 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:15:38.612982 2579 generic.go:356] "Generic (PLEG): container finished" podID="c3770ca5-9562-4939-bf80-3ff488746902" containerID="19cbb4caf67cd57f1d13e0e903ccaab95b082bfc6f56451ed97907f503b283e2" exitCode=0 Jul 09 22:15:38.613422 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:15:38.613072 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"c3770ca5-9562-4939-bf80-3ff488746902","Type":"ContainerDied","Data":"19cbb4caf67cd57f1d13e0e903ccaab95b082bfc6f56451ed97907f503b283e2"} Jul 09 22:15:39.624815 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:15:39.624772 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-74f4c8dc89-jclzs" event={"ID":"f27c5db7-84bc-49ca-8654-64590044faa0","Type":"ContainerStarted","Data":"c73e50fb81a6a7c4a991f80d87963f3c24edd8bf54decf84662560aaf64fc010"} Jul 09 22:15:39.625317 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:15:39.624897 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-74f4c8dc89-jclzs" event={"ID":"f27c5db7-84bc-49ca-8654-64590044faa0","Type":"ContainerStarted","Data":"3cb36a4fbbbf43f1c30558c3fa400b4751ecb536196ca73ff994580372101a2c"} Jul 09 22:15:39.625317 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:15:39.624965 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-74f4c8dc89-jclzs" event={"ID":"f27c5db7-84bc-49ca-8654-64590044faa0","Type":"ContainerStarted","Data":"821ad1e5a2789b6fc3da841ec6cb2e0a62f27419f27fd2287109814eb26202ed"} Jul 09 22:15:39.647164 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:15:39.647101 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-74f4c8dc89-jclzs" podStartSLOduration=2.523804226 podStartE2EDuration="8.64708114s" totalImagesPullingTime="6.123276914s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 22:15:31 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 22:15:39.645434133 +0000 UTC m=+331.556886859" watchObservedRunningTime="2026-07-09 22:15:39.64708114 +0000 UTC m=+331.558533840" Jul 09 22:15:40.629411 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:15:40.629374 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-74f4c8dc89-jclzs" Jul 09 22:15:41.639637 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:15:41.639607 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-74f4c8dc89-jclzs" Jul 09 22:15:43.643851 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:15:43.643819 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"c3770ca5-9562-4939-bf80-3ff488746902","Type":"ContainerStarted","Data":"23f1c9c3bf828aaee49e738918858691444ada6f638c22239ebd86e9853d96af"} Jul 09 22:15:43.644277 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:15:43.643909 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"c3770ca5-9562-4939-bf80-3ff488746902","Type":"ContainerStarted","Data":"7670dca17bdc0a0b3bf4388eeeef5224cb22bbc41304fb5fc87e4d75ba5ed841"} Jul 09 22:15:43.644277 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:15:43.643964 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"c3770ca5-9562-4939-bf80-3ff488746902","Type":"ContainerStarted","Data":"93c138b14efe17c0d48f681126827f41a1343d951c2f495e04eb4286765f879a"} Jul 09 22:15:43.644277 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:15:43.643994 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"c3770ca5-9562-4939-bf80-3ff488746902","Type":"ContainerStarted","Data":"5cfb473266a26692c5c8007de5935b6adc2cab2f983cbc0274c398fe197bfb40"} Jul 09 22:15:43.644277 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:15:43.644021 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"c3770ca5-9562-4939-bf80-3ff488746902","Type":"ContainerStarted","Data":"ed7df503d55418403bb26319b1ff2ab0bfe80ef749750b502648108dbe28741d"} Jul 09 22:15:43.644277 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:15:43.644047 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"c3770ca5-9562-4939-bf80-3ff488746902","Type":"ContainerStarted","Data":"3be4769ee12eab0ef95e498d1b355ecf5aab905eb07088fb987bd2f99fc43048"} Jul 09 22:15:43.675967 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:15:43.675883 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=3.170557329 podStartE2EDuration="8.67586365s" totalImagesPullingTime="5.486398605s" totalInitContainerRuntime="18.907716ms" isStatefulPod=false podCreationTimestamp="2026-07-09 22:15:35 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 22:15:43.673936754 +0000 UTC m=+335.585389470" watchObservedRunningTime="2026-07-09 22:15:43.67586365 +0000 UTC m=+335.587316351" Jul 09 22:15:45.571683 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:15:45.571648 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Jul 09 22:15:49.204514 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:15:49.204466 2579 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-image-registry/image-registry-5957744766-cmxpg" podUID="e8d76e6f-c1a2-4c70-8c4e-ce0b0480d75a" containerName="registry" containerID="cri-o://1dddb73c32c9d749193d8fada82d30f394177f3ac6365bf38c231f926bc543d4" gracePeriod=30 Jul 09 22:15:49.494314 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:15:49.494284 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5957744766-cmxpg" Jul 09 22:15:49.584798 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:15:49.584762 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vdhgh\" (UniqueName: \"kubernetes.io/projected/e8d76e6f-c1a2-4c70-8c4e-ce0b0480d75a-kube-api-access-vdhgh\") pod \"e8d76e6f-c1a2-4c70-8c4e-ce0b0480d75a\" (UID: \"e8d76e6f-c1a2-4c70-8c4e-ce0b0480d75a\") " Jul 09 22:15:49.584798 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:15:49.584804 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/e8d76e6f-c1a2-4c70-8c4e-ce0b0480d75a-bound-sa-token\") pod \"e8d76e6f-c1a2-4c70-8c4e-ce0b0480d75a\" (UID: \"e8d76e6f-c1a2-4c70-8c4e-ce0b0480d75a\") " Jul 09 22:15:49.585020 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:15:49.584828 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/e8d76e6f-c1a2-4c70-8c4e-ce0b0480d75a-trusted-ca\") pod \"e8d76e6f-c1a2-4c70-8c4e-ce0b0480d75a\" (UID: \"e8d76e6f-c1a2-4c70-8c4e-ce0b0480d75a\") " Jul 09 22:15:49.585020 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:15:49.584863 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/e8d76e6f-c1a2-4c70-8c4e-ce0b0480d75a-registry-tls\") pod \"e8d76e6f-c1a2-4c70-8c4e-ce0b0480d75a\" (UID: \"e8d76e6f-c1a2-4c70-8c4e-ce0b0480d75a\") " Jul 09 22:15:49.585020 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:15:49.584882 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/e8d76e6f-c1a2-4c70-8c4e-ce0b0480d75a-registry-certificates\") pod \"e8d76e6f-c1a2-4c70-8c4e-ce0b0480d75a\" (UID: \"e8d76e6f-c1a2-4c70-8c4e-ce0b0480d75a\") " Jul 09 22:15:49.585020 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:15:49.584933 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/e8d76e6f-c1a2-4c70-8c4e-ce0b0480d75a-ca-trust-extracted\") pod \"e8d76e6f-c1a2-4c70-8c4e-ce0b0480d75a\" (UID: \"e8d76e6f-c1a2-4c70-8c4e-ce0b0480d75a\") " Jul 09 22:15:49.585020 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:15:49.584988 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/e8d76e6f-c1a2-4c70-8c4e-ce0b0480d75a-installation-pull-secrets\") pod \"e8d76e6f-c1a2-4c70-8c4e-ce0b0480d75a\" (UID: \"e8d76e6f-c1a2-4c70-8c4e-ce0b0480d75a\") " Jul 09 22:15:49.585263 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:15:49.585031 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/e8d76e6f-c1a2-4c70-8c4e-ce0b0480d75a-image-registry-private-configuration\") pod \"e8d76e6f-c1a2-4c70-8c4e-ce0b0480d75a\" (UID: \"e8d76e6f-c1a2-4c70-8c4e-ce0b0480d75a\") " Jul 09 22:15:49.585808 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:15:49.585380 2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e8d76e6f-c1a2-4c70-8c4e-ce0b0480d75a-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "e8d76e6f-c1a2-4c70-8c4e-ce0b0480d75a" (UID: "e8d76e6f-c1a2-4c70-8c4e-ce0b0480d75a"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 09 22:15:49.585808 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:15:49.585390 2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e8d76e6f-c1a2-4c70-8c4e-ce0b0480d75a-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "e8d76e6f-c1a2-4c70-8c4e-ce0b0480d75a" (UID: "e8d76e6f-c1a2-4c70-8c4e-ce0b0480d75a"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 09 22:15:49.587732 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:15:49.587700 2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e8d76e6f-c1a2-4c70-8c4e-ce0b0480d75a-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "e8d76e6f-c1a2-4c70-8c4e-ce0b0480d75a" (UID: "e8d76e6f-c1a2-4c70-8c4e-ce0b0480d75a"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 09 22:15:49.587851 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:15:49.587772 2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e8d76e6f-c1a2-4c70-8c4e-ce0b0480d75a-kube-api-access-vdhgh" (OuterVolumeSpecName: "kube-api-access-vdhgh") pod "e8d76e6f-c1a2-4c70-8c4e-ce0b0480d75a" (UID: "e8d76e6f-c1a2-4c70-8c4e-ce0b0480d75a"). InnerVolumeSpecName "kube-api-access-vdhgh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 22:15:49.587851 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:15:49.587715 2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e8d76e6f-c1a2-4c70-8c4e-ce0b0480d75a-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "e8d76e6f-c1a2-4c70-8c4e-ce0b0480d75a" (UID: "e8d76e6f-c1a2-4c70-8c4e-ce0b0480d75a"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 22:15:49.588093 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:15:49.587855 2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e8d76e6f-c1a2-4c70-8c4e-ce0b0480d75a-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "e8d76e6f-c1a2-4c70-8c4e-ce0b0480d75a" (UID: "e8d76e6f-c1a2-4c70-8c4e-ce0b0480d75a"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 22:15:49.588093 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:15:49.587860 2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e8d76e6f-c1a2-4c70-8c4e-ce0b0480d75a-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "e8d76e6f-c1a2-4c70-8c4e-ce0b0480d75a" (UID: "e8d76e6f-c1a2-4c70-8c4e-ce0b0480d75a"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 09 22:15:49.593894 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:15:49.593869 2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e8d76e6f-c1a2-4c70-8c4e-ce0b0480d75a-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "e8d76e6f-c1a2-4c70-8c4e-ce0b0480d75a" (UID: "e8d76e6f-c1a2-4c70-8c4e-ce0b0480d75a"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 22:15:49.668510 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:15:49.668475 2579 generic.go:356] "Generic (PLEG): container finished" podID="dc1ba5a9-cd3c-46dc-8320-124980722f22" containerID="6d69df8d08a4c097f20715963fe964dfeef2df54a886e6907674f9de447b6c74" exitCode=0 Jul 09 22:15:49.668701 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:15:49.668550 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-lvwg4" event={"ID":"dc1ba5a9-cd3c-46dc-8320-124980722f22","Type":"ContainerDied","Data":"6d69df8d08a4c097f20715963fe964dfeef2df54a886e6907674f9de447b6c74"} Jul 09 22:15:49.668901 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:15:49.668886 2579 scope.go:117] "RemoveContainer" containerID="6d69df8d08a4c097f20715963fe964dfeef2df54a886e6907674f9de447b6c74" Jul 09 22:15:49.669847 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:15:49.669818 2579 generic.go:356] "Generic (PLEG): container finished" podID="e8d76e6f-c1a2-4c70-8c4e-ce0b0480d75a" containerID="1dddb73c32c9d749193d8fada82d30f394177f3ac6365bf38c231f926bc543d4" exitCode=0 Jul 09 22:15:49.669932 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:15:49.669844 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5957744766-cmxpg" event={"ID":"e8d76e6f-c1a2-4c70-8c4e-ce0b0480d75a","Type":"ContainerDied","Data":"1dddb73c32c9d749193d8fada82d30f394177f3ac6365bf38c231f926bc543d4"} Jul 09 22:15:49.669932 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:15:49.669866 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5957744766-cmxpg" Jul 09 22:15:49.669932 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:15:49.669892 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5957744766-cmxpg" event={"ID":"e8d76e6f-c1a2-4c70-8c4e-ce0b0480d75a","Type":"ContainerDied","Data":"579f930798f03de904629c2b7689c5fc3535d2e4cafc9743e8de6a739fe897ed"} Jul 09 22:15:49.670038 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:15:49.669941 2579 scope.go:117] "RemoveContainer" containerID="1dddb73c32c9d749193d8fada82d30f394177f3ac6365bf38c231f926bc543d4" Jul 09 22:15:49.679335 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:15:49.679315 2579 scope.go:117] "RemoveContainer" containerID="1dddb73c32c9d749193d8fada82d30f394177f3ac6365bf38c231f926bc543d4" Jul 09 22:15:49.679591 ip-10-0-136-251 kubenswrapper[2579]: E0709 22:15:49.679571 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1dddb73c32c9d749193d8fada82d30f394177f3ac6365bf38c231f926bc543d4\": container with ID starting with 1dddb73c32c9d749193d8fada82d30f394177f3ac6365bf38c231f926bc543d4 not found: ID does not exist" containerID="1dddb73c32c9d749193d8fada82d30f394177f3ac6365bf38c231f926bc543d4" Jul 09 22:15:49.679654 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:15:49.679599 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1dddb73c32c9d749193d8fada82d30f394177f3ac6365bf38c231f926bc543d4"} err="failed to get container status \"1dddb73c32c9d749193d8fada82d30f394177f3ac6365bf38c231f926bc543d4\": rpc error: code = NotFound desc = could not find container \"1dddb73c32c9d749193d8fada82d30f394177f3ac6365bf38c231f926bc543d4\": container with ID starting with 1dddb73c32c9d749193d8fada82d30f394177f3ac6365bf38c231f926bc543d4 not found: ID does not exist" Jul 09 22:15:49.686030 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:15:49.686014 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vdhgh\" (UniqueName: \"kubernetes.io/projected/e8d76e6f-c1a2-4c70-8c4e-ce0b0480d75a-kube-api-access-vdhgh\") on node \"ip-10-0-136-251.ec2.internal\" DevicePath \"\"" Jul 09 22:15:49.686099 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:15:49.686036 2579 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/e8d76e6f-c1a2-4c70-8c4e-ce0b0480d75a-bound-sa-token\") on node \"ip-10-0-136-251.ec2.internal\" DevicePath \"\"" Jul 09 22:15:49.686099 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:15:49.686050 2579 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/e8d76e6f-c1a2-4c70-8c4e-ce0b0480d75a-trusted-ca\") on node \"ip-10-0-136-251.ec2.internal\" DevicePath \"\"" Jul 09 22:15:49.686099 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:15:49.686063 2579 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/e8d76e6f-c1a2-4c70-8c4e-ce0b0480d75a-registry-tls\") on node \"ip-10-0-136-251.ec2.internal\" DevicePath \"\"" Jul 09 22:15:49.686099 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:15:49.686074 2579 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/e8d76e6f-c1a2-4c70-8c4e-ce0b0480d75a-registry-certificates\") on node \"ip-10-0-136-251.ec2.internal\" DevicePath \"\"" Jul 09 22:15:49.686099 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:15:49.686082 2579 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/e8d76e6f-c1a2-4c70-8c4e-ce0b0480d75a-ca-trust-extracted\") on node \"ip-10-0-136-251.ec2.internal\" DevicePath \"\"" Jul 09 22:15:49.686099 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:15:49.686090 2579 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/e8d76e6f-c1a2-4c70-8c4e-ce0b0480d75a-installation-pull-secrets\") on node \"ip-10-0-136-251.ec2.internal\" DevicePath \"\"" Jul 09 22:15:49.686099 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:15:49.686099 2579 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/e8d76e6f-c1a2-4c70-8c4e-ce0b0480d75a-image-registry-private-configuration\") on node \"ip-10-0-136-251.ec2.internal\" DevicePath \"\"" Jul 09 22:15:49.702079 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:15:49.702055 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-5957744766-cmxpg"] Jul 09 22:15:49.707238 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:15:49.707216 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-5957744766-cmxpg"] Jul 09 22:15:50.570623 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:15:50.570588 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e8d76e6f-c1a2-4c70-8c4e-ce0b0480d75a" path="/var/lib/kubelet/pods/e8d76e6f-c1a2-4c70-8c4e-ce0b0480d75a/volumes" Jul 09 22:15:50.675339 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:15:50.675302 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-lvwg4" event={"ID":"dc1ba5a9-cd3c-46dc-8320-124980722f22","Type":"ContainerStarted","Data":"da4540eb6f56cfa67d75bd57b6de0a597c41d8f2244ba4aa8b04ff24221df428"} Jul 09 22:15:51.501374 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:15:51.501333 2579 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-console/console-7445556d44-f87lt" podUID="3a476bd5-1f8c-47dd-8fac-c44d2dcea62d" containerName="console" containerID="cri-o://36b512474158f14a2bf4998812aebb0410cdda0c68dcf790b5f27641d4c158b8" gracePeriod=15 Jul 09 22:15:51.683990 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:15:51.683963 2579 generic.go:356] "Generic (PLEG): container finished" podID="3a476bd5-1f8c-47dd-8fac-c44d2dcea62d" containerID="36b512474158f14a2bf4998812aebb0410cdda0c68dcf790b5f27641d4c158b8" exitCode=0 Jul 09 22:15:51.684318 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:15:51.684038 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7445556d44-f87lt" event={"ID":"3a476bd5-1f8c-47dd-8fac-c44d2dcea62d","Type":"ContainerDied","Data":"36b512474158f14a2bf4998812aebb0410cdda0c68dcf790b5f27641d4c158b8"} Jul 09 22:15:51.782440 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:15:51.782372 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7445556d44-f87lt" Jul 09 22:15:51.801963 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:15:51.801930 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/3a476bd5-1f8c-47dd-8fac-c44d2dcea62d-console-config\") pod \"3a476bd5-1f8c-47dd-8fac-c44d2dcea62d\" (UID: \"3a476bd5-1f8c-47dd-8fac-c44d2dcea62d\") " Jul 09 22:15:51.802123 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:15:51.801986 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/3a476bd5-1f8c-47dd-8fac-c44d2dcea62d-console-serving-cert\") pod \"3a476bd5-1f8c-47dd-8fac-c44d2dcea62d\" (UID: \"3a476bd5-1f8c-47dd-8fac-c44d2dcea62d\") " Jul 09 22:15:51.802123 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:15:51.802006 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/3a476bd5-1f8c-47dd-8fac-c44d2dcea62d-console-oauth-config\") pod \"3a476bd5-1f8c-47dd-8fac-c44d2dcea62d\" (UID: \"3a476bd5-1f8c-47dd-8fac-c44d2dcea62d\") " Jul 09 22:15:51.802123 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:15:51.802023 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/3a476bd5-1f8c-47dd-8fac-c44d2dcea62d-oauth-serving-cert\") pod \"3a476bd5-1f8c-47dd-8fac-c44d2dcea62d\" (UID: \"3a476bd5-1f8c-47dd-8fac-c44d2dcea62d\") " Jul 09 22:15:51.802123 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:15:51.802078 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/3a476bd5-1f8c-47dd-8fac-c44d2dcea62d-service-ca\") pod \"3a476bd5-1f8c-47dd-8fac-c44d2dcea62d\" (UID: \"3a476bd5-1f8c-47dd-8fac-c44d2dcea62d\") " Jul 09 22:15:51.802123 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:15:51.802116 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bsdjz\" (UniqueName: \"kubernetes.io/projected/3a476bd5-1f8c-47dd-8fac-c44d2dcea62d-kube-api-access-bsdjz\") pod \"3a476bd5-1f8c-47dd-8fac-c44d2dcea62d\" (UID: \"3a476bd5-1f8c-47dd-8fac-c44d2dcea62d\") " Jul 09 22:15:51.802397 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:15:51.802292 2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3a476bd5-1f8c-47dd-8fac-c44d2dcea62d-console-config" (OuterVolumeSpecName: "console-config") pod "3a476bd5-1f8c-47dd-8fac-c44d2dcea62d" (UID: "3a476bd5-1f8c-47dd-8fac-c44d2dcea62d"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 09 22:15:51.802519 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:15:51.802489 2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3a476bd5-1f8c-47dd-8fac-c44d2dcea62d-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "3a476bd5-1f8c-47dd-8fac-c44d2dcea62d" (UID: "3a476bd5-1f8c-47dd-8fac-c44d2dcea62d"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 09 22:15:51.802831 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:15:51.802783 2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3a476bd5-1f8c-47dd-8fac-c44d2dcea62d-service-ca" (OuterVolumeSpecName: "service-ca") pod "3a476bd5-1f8c-47dd-8fac-c44d2dcea62d" (UID: "3a476bd5-1f8c-47dd-8fac-c44d2dcea62d"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 09 22:15:51.804688 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:15:51.804645 2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3a476bd5-1f8c-47dd-8fac-c44d2dcea62d-kube-api-access-bsdjz" (OuterVolumeSpecName: "kube-api-access-bsdjz") pod "3a476bd5-1f8c-47dd-8fac-c44d2dcea62d" (UID: "3a476bd5-1f8c-47dd-8fac-c44d2dcea62d"). InnerVolumeSpecName "kube-api-access-bsdjz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 22:15:51.804809 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:15:51.804760 2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3a476bd5-1f8c-47dd-8fac-c44d2dcea62d-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "3a476bd5-1f8c-47dd-8fac-c44d2dcea62d" (UID: "3a476bd5-1f8c-47dd-8fac-c44d2dcea62d"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 09 22:15:51.804809 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:15:51.804774 2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3a476bd5-1f8c-47dd-8fac-c44d2dcea62d-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "3a476bd5-1f8c-47dd-8fac-c44d2dcea62d" (UID: "3a476bd5-1f8c-47dd-8fac-c44d2dcea62d"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 09 22:15:51.903628 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:15:51.903591 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bsdjz\" (UniqueName: \"kubernetes.io/projected/3a476bd5-1f8c-47dd-8fac-c44d2dcea62d-kube-api-access-bsdjz\") on node \"ip-10-0-136-251.ec2.internal\" DevicePath \"\"" Jul 09 22:15:51.903628 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:15:51.903622 2579 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/3a476bd5-1f8c-47dd-8fac-c44d2dcea62d-console-config\") on node \"ip-10-0-136-251.ec2.internal\" DevicePath \"\"" Jul 09 22:15:51.903628 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:15:51.903632 2579 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/3a476bd5-1f8c-47dd-8fac-c44d2dcea62d-console-serving-cert\") on node \"ip-10-0-136-251.ec2.internal\" DevicePath \"\"" Jul 09 22:15:51.903854 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:15:51.903641 2579 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/3a476bd5-1f8c-47dd-8fac-c44d2dcea62d-console-oauth-config\") on node \"ip-10-0-136-251.ec2.internal\" DevicePath \"\"" Jul 09 22:15:51.903854 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:15:51.903650 2579 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/3a476bd5-1f8c-47dd-8fac-c44d2dcea62d-oauth-serving-cert\") on node \"ip-10-0-136-251.ec2.internal\" DevicePath \"\"" Jul 09 22:15:51.903854 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:15:51.903658 2579 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/3a476bd5-1f8c-47dd-8fac-c44d2dcea62d-service-ca\") on node \"ip-10-0-136-251.ec2.internal\" DevicePath \"\"" Jul 09 22:15:52.690214 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:15:52.690179 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7445556d44-f87lt" Jul 09 22:15:52.690214 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:15:52.690197 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7445556d44-f87lt" event={"ID":"3a476bd5-1f8c-47dd-8fac-c44d2dcea62d","Type":"ContainerDied","Data":"de832b196adf67e28d021a488d729ab031951364ca6b675e8b8a47683ba64ffa"} Jul 09 22:15:52.690712 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:15:52.690247 2579 scope.go:117] "RemoveContainer" containerID="36b512474158f14a2bf4998812aebb0410cdda0c68dcf790b5f27641d4c158b8" Jul 09 22:15:52.708641 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:15:52.708610 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7445556d44-f87lt"] Jul 09 22:15:52.714301 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:15:52.714278 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-7445556d44-f87lt"] Jul 09 22:15:54.563614 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:15:54.563578 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3a476bd5-1f8c-47dd-8fac-c44d2dcea62d" path="/var/lib/kubelet/pods/3a476bd5-1f8c-47dd-8fac-c44d2dcea62d/volumes" Jul 09 22:16:10.762586 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:16:10.762548 2579 generic.go:356] "Generic (PLEG): container finished" podID="7096ce79-14fb-4803-ab44-3a2eb01adf74" containerID="79bb1cdd0541f9e7edc8a9f42de4ea9f4a77e41c3dec5fe3285ac40ff9e82479" exitCode=0 Jul 09 22:16:10.763053 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:16:10.762614 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-xnjc7" event={"ID":"7096ce79-14fb-4803-ab44-3a2eb01adf74","Type":"ContainerDied","Data":"79bb1cdd0541f9e7edc8a9f42de4ea9f4a77e41c3dec5fe3285ac40ff9e82479"} Jul 09 22:16:10.763053 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:16:10.762958 2579 scope.go:117] "RemoveContainer" containerID="79bb1cdd0541f9e7edc8a9f42de4ea9f4a77e41c3dec5fe3285ac40ff9e82479" Jul 09 22:16:11.767995 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:16:11.767958 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-xnjc7" event={"ID":"7096ce79-14fb-4803-ab44-3a2eb01adf74","Type":"ContainerStarted","Data":"b566f5659a395b35dd208c4f6a8a0dc6158cf57d67abc7477d645e112358682e"} Jul 09 22:16:22.466753 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:16:22.466713 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/23b023ec-a254-462d-a0b4-5c22fd905e72-metrics-certs\") pod \"network-metrics-daemon-8pj95\" (UID: \"23b023ec-a254-462d-a0b4-5c22fd905e72\") " pod="openshift-multus/network-metrics-daemon-8pj95" Jul 09 22:16:22.469048 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:16:22.469027 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/23b023ec-a254-462d-a0b4-5c22fd905e72-metrics-certs\") pod \"network-metrics-daemon-8pj95\" (UID: \"23b023ec-a254-462d-a0b4-5c22fd905e72\") " pod="openshift-multus/network-metrics-daemon-8pj95" Jul 09 22:16:22.663175 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:16:22.663138 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-jtrvn\"" Jul 09 22:16:22.670407 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:16:22.670379 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8pj95" Jul 09 22:16:22.794933 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:16:22.794897 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-8pj95"] Jul 09 22:16:22.797164 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:16:22.797134 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod23b023ec_a254_462d_a0b4_5c22fd905e72.slice/crio-c549c75b25528274a44e00921448af8ad60df1e3457f1aaa8d7d1000865cf421 WatchSource:0}: Error finding container c549c75b25528274a44e00921448af8ad60df1e3457f1aaa8d7d1000865cf421: Status 404 returned error can't find the container with id c549c75b25528274a44e00921448af8ad60df1e3457f1aaa8d7d1000865cf421 Jul 09 22:16:22.813163 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:16:22.813138 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-8pj95" event={"ID":"23b023ec-a254-462d-a0b4-5c22fd905e72","Type":"ContainerStarted","Data":"c549c75b25528274a44e00921448af8ad60df1e3457f1aaa8d7d1000865cf421"} Jul 09 22:16:24.824800 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:16:24.824760 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-8pj95" event={"ID":"23b023ec-a254-462d-a0b4-5c22fd905e72","Type":"ContainerStarted","Data":"0edd626902edd9e96bc8f0147921b0a3b32034ad782c4f95e6fff14ea45ebe2d"} Jul 09 22:16:24.825263 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:16:24.824825 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-8pj95" event={"ID":"23b023ec-a254-462d-a0b4-5c22fd905e72","Type":"ContainerStarted","Data":"bcea95e10e2b146ec3c74e81e697546d91f7f93e341b05d665a1181b2602a198"} Jul 09 22:16:24.842296 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:16:24.842245 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-8pj95" podStartSLOduration=375.806965226 podStartE2EDuration="6m16.842229879s" totalImagesPullingTime="1.035264653s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 22:10:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 22:16:24.840944048 +0000 UTC m=+376.752396770" watchObservedRunningTime="2026-07-09 22:16:24.842229879 +0000 UTC m=+376.753682578" Jul 09 22:16:35.571550 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:16:35.571510 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0" Jul 09 22:16:35.587129 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:16:35.587105 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0" Jul 09 22:16:35.885939 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:16:35.885838 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0" Jul 09 22:19:30.836605 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:19:30.836569 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-99qx4"] Jul 09 22:19:30.837111 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:19:30.836885 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e8d76e6f-c1a2-4c70-8c4e-ce0b0480d75a" containerName="registry" Jul 09 22:19:30.837111 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:19:30.836896 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="e8d76e6f-c1a2-4c70-8c4e-ce0b0480d75a" containerName="registry" Jul 09 22:19:30.837111 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:19:30.836908 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3a476bd5-1f8c-47dd-8fac-c44d2dcea62d" containerName="console" Jul 09 22:19:30.837111 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:19:30.836913 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="3a476bd5-1f8c-47dd-8fac-c44d2dcea62d" containerName="console" Jul 09 22:19:30.837111 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:19:30.837002 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="3a476bd5-1f8c-47dd-8fac-c44d2dcea62d" containerName="console" Jul 09 22:19:30.837111 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:19:30.837013 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="e8d76e6f-c1a2-4c70-8c4e-ce0b0480d75a" containerName="registry" Jul 09 22:19:30.839989 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:19:30.839971 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-99qx4" Jul 09 22:19:30.842868 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:19:30.842848 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\"" Jul 09 22:19:30.843608 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:19:30.843591 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\"" Jul 09 22:19:30.844192 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:19:30.844177 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"openshift-lws-operator-dockercfg-q9vr5\"" Jul 09 22:19:30.853661 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:19:30.853638 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-99qx4"] Jul 09 22:19:31.012372 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:19:31.012334 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b4e0a4e6-0a52-4def-9fde-0d3b5ab3805f-tmp\") pod \"openshift-lws-operator-fd8ccff4c-99qx4\" (UID: \"b4e0a4e6-0a52-4def-9fde-0d3b5ab3805f\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-99qx4" Jul 09 22:19:31.012547 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:19:31.012397 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gwtn4\" (UniqueName: \"kubernetes.io/projected/b4e0a4e6-0a52-4def-9fde-0d3b5ab3805f-kube-api-access-gwtn4\") pod \"openshift-lws-operator-fd8ccff4c-99qx4\" (UID: \"b4e0a4e6-0a52-4def-9fde-0d3b5ab3805f\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-99qx4" Jul 09 22:19:31.113528 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:19:31.113427 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b4e0a4e6-0a52-4def-9fde-0d3b5ab3805f-tmp\") pod \"openshift-lws-operator-fd8ccff4c-99qx4\" (UID: \"b4e0a4e6-0a52-4def-9fde-0d3b5ab3805f\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-99qx4" Jul 09 22:19:31.113528 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:19:31.113505 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gwtn4\" (UniqueName: \"kubernetes.io/projected/b4e0a4e6-0a52-4def-9fde-0d3b5ab3805f-kube-api-access-gwtn4\") pod \"openshift-lws-operator-fd8ccff4c-99qx4\" (UID: \"b4e0a4e6-0a52-4def-9fde-0d3b5ab3805f\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-99qx4" Jul 09 22:19:31.113828 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:19:31.113806 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b4e0a4e6-0a52-4def-9fde-0d3b5ab3805f-tmp\") pod \"openshift-lws-operator-fd8ccff4c-99qx4\" (UID: \"b4e0a4e6-0a52-4def-9fde-0d3b5ab3805f\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-99qx4" Jul 09 22:19:31.122539 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:19:31.122506 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gwtn4\" (UniqueName: \"kubernetes.io/projected/b4e0a4e6-0a52-4def-9fde-0d3b5ab3805f-kube-api-access-gwtn4\") pod \"openshift-lws-operator-fd8ccff4c-99qx4\" (UID: \"b4e0a4e6-0a52-4def-9fde-0d3b5ab3805f\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-99qx4" Jul 09 22:19:31.162910 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:19:31.162876 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-99qx4" Jul 09 22:19:31.298517 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:19:31.298494 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-99qx4"] Jul 09 22:19:31.300673 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:19:31.300645 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb4e0a4e6_0a52_4def_9fde_0d3b5ab3805f.slice/crio-7aee87db5edec823e8d126679a580df32bbf017ae5909ff26b33b78cf831ba42 WatchSource:0}: Error finding container 7aee87db5edec823e8d126679a580df32bbf017ae5909ff26b33b78cf831ba42: Status 404 returned error can't find the container with id 7aee87db5edec823e8d126679a580df32bbf017ae5909ff26b33b78cf831ba42 Jul 09 22:19:31.542121 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:19:31.542085 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-99qx4" event={"ID":"b4e0a4e6-0a52-4def-9fde-0d3b5ab3805f","Type":"ContainerStarted","Data":"7aee87db5edec823e8d126679a580df32bbf017ae5909ff26b33b78cf831ba42"} Jul 09 22:19:34.557702 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:19:34.557655 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-99qx4" event={"ID":"b4e0a4e6-0a52-4def-9fde-0d3b5ab3805f","Type":"ContainerStarted","Data":"8f3a66bf0b17445b164a78cb655d2e4b4f70ab3f1bb0d07adba1c3b1123c757a"} Jul 09 22:19:54.125847 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:19:54.125788 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-99qx4" podStartSLOduration=21.513040003 podStartE2EDuration="24.125771465s" totalImagesPullingTime="2.612731462s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 22:19:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 22:19:34.57543087 +0000 UTC m=+566.486883571" watchObservedRunningTime="2026-07-09 22:19:54.125771465 +0000 UTC m=+586.037224165" Jul 09 22:19:54.126250 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:19:54.126229 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-5cf875db85-78gjk"] Jul 09 22:19:54.129552 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:19:54.129533 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-5cf875db85-78gjk" Jul 09 22:19:54.135552 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:19:54.135525 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Jul 09 22:19:54.135688 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:19:54.135535 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Jul 09 22:19:54.135688 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:19:54.135590 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Jul 09 22:19:54.135981 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:19:54.135963 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-8n8ll\"" Jul 09 22:19:54.136306 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:19:54.136290 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Jul 09 22:19:54.150533 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:19:54.150508 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-5cf875db85-78gjk"] Jul 09 22:19:54.211160 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:19:54.211123 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0083cb04-279c-4d25-bd33-b639a0b1a0f4-tmp\") pod \"opendatahub-operator-controller-manager-5cf875db85-78gjk\" (UID: \"0083cb04-279c-4d25-bd33-b639a0b1a0f4\") " pod="opendatahub/opendatahub-operator-controller-manager-5cf875db85-78gjk" Jul 09 22:19:54.211160 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:19:54.211159 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5xsrh\" (UniqueName: \"kubernetes.io/projected/0083cb04-279c-4d25-bd33-b639a0b1a0f4-kube-api-access-5xsrh\") pod \"opendatahub-operator-controller-manager-5cf875db85-78gjk\" (UID: \"0083cb04-279c-4d25-bd33-b639a0b1a0f4\") " pod="opendatahub/opendatahub-operator-controller-manager-5cf875db85-78gjk" Jul 09 22:19:54.211376 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:19:54.211188 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/0083cb04-279c-4d25-bd33-b639a0b1a0f4-manifests\") pod \"opendatahub-operator-controller-manager-5cf875db85-78gjk\" (UID: \"0083cb04-279c-4d25-bd33-b639a0b1a0f4\") " pod="opendatahub/opendatahub-operator-controller-manager-5cf875db85-78gjk" Jul 09 22:19:54.211376 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:19:54.211254 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/0083cb04-279c-4d25-bd33-b639a0b1a0f4-apiservice-cert\") pod \"opendatahub-operator-controller-manager-5cf875db85-78gjk\" (UID: \"0083cb04-279c-4d25-bd33-b639a0b1a0f4\") " pod="opendatahub/opendatahub-operator-controller-manager-5cf875db85-78gjk" Jul 09 22:19:54.211376 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:19:54.211295 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/0083cb04-279c-4d25-bd33-b639a0b1a0f4-webhook-cert\") pod \"opendatahub-operator-controller-manager-5cf875db85-78gjk\" (UID: \"0083cb04-279c-4d25-bd33-b639a0b1a0f4\") " pod="opendatahub/opendatahub-operator-controller-manager-5cf875db85-78gjk" Jul 09 22:19:54.312432 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:19:54.312397 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/0083cb04-279c-4d25-bd33-b639a0b1a0f4-manifests\") pod \"opendatahub-operator-controller-manager-5cf875db85-78gjk\" (UID: \"0083cb04-279c-4d25-bd33-b639a0b1a0f4\") " pod="opendatahub/opendatahub-operator-controller-manager-5cf875db85-78gjk" Jul 09 22:19:54.312608 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:19:54.312442 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/0083cb04-279c-4d25-bd33-b639a0b1a0f4-apiservice-cert\") pod \"opendatahub-operator-controller-manager-5cf875db85-78gjk\" (UID: \"0083cb04-279c-4d25-bd33-b639a0b1a0f4\") " pod="opendatahub/opendatahub-operator-controller-manager-5cf875db85-78gjk" Jul 09 22:19:54.312608 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:19:54.312469 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/0083cb04-279c-4d25-bd33-b639a0b1a0f4-webhook-cert\") pod \"opendatahub-operator-controller-manager-5cf875db85-78gjk\" (UID: \"0083cb04-279c-4d25-bd33-b639a0b1a0f4\") " pod="opendatahub/opendatahub-operator-controller-manager-5cf875db85-78gjk" Jul 09 22:19:54.312608 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:19:54.312515 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0083cb04-279c-4d25-bd33-b639a0b1a0f4-tmp\") pod \"opendatahub-operator-controller-manager-5cf875db85-78gjk\" (UID: \"0083cb04-279c-4d25-bd33-b639a0b1a0f4\") " pod="opendatahub/opendatahub-operator-controller-manager-5cf875db85-78gjk" Jul 09 22:19:54.312608 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:19:54.312534 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5xsrh\" (UniqueName: \"kubernetes.io/projected/0083cb04-279c-4d25-bd33-b639a0b1a0f4-kube-api-access-5xsrh\") pod \"opendatahub-operator-controller-manager-5cf875db85-78gjk\" (UID: \"0083cb04-279c-4d25-bd33-b639a0b1a0f4\") " pod="opendatahub/opendatahub-operator-controller-manager-5cf875db85-78gjk" Jul 09 22:19:54.313019 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:19:54.313000 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/0083cb04-279c-4d25-bd33-b639a0b1a0f4-manifests\") pod \"opendatahub-operator-controller-manager-5cf875db85-78gjk\" (UID: \"0083cb04-279c-4d25-bd33-b639a0b1a0f4\") " pod="opendatahub/opendatahub-operator-controller-manager-5cf875db85-78gjk" Jul 09 22:19:54.314810 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:19:54.314779 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0083cb04-279c-4d25-bd33-b639a0b1a0f4-tmp\") pod \"opendatahub-operator-controller-manager-5cf875db85-78gjk\" (UID: \"0083cb04-279c-4d25-bd33-b639a0b1a0f4\") " pod="opendatahub/opendatahub-operator-controller-manager-5cf875db85-78gjk" Jul 09 22:19:54.315110 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:19:54.315091 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/0083cb04-279c-4d25-bd33-b639a0b1a0f4-apiservice-cert\") pod \"opendatahub-operator-controller-manager-5cf875db85-78gjk\" (UID: \"0083cb04-279c-4d25-bd33-b639a0b1a0f4\") " pod="opendatahub/opendatahub-operator-controller-manager-5cf875db85-78gjk" Jul 09 22:19:54.315214 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:19:54.315191 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/0083cb04-279c-4d25-bd33-b639a0b1a0f4-webhook-cert\") pod \"opendatahub-operator-controller-manager-5cf875db85-78gjk\" (UID: \"0083cb04-279c-4d25-bd33-b639a0b1a0f4\") " pod="opendatahub/opendatahub-operator-controller-manager-5cf875db85-78gjk" Jul 09 22:19:54.320944 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:19:54.320904 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5xsrh\" (UniqueName: \"kubernetes.io/projected/0083cb04-279c-4d25-bd33-b639a0b1a0f4-kube-api-access-5xsrh\") pod \"opendatahub-operator-controller-manager-5cf875db85-78gjk\" (UID: \"0083cb04-279c-4d25-bd33-b639a0b1a0f4\") " pod="opendatahub/opendatahub-operator-controller-manager-5cf875db85-78gjk" Jul 09 22:19:54.444849 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:19:54.444766 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-5cf875db85-78gjk" Jul 09 22:19:54.581563 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:19:54.581538 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-5cf875db85-78gjk"] Jul 09 22:19:54.584196 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:19:54.584167 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0083cb04_279c_4d25_bd33_b639a0b1a0f4.slice/crio-761687dcac5a428671bebf895b16fe1627bedaf68da5b47411aad402fa1b71f4 WatchSource:0}: Error finding container 761687dcac5a428671bebf895b16fe1627bedaf68da5b47411aad402fa1b71f4: Status 404 returned error can't find the container with id 761687dcac5a428671bebf895b16fe1627bedaf68da5b47411aad402fa1b71f4 Jul 09 22:19:54.635389 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:19:54.635354 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-5cf875db85-78gjk" event={"ID":"0083cb04-279c-4d25-bd33-b639a0b1a0f4","Type":"ContainerStarted","Data":"761687dcac5a428671bebf895b16fe1627bedaf68da5b47411aad402fa1b71f4"} Jul 09 22:19:57.651860 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:19:57.651820 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-5cf875db85-78gjk" event={"ID":"0083cb04-279c-4d25-bd33-b639a0b1a0f4","Type":"ContainerStarted","Data":"219f8113e1aa018af8f66b16010ecdb26fa974cf60af8b59c1a8f3cd0dbc9bbc"} Jul 09 22:19:58.657722 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:19:58.657696 2579 generic.go:356] "Generic (PLEG): container finished" podID="0083cb04-279c-4d25-bd33-b639a0b1a0f4" containerID="219f8113e1aa018af8f66b16010ecdb26fa974cf60af8b59c1a8f3cd0dbc9bbc" exitCode=0 Jul 09 22:19:58.658069 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:19:58.657738 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-5cf875db85-78gjk" event={"ID":"0083cb04-279c-4d25-bd33-b639a0b1a0f4","Type":"ContainerDied","Data":"219f8113e1aa018af8f66b16010ecdb26fa974cf60af8b59c1a8f3cd0dbc9bbc"} Jul 09 22:19:59.664558 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:19:59.664522 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-5cf875db85-78gjk" event={"ID":"0083cb04-279c-4d25-bd33-b639a0b1a0f4","Type":"ContainerStarted","Data":"8372eeed4b1f785aaf8ef08d0dbc92b21e68eda059d5f648cb4849f46be66f36"} Jul 09 22:19:59.665019 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:19:59.664628 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-5cf875db85-78gjk" Jul 09 22:19:59.697793 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:19:59.693209 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-5cf875db85-78gjk" podStartSLOduration=1.35161223 podStartE2EDuration="5.69317287s" totalImagesPullingTime="3.224637548s" totalInitContainerRuntime="1.116923092s" isStatefulPod=false podCreationTimestamp="2026-07-09 22:19:54 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 22:19:59.691216815 +0000 UTC m=+591.602669535" watchObservedRunningTime="2026-07-09 22:19:59.69317287 +0000 UTC m=+591.604625571" Jul 09 22:20:10.671367 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:20:10.671333 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-5cf875db85-78gjk" Jul 09 22:20:21.611162 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:20:21.611125 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-84cc848bd-shdfd"] Jul 09 22:20:21.616064 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:20:21.616040 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-84cc848bd-shdfd" Jul 09 22:20:21.619125 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:20:21.619105 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Jul 09 22:20:21.619512 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:20:21.619494 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"data-science-gateway-data-science-gateway-class-dockercfg-dpwhr\"" Jul 09 22:20:21.652825 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:20:21.652790 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-84cc848bd-shdfd"] Jul 09 22:20:21.752178 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:20:21.752141 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/38dc5337-f873-4426-9555-37d3d6a124b8-istio-data\") pod \"data-science-gateway-data-science-gateway-class-84cc848bd-shdfd\" (UID: \"38dc5337-f873-4426-9555-37d3d6a124b8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-84cc848bd-shdfd" Jul 09 22:20:21.752397 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:20:21.752203 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/38dc5337-f873-4426-9555-37d3d6a124b8-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-84cc848bd-shdfd\" (UID: \"38dc5337-f873-4426-9555-37d3d6a124b8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-84cc848bd-shdfd" Jul 09 22:20:21.752397 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:20:21.752243 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/38dc5337-f873-4426-9555-37d3d6a124b8-istio-token\") pod \"data-science-gateway-data-science-gateway-class-84cc848bd-shdfd\" (UID: \"38dc5337-f873-4426-9555-37d3d6a124b8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-84cc848bd-shdfd" Jul 09 22:20:21.752397 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:20:21.752287 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/38dc5337-f873-4426-9555-37d3d6a124b8-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-84cc848bd-shdfd\" (UID: \"38dc5337-f873-4426-9555-37d3d6a124b8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-84cc848bd-shdfd" Jul 09 22:20:21.752397 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:20:21.752328 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/38dc5337-f873-4426-9555-37d3d6a124b8-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-84cc848bd-shdfd\" (UID: \"38dc5337-f873-4426-9555-37d3d6a124b8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-84cc848bd-shdfd" Jul 09 22:20:21.752397 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:20:21.752358 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/38dc5337-f873-4426-9555-37d3d6a124b8-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-84cc848bd-shdfd\" (UID: \"38dc5337-f873-4426-9555-37d3d6a124b8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-84cc848bd-shdfd" Jul 09 22:20:21.752397 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:20:21.752381 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/38dc5337-f873-4426-9555-37d3d6a124b8-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-84cc848bd-shdfd\" (UID: \"38dc5337-f873-4426-9555-37d3d6a124b8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-84cc848bd-shdfd" Jul 09 22:20:21.752623 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:20:21.752408 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/38dc5337-f873-4426-9555-37d3d6a124b8-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-84cc848bd-shdfd\" (UID: \"38dc5337-f873-4426-9555-37d3d6a124b8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-84cc848bd-shdfd" Jul 09 22:20:21.752623 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:20:21.752440 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cx4pg\" (UniqueName: \"kubernetes.io/projected/38dc5337-f873-4426-9555-37d3d6a124b8-kube-api-access-cx4pg\") pod \"data-science-gateway-data-science-gateway-class-84cc848bd-shdfd\" (UID: \"38dc5337-f873-4426-9555-37d3d6a124b8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-84cc848bd-shdfd" Jul 09 22:20:21.853501 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:20:21.852884 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/38dc5337-f873-4426-9555-37d3d6a124b8-istio-data\") pod \"data-science-gateway-data-science-gateway-class-84cc848bd-shdfd\" (UID: \"38dc5337-f873-4426-9555-37d3d6a124b8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-84cc848bd-shdfd" Jul 09 22:20:21.853501 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:20:21.852979 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/38dc5337-f873-4426-9555-37d3d6a124b8-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-84cc848bd-shdfd\" (UID: \"38dc5337-f873-4426-9555-37d3d6a124b8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-84cc848bd-shdfd" Jul 09 22:20:21.853501 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:20:21.853100 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/38dc5337-f873-4426-9555-37d3d6a124b8-istio-token\") pod \"data-science-gateway-data-science-gateway-class-84cc848bd-shdfd\" (UID: \"38dc5337-f873-4426-9555-37d3d6a124b8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-84cc848bd-shdfd" Jul 09 22:20:21.853501 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:20:21.853182 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/38dc5337-f873-4426-9555-37d3d6a124b8-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-84cc848bd-shdfd\" (UID: \"38dc5337-f873-4426-9555-37d3d6a124b8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-84cc848bd-shdfd" Jul 09 22:20:21.853501 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:20:21.853461 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/38dc5337-f873-4426-9555-37d3d6a124b8-istio-data\") pod \"data-science-gateway-data-science-gateway-class-84cc848bd-shdfd\" (UID: \"38dc5337-f873-4426-9555-37d3d6a124b8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-84cc848bd-shdfd" Jul 09 22:20:21.853501 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:20:21.853503 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/38dc5337-f873-4426-9555-37d3d6a124b8-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-84cc848bd-shdfd\" (UID: \"38dc5337-f873-4426-9555-37d3d6a124b8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-84cc848bd-shdfd" Jul 09 22:20:21.853943 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:20:21.853550 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/38dc5337-f873-4426-9555-37d3d6a124b8-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-84cc848bd-shdfd\" (UID: \"38dc5337-f873-4426-9555-37d3d6a124b8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-84cc848bd-shdfd" Jul 09 22:20:21.853943 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:20:21.853592 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/38dc5337-f873-4426-9555-37d3d6a124b8-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-84cc848bd-shdfd\" (UID: \"38dc5337-f873-4426-9555-37d3d6a124b8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-84cc848bd-shdfd" Jul 09 22:20:21.853943 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:20:21.853620 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/38dc5337-f873-4426-9555-37d3d6a124b8-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-84cc848bd-shdfd\" (UID: \"38dc5337-f873-4426-9555-37d3d6a124b8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-84cc848bd-shdfd" Jul 09 22:20:21.853943 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:20:21.853647 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/38dc5337-f873-4426-9555-37d3d6a124b8-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-84cc848bd-shdfd\" (UID: \"38dc5337-f873-4426-9555-37d3d6a124b8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-84cc848bd-shdfd" Jul 09 22:20:21.853943 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:20:21.853683 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cx4pg\" (UniqueName: \"kubernetes.io/projected/38dc5337-f873-4426-9555-37d3d6a124b8-kube-api-access-cx4pg\") pod \"data-science-gateway-data-science-gateway-class-84cc848bd-shdfd\" (UID: \"38dc5337-f873-4426-9555-37d3d6a124b8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-84cc848bd-shdfd" Jul 09 22:20:21.853943 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:20:21.853789 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/38dc5337-f873-4426-9555-37d3d6a124b8-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-84cc848bd-shdfd\" (UID: \"38dc5337-f873-4426-9555-37d3d6a124b8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-84cc848bd-shdfd" Jul 09 22:20:21.854261 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:20:21.853982 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/38dc5337-f873-4426-9555-37d3d6a124b8-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-84cc848bd-shdfd\" (UID: \"38dc5337-f873-4426-9555-37d3d6a124b8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-84cc848bd-shdfd" Jul 09 22:20:21.854616 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:20:21.854593 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/38dc5337-f873-4426-9555-37d3d6a124b8-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-84cc848bd-shdfd\" (UID: \"38dc5337-f873-4426-9555-37d3d6a124b8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-84cc848bd-shdfd" Jul 09 22:20:21.855587 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:20:21.855545 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4x6nwz"] Jul 09 22:20:21.856249 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:20:21.856227 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/38dc5337-f873-4426-9555-37d3d6a124b8-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-84cc848bd-shdfd\" (UID: \"38dc5337-f873-4426-9555-37d3d6a124b8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-84cc848bd-shdfd" Jul 09 22:20:21.856603 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:20:21.856582 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/38dc5337-f873-4426-9555-37d3d6a124b8-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-84cc848bd-shdfd\" (UID: \"38dc5337-f873-4426-9555-37d3d6a124b8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-84cc848bd-shdfd" Jul 09 22:20:21.859388 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:20:21.859369 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4x6nwz" Jul 09 22:20:21.868636 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:20:21.868578 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/38dc5337-f873-4426-9555-37d3d6a124b8-istio-token\") pod \"data-science-gateway-data-science-gateway-class-84cc848bd-shdfd\" (UID: \"38dc5337-f873-4426-9555-37d3d6a124b8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-84cc848bd-shdfd" Jul 09 22:20:21.869858 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:20:21.869822 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cx4pg\" (UniqueName: \"kubernetes.io/projected/38dc5337-f873-4426-9555-37d3d6a124b8-kube-api-access-cx4pg\") pod \"data-science-gateway-data-science-gateway-class-84cc848bd-shdfd\" (UID: \"38dc5337-f873-4426-9555-37d3d6a124b8\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-84cc848bd-shdfd" Jul 09 22:20:21.879636 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:20:21.879608 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4x6nwz"] Jul 09 22:20:21.927019 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:20:21.926975 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-84cc848bd-shdfd" Jul 09 22:20:21.955138 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:20:21.955100 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-94bvh\" (UniqueName: \"kubernetes.io/projected/bc747b34-dcf6-43d4-989f-35f49f02b32d-kube-api-access-94bvh\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4x6nwz\" (UID: \"bc747b34-dcf6-43d4-989f-35f49f02b32d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4x6nwz" Jul 09 22:20:21.955327 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:20:21.955169 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bc747b34-dcf6-43d4-989f-35f49f02b32d-istio-data\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4x6nwz\" (UID: \"bc747b34-dcf6-43d4-989f-35f49f02b32d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4x6nwz" Jul 09 22:20:21.955327 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:20:21.955221 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bc747b34-dcf6-43d4-989f-35f49f02b32d-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4x6nwz\" (UID: \"bc747b34-dcf6-43d4-989f-35f49f02b32d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4x6nwz" Jul 09 22:20:21.955327 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:20:21.955266 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bc747b34-dcf6-43d4-989f-35f49f02b32d-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4x6nwz\" (UID: \"bc747b34-dcf6-43d4-989f-35f49f02b32d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4x6nwz" Jul 09 22:20:21.955327 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:20:21.955294 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bc747b34-dcf6-43d4-989f-35f49f02b32d-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4x6nwz\" (UID: \"bc747b34-dcf6-43d4-989f-35f49f02b32d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4x6nwz" Jul 09 22:20:21.955502 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:20:21.955337 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc747b34-dcf6-43d4-989f-35f49f02b32d-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4x6nwz\" (UID: \"bc747b34-dcf6-43d4-989f-35f49f02b32d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4x6nwz" Jul 09 22:20:21.955502 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:20:21.955362 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc747b34-dcf6-43d4-989f-35f49f02b32d-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4x6nwz\" (UID: \"bc747b34-dcf6-43d4-989f-35f49f02b32d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4x6nwz" Jul 09 22:20:21.955502 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:20:21.955394 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bc747b34-dcf6-43d4-989f-35f49f02b32d-istio-token\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4x6nwz\" (UID: \"bc747b34-dcf6-43d4-989f-35f49f02b32d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4x6nwz" Jul 09 22:20:21.955502 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:20:21.955425 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bc747b34-dcf6-43d4-989f-35f49f02b32d-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4x6nwz\" (UID: \"bc747b34-dcf6-43d4-989f-35f49f02b32d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4x6nwz" Jul 09 22:20:22.056325 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:20:22.056288 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-94bvh\" (UniqueName: \"kubernetes.io/projected/bc747b34-dcf6-43d4-989f-35f49f02b32d-kube-api-access-94bvh\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4x6nwz\" (UID: \"bc747b34-dcf6-43d4-989f-35f49f02b32d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4x6nwz" Jul 09 22:20:22.056485 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:20:22.056342 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bc747b34-dcf6-43d4-989f-35f49f02b32d-istio-data\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4x6nwz\" (UID: \"bc747b34-dcf6-43d4-989f-35f49f02b32d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4x6nwz" Jul 09 22:20:22.056485 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:20:22.056366 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bc747b34-dcf6-43d4-989f-35f49f02b32d-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4x6nwz\" (UID: \"bc747b34-dcf6-43d4-989f-35f49f02b32d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4x6nwz" Jul 09 22:20:22.056485 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:20:22.056397 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bc747b34-dcf6-43d4-989f-35f49f02b32d-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4x6nwz\" (UID: \"bc747b34-dcf6-43d4-989f-35f49f02b32d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4x6nwz" Jul 09 22:20:22.056485 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:20:22.056417 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bc747b34-dcf6-43d4-989f-35f49f02b32d-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4x6nwz\" (UID: \"bc747b34-dcf6-43d4-989f-35f49f02b32d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4x6nwz" Jul 09 22:20:22.056641 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:20:22.056554 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc747b34-dcf6-43d4-989f-35f49f02b32d-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4x6nwz\" (UID: \"bc747b34-dcf6-43d4-989f-35f49f02b32d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4x6nwz" Jul 09 22:20:22.056641 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:20:22.056595 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc747b34-dcf6-43d4-989f-35f49f02b32d-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4x6nwz\" (UID: \"bc747b34-dcf6-43d4-989f-35f49f02b32d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4x6nwz" Jul 09 22:20:22.056641 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:20:22.056632 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bc747b34-dcf6-43d4-989f-35f49f02b32d-istio-token\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4x6nwz\" (UID: \"bc747b34-dcf6-43d4-989f-35f49f02b32d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4x6nwz" Jul 09 22:20:22.056743 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:20:22.056668 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bc747b34-dcf6-43d4-989f-35f49f02b32d-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4x6nwz\" (UID: \"bc747b34-dcf6-43d4-989f-35f49f02b32d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4x6nwz" Jul 09 22:20:22.056827 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:20:22.056805 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bc747b34-dcf6-43d4-989f-35f49f02b32d-istio-data\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4x6nwz\" (UID: \"bc747b34-dcf6-43d4-989f-35f49f02b32d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4x6nwz" Jul 09 22:20:22.056899 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:20:22.056821 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bc747b34-dcf6-43d4-989f-35f49f02b32d-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4x6nwz\" (UID: \"bc747b34-dcf6-43d4-989f-35f49f02b32d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4x6nwz" Jul 09 22:20:22.056980 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:20:22.056908 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc747b34-dcf6-43d4-989f-35f49f02b32d-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4x6nwz\" (UID: \"bc747b34-dcf6-43d4-989f-35f49f02b32d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4x6nwz" Jul 09 22:20:22.057099 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:20:22.057076 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc747b34-dcf6-43d4-989f-35f49f02b32d-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4x6nwz\" (UID: \"bc747b34-dcf6-43d4-989f-35f49f02b32d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4x6nwz" Jul 09 22:20:22.057314 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:20:22.057298 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bc747b34-dcf6-43d4-989f-35f49f02b32d-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4x6nwz\" (UID: \"bc747b34-dcf6-43d4-989f-35f49f02b32d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4x6nwz" Jul 09 22:20:22.059248 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:20:22.059219 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bc747b34-dcf6-43d4-989f-35f49f02b32d-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4x6nwz\" (UID: \"bc747b34-dcf6-43d4-989f-35f49f02b32d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4x6nwz" Jul 09 22:20:22.059444 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:20:22.059429 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bc747b34-dcf6-43d4-989f-35f49f02b32d-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4x6nwz\" (UID: \"bc747b34-dcf6-43d4-989f-35f49f02b32d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4x6nwz" Jul 09 22:20:22.070311 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:20:22.070284 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-94bvh\" (UniqueName: \"kubernetes.io/projected/bc747b34-dcf6-43d4-989f-35f49f02b32d-kube-api-access-94bvh\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4x6nwz\" (UID: \"bc747b34-dcf6-43d4-989f-35f49f02b32d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4x6nwz" Jul 09 22:20:22.071708 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:20:22.071680 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bc747b34-dcf6-43d4-989f-35f49f02b32d-istio-token\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4x6nwz\" (UID: \"bc747b34-dcf6-43d4-989f-35f49f02b32d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4x6nwz" Jul 09 22:20:22.077759 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:20:22.077739 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-84cc848bd-shdfd"] Jul 09 22:20:22.080541 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:20:22.080516 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod38dc5337_f873_4426_9555_37d3d6a124b8.slice/crio-7045872a0b8a9a16be4b2ad11194b4198209bd8f2247bf82274a6ec569776dc2 WatchSource:0}: Error finding container 7045872a0b8a9a16be4b2ad11194b4198209bd8f2247bf82274a6ec569776dc2: Status 404 returned error can't find the container with id 7045872a0b8a9a16be4b2ad11194b4198209bd8f2247bf82274a6ec569776dc2 Jul 09 22:20:22.082600 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:20:22.082582 2579 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 09 22:20:22.193528 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:20:22.193440 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4x6nwz" Jul 09 22:20:22.347323 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:20:22.347299 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4x6nwz"] Jul 09 22:20:22.349856 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:20:22.349815 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbc747b34_dcf6_43d4_989f_35f49f02b32d.slice/crio-316098a52ea96f545c9e409e80fa7c7c7ae02dc68faf5d0c96334231e83ace4f WatchSource:0}: Error finding container 316098a52ea96f545c9e409e80fa7c7c7ae02dc68faf5d0c96334231e83ace4f: Status 404 returned error can't find the container with id 316098a52ea96f545c9e409e80fa7c7c7ae02dc68faf5d0c96334231e83ace4f Jul 09 22:20:22.772212 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:20:22.772107 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-84cc848bd-shdfd" event={"ID":"38dc5337-f873-4426-9555-37d3d6a124b8","Type":"ContainerStarted","Data":"7045872a0b8a9a16be4b2ad11194b4198209bd8f2247bf82274a6ec569776dc2"} Jul 09 22:20:22.773744 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:20:22.773681 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4x6nwz" event={"ID":"bc747b34-dcf6-43d4-989f-35f49f02b32d","Type":"ContainerStarted","Data":"316098a52ea96f545c9e409e80fa7c7c7ae02dc68faf5d0c96334231e83ace4f"} Jul 09 22:20:24.597207 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:20:24.597173 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-b6tv8"] Jul 09 22:20:24.600868 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:20:24.600849 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-b6tv8" Jul 09 22:20:24.603814 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:20:24.603792 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-controller-manager-dockercfg-4zqpt\"" Jul 09 22:20:24.604081 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:20:24.604066 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-webhook-server-cert\"" Jul 09 22:20:24.613243 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:20:24.613216 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-btzms"] Jul 09 22:20:24.617131 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:20:24.617108 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-b6tv8"] Jul 09 22:20:24.617260 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:20:24.617206 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-btzms" Jul 09 22:20:24.619550 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:20:24.619529 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-controller-manager-dockercfg-k26pt\"" Jul 09 22:20:24.619844 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:20:24.619826 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-webhook-server-cert\"" Jul 09 22:20:24.631799 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:20:24.631767 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-btzms"] Jul 09 22:20:24.655976 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:20:24.655901 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 09 22:20:24.656233 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:20:24.656014 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 09 22:20:24.656233 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:20:24.656057 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 09 22:20:24.781383 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:20:24.781345 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-28897\" (UniqueName: \"kubernetes.io/projected/6a02a146-e107-4650-828d-e092ad914d87-kube-api-access-28897\") pod \"kserve-controller-manager-d6f4b65b8-b6tv8\" (UID: \"6a02a146-e107-4650-828d-e092ad914d87\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-b6tv8" Jul 09 22:20:24.781593 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:20:24.781409 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/29c83de2-afa8-48fa-b975-93747a089549-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-btzms\" (UID: \"29c83de2-afa8-48fa-b975-93747a089549\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-btzms" Jul 09 22:20:24.781593 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:20:24.781482 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6a02a146-e107-4650-828d-e092ad914d87-cert\") pod \"kserve-controller-manager-d6f4b65b8-b6tv8\" (UID: \"6a02a146-e107-4650-828d-e092ad914d87\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-b6tv8" Jul 09 22:20:24.781593 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:20:24.781526 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gfqt9\" (UniqueName: \"kubernetes.io/projected/29c83de2-afa8-48fa-b975-93747a089549-kube-api-access-gfqt9\") pod \"llmisvc-controller-manager-777c6bd7d5-btzms\" (UID: \"29c83de2-afa8-48fa-b975-93747a089549\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-btzms" Jul 09 22:20:24.788377 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:20:24.788341 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-84cc848bd-shdfd" event={"ID":"38dc5337-f873-4426-9555-37d3d6a124b8","Type":"ContainerStarted","Data":"8aa3157933aff49f19b71f63f809d588a2e132bc7d38bba28bfe015f42d28f46"} Jul 09 22:20:24.848463 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:20:24.848304 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-84cc848bd-shdfd" podStartSLOduration=1.275412971 podStartE2EDuration="3.848284459s" totalImagesPullingTime="2.572871488s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 22:20:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 22:20:24.847706272 +0000 UTC m=+616.759158975" watchObservedRunningTime="2026-07-09 22:20:24.848284459 +0000 UTC m=+616.759737159" Jul 09 22:20:24.882666 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:20:24.882628 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6a02a146-e107-4650-828d-e092ad914d87-cert\") pod \"kserve-controller-manager-d6f4b65b8-b6tv8\" (UID: \"6a02a146-e107-4650-828d-e092ad914d87\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-b6tv8" Jul 09 22:20:24.882666 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:20:24.882674 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gfqt9\" (UniqueName: \"kubernetes.io/projected/29c83de2-afa8-48fa-b975-93747a089549-kube-api-access-gfqt9\") pod \"llmisvc-controller-manager-777c6bd7d5-btzms\" (UID: \"29c83de2-afa8-48fa-b975-93747a089549\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-btzms" Jul 09 22:20:24.883078 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:20:24.883056 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-28897\" (UniqueName: \"kubernetes.io/projected/6a02a146-e107-4650-828d-e092ad914d87-kube-api-access-28897\") pod \"kserve-controller-manager-d6f4b65b8-b6tv8\" (UID: \"6a02a146-e107-4650-828d-e092ad914d87\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-b6tv8" Jul 09 22:20:24.883157 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:20:24.883128 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/29c83de2-afa8-48fa-b975-93747a089549-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-btzms\" (UID: \"29c83de2-afa8-48fa-b975-93747a089549\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-btzms" Jul 09 22:20:24.883512 ip-10-0-136-251 kubenswrapper[2579]: E0709 22:20:24.883246 2579 secret.go:189] Couldn't get secret opendatahub/llmisvc-webhook-server-cert: secret "llmisvc-webhook-server-cert" not found Jul 09 22:20:24.883512 ip-10-0-136-251 kubenswrapper[2579]: E0709 22:20:24.883323 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/29c83de2-afa8-48fa-b975-93747a089549-cert podName:29c83de2-afa8-48fa-b975-93747a089549 nodeName:}" failed. No retries permitted until 2026-07-09 22:20:25.38329978 +0000 UTC m=+617.294752467 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/29c83de2-afa8-48fa-b975-93747a089549-cert") pod "llmisvc-controller-manager-777c6bd7d5-btzms" (UID: "29c83de2-afa8-48fa-b975-93747a089549") : secret "llmisvc-webhook-server-cert" not found Jul 09 22:20:24.885903 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:20:24.885880 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6a02a146-e107-4650-828d-e092ad914d87-cert\") pod \"kserve-controller-manager-d6f4b65b8-b6tv8\" (UID: \"6a02a146-e107-4650-828d-e092ad914d87\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-b6tv8" Jul 09 22:20:24.907420 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:20:24.907384 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-28897\" (UniqueName: \"kubernetes.io/projected/6a02a146-e107-4650-828d-e092ad914d87-kube-api-access-28897\") pod \"kserve-controller-manager-d6f4b65b8-b6tv8\" (UID: \"6a02a146-e107-4650-828d-e092ad914d87\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-b6tv8" Jul 09 22:20:24.910743 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:20:24.910717 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gfqt9\" (UniqueName: \"kubernetes.io/projected/29c83de2-afa8-48fa-b975-93747a089549-kube-api-access-gfqt9\") pod \"llmisvc-controller-manager-777c6bd7d5-btzms\" (UID: \"29c83de2-afa8-48fa-b975-93747a089549\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-btzms" Jul 09 22:20:24.912598 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:20:24.912578 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-b6tv8" Jul 09 22:20:24.928009 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:20:24.927985 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-84cc848bd-shdfd" Jul 09 22:20:25.000682 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:20:25.000648 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-76dfc7bb46-sxtvx"] Jul 09 22:20:25.004589 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:20:25.004564 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-76dfc7bb46-sxtvx" Jul 09 22:20:25.010580 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:20:25.010533 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-bnfgw\"" Jul 09 22:20:25.011399 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:20:25.011372 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\"" Jul 09 22:20:25.013290 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:20:25.013246 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\"" Jul 09 22:20:25.015228 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:20:25.015210 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\"" Jul 09 22:20:25.032177 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:20:25.032139 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-76dfc7bb46-sxtvx"] Jul 09 22:20:25.121616 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:20:25.121589 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-b6tv8"] Jul 09 22:20:25.122966 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:20:25.122933 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6a02a146_e107_4650_828d_e092ad914d87.slice/crio-1c69ca97b6294f8806cab0b7eab019aa406ba16b07bf8eb6eee35da35922f925 WatchSource:0}: Error finding container 1c69ca97b6294f8806cab0b7eab019aa406ba16b07bf8eb6eee35da35922f925: Status 404 returned error can't find the container with id 1c69ca97b6294f8806cab0b7eab019aa406ba16b07bf8eb6eee35da35922f925 Jul 09 22:20:25.185601 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:20:25.185562 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/7994ef1d-61df-4510-9779-a5b6ff8e723c-manager-config\") pod \"lws-controller-manager-76dfc7bb46-sxtvx\" (UID: \"7994ef1d-61df-4510-9779-a5b6ff8e723c\") " pod="openshift-lws-operator/lws-controller-manager-76dfc7bb46-sxtvx" Jul 09 22:20:25.185764 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:20:25.185614 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kddg4\" (UniqueName: \"kubernetes.io/projected/7994ef1d-61df-4510-9779-a5b6ff8e723c-kube-api-access-kddg4\") pod \"lws-controller-manager-76dfc7bb46-sxtvx\" (UID: \"7994ef1d-61df-4510-9779-a5b6ff8e723c\") " pod="openshift-lws-operator/lws-controller-manager-76dfc7bb46-sxtvx" Jul 09 22:20:25.185764 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:20:25.185697 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/7994ef1d-61df-4510-9779-a5b6ff8e723c-metrics-cert\") pod \"lws-controller-manager-76dfc7bb46-sxtvx\" (UID: \"7994ef1d-61df-4510-9779-a5b6ff8e723c\") " pod="openshift-lws-operator/lws-controller-manager-76dfc7bb46-sxtvx" Jul 09 22:20:25.185839 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:20:25.185769 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7994ef1d-61df-4510-9779-a5b6ff8e723c-cert\") pod \"lws-controller-manager-76dfc7bb46-sxtvx\" (UID: \"7994ef1d-61df-4510-9779-a5b6ff8e723c\") " pod="openshift-lws-operator/lws-controller-manager-76dfc7bb46-sxtvx" Jul 09 22:20:25.287039 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:20:25.287000 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7994ef1d-61df-4510-9779-a5b6ff8e723c-cert\") pod \"lws-controller-manager-76dfc7bb46-sxtvx\" (UID: \"7994ef1d-61df-4510-9779-a5b6ff8e723c\") " pod="openshift-lws-operator/lws-controller-manager-76dfc7bb46-sxtvx" Jul 09 22:20:25.287212 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:20:25.287045 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/7994ef1d-61df-4510-9779-a5b6ff8e723c-manager-config\") pod \"lws-controller-manager-76dfc7bb46-sxtvx\" (UID: \"7994ef1d-61df-4510-9779-a5b6ff8e723c\") " pod="openshift-lws-operator/lws-controller-manager-76dfc7bb46-sxtvx" Jul 09 22:20:25.287212 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:20:25.287096 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kddg4\" (UniqueName: \"kubernetes.io/projected/7994ef1d-61df-4510-9779-a5b6ff8e723c-kube-api-access-kddg4\") pod \"lws-controller-manager-76dfc7bb46-sxtvx\" (UID: \"7994ef1d-61df-4510-9779-a5b6ff8e723c\") " pod="openshift-lws-operator/lws-controller-manager-76dfc7bb46-sxtvx" Jul 09 22:20:25.287212 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:20:25.287165 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/7994ef1d-61df-4510-9779-a5b6ff8e723c-metrics-cert\") pod \"lws-controller-manager-76dfc7bb46-sxtvx\" (UID: \"7994ef1d-61df-4510-9779-a5b6ff8e723c\") " pod="openshift-lws-operator/lws-controller-manager-76dfc7bb46-sxtvx" Jul 09 22:20:25.287769 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:20:25.287737 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/7994ef1d-61df-4510-9779-a5b6ff8e723c-manager-config\") pod \"lws-controller-manager-76dfc7bb46-sxtvx\" (UID: \"7994ef1d-61df-4510-9779-a5b6ff8e723c\") " pod="openshift-lws-operator/lws-controller-manager-76dfc7bb46-sxtvx" Jul 09 22:20:25.289531 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:20:25.289512 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/7994ef1d-61df-4510-9779-a5b6ff8e723c-metrics-cert\") pod \"lws-controller-manager-76dfc7bb46-sxtvx\" (UID: \"7994ef1d-61df-4510-9779-a5b6ff8e723c\") " pod="openshift-lws-operator/lws-controller-manager-76dfc7bb46-sxtvx" Jul 09 22:20:25.289624 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:20:25.289542 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7994ef1d-61df-4510-9779-a5b6ff8e723c-cert\") pod \"lws-controller-manager-76dfc7bb46-sxtvx\" (UID: \"7994ef1d-61df-4510-9779-a5b6ff8e723c\") " pod="openshift-lws-operator/lws-controller-manager-76dfc7bb46-sxtvx" Jul 09 22:20:25.299607 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:20:25.299573 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kddg4\" (UniqueName: \"kubernetes.io/projected/7994ef1d-61df-4510-9779-a5b6ff8e723c-kube-api-access-kddg4\") pod \"lws-controller-manager-76dfc7bb46-sxtvx\" (UID: \"7994ef1d-61df-4510-9779-a5b6ff8e723c\") " pod="openshift-lws-operator/lws-controller-manager-76dfc7bb46-sxtvx" Jul 09 22:20:25.317516 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:20:25.317484 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-76dfc7bb46-sxtvx" Jul 09 22:20:25.388150 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:20:25.387905 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/29c83de2-afa8-48fa-b975-93747a089549-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-btzms\" (UID: \"29c83de2-afa8-48fa-b975-93747a089549\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-btzms" Jul 09 22:20:25.390726 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:20:25.390700 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/29c83de2-afa8-48fa-b975-93747a089549-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-btzms\" (UID: \"29c83de2-afa8-48fa-b975-93747a089549\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-btzms" Jul 09 22:20:25.483558 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:20:25.483527 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7994ef1d_61df_4510_9779_a5b6ff8e723c.slice/crio-1eb84339430cfdc7db281f34b093a5dae5ef2bde05982e4d7c686c0289e9fcf9 WatchSource:0}: Error finding container 1eb84339430cfdc7db281f34b093a5dae5ef2bde05982e4d7c686c0289e9fcf9: Status 404 returned error can't find the container with id 1eb84339430cfdc7db281f34b093a5dae5ef2bde05982e4d7c686c0289e9fcf9 Jul 09 22:20:25.483760 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:20:25.483739 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-76dfc7bb46-sxtvx"] Jul 09 22:20:25.527311 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:20:25.527281 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-btzms" Jul 09 22:20:25.679947 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:20:25.679906 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-btzms"] Jul 09 22:20:25.682315 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:20:25.682291 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod29c83de2_afa8_48fa_b975_93747a089549.slice/crio-1186a1c73552c2ad51c74a6690259b45b4e791a24e80e81941599fe6bef2e8a2 WatchSource:0}: Error finding container 1186a1c73552c2ad51c74a6690259b45b4e791a24e80e81941599fe6bef2e8a2: Status 404 returned error can't find the container with id 1186a1c73552c2ad51c74a6690259b45b4e791a24e80e81941599fe6bef2e8a2 Jul 09 22:20:25.795953 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:20:25.795896 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-76dfc7bb46-sxtvx" event={"ID":"7994ef1d-61df-4510-9779-a5b6ff8e723c","Type":"ContainerStarted","Data":"1eb84339430cfdc7db281f34b093a5dae5ef2bde05982e4d7c686c0289e9fcf9"} Jul 09 22:20:25.797075 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:20:25.797043 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-b6tv8" event={"ID":"6a02a146-e107-4650-828d-e092ad914d87","Type":"ContainerStarted","Data":"1c69ca97b6294f8806cab0b7eab019aa406ba16b07bf8eb6eee35da35922f925"} Jul 09 22:20:25.798087 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:20:25.798066 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-btzms" event={"ID":"29c83de2-afa8-48fa-b975-93747a089549","Type":"ContainerStarted","Data":"1186a1c73552c2ad51c74a6690259b45b4e791a24e80e81941599fe6bef2e8a2"} Jul 09 22:20:25.928477 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:20:25.928445 2579 patch_prober.go:28] interesting pod/data-science-gateway-data-science-gateway-class-84cc848bd-shdfd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.24:15021/healthz/ready\": dial tcp 10.132.0.24:15021: connect: connection refused" start-of-body= Jul 09 22:20:25.928635 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:20:25.928500 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-84cc848bd-shdfd" podUID="38dc5337-f873-4426-9555-37d3d6a124b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.24:15021/healthz/ready\": dial tcp 10.132.0.24:15021: connect: connection refused" Jul 09 22:20:26.076791 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:20:26.076750 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 09 22:20:26.076909 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:20:26.076843 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 09 22:20:26.076909 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:20:26.076881 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 09 22:20:26.805648 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:20:26.805609 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4x6nwz" event={"ID":"bc747b34-dcf6-43d4-989f-35f49f02b32d","Type":"ContainerStarted","Data":"09dcbb584b21f3614a1ba840a38b4f02603296927c6a11a14cf80364687c4f6c"} Jul 09 22:20:26.835578 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:20:26.835523 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4x6nwz" podStartSLOduration=2.11103853 podStartE2EDuration="5.835501829s" totalImagesPullingTime="3.724463299s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 22:20:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 22:20:26.833722808 +0000 UTC m=+618.745175529" watchObservedRunningTime="2026-07-09 22:20:26.835501829 +0000 UTC m=+618.746954530" Jul 09 22:20:26.927472 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:20:26.927422 2579 patch_prober.go:28] interesting pod/data-science-gateway-data-science-gateway-class-84cc848bd-shdfd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.24:15021/healthz/ready\": dial tcp 10.132.0.24:15021: connect: connection refused" start-of-body= Jul 09 22:20:26.927652 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:20:26.927490 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-84cc848bd-shdfd" podUID="38dc5337-f873-4426-9555-37d3d6a124b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.24:15021/healthz/ready\": dial tcp 10.132.0.24:15021: connect: connection refused" Jul 09 22:20:27.194742 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:20:27.194643 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4x6nwz" Jul 09 22:20:27.200629 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:20:27.200598 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4x6nwz" Jul 09 22:20:27.813331 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:20:27.813303 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4x6nwz" Jul 09 22:20:27.814377 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:20:27.814349 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4x6nwz" Jul 09 22:20:27.871170 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:20:27.871127 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-84cc848bd-shdfd"] Jul 09 22:20:27.871383 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:20:27.871345 2579 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-84cc848bd-shdfd" podUID="38dc5337-f873-4426-9555-37d3d6a124b8" containerName="istio-proxy" containerID="cri-o://8aa3157933aff49f19b71f63f809d588a2e132bc7d38bba28bfe015f42d28f46" gracePeriod=30 Jul 09 22:20:28.820291 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:20:28.820237 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-76dfc7bb46-sxtvx" event={"ID":"7994ef1d-61df-4510-9779-a5b6ff8e723c","Type":"ContainerStarted","Data":"715574b02d6a5d00b97f44666b59bf078b75e6a0f8a485dd1bfac5733f684aa9"} Jul 09 22:20:28.820806 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:20:28.820370 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-76dfc7bb46-sxtvx" Jul 09 22:20:28.822070 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:20:28.822038 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-b6tv8" event={"ID":"6a02a146-e107-4650-828d-e092ad914d87","Type":"ContainerStarted","Data":"f22883a054905e724fdb457d68b3c3cb90ed016d482cf2a73cefc2fc5e112589"} Jul 09 22:20:28.822291 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:20:28.822272 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-b6tv8" Jul 09 22:20:28.843344 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:20:28.843257 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-76dfc7bb46-sxtvx" podStartSLOduration=2.35869932 podStartE2EDuration="4.843235305s" totalImagesPullingTime="2.484535985s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 22:20:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 22:20:28.842125503 +0000 UTC m=+620.753578204" watchObservedRunningTime="2026-07-09 22:20:28.843235305 +0000 UTC m=+620.754688006" Jul 09 22:20:28.869275 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:20:28.869215 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/kserve-controller-manager-d6f4b65b8-b6tv8" podStartSLOduration=1.8261210989999999 podStartE2EDuration="4.869192152s" totalImagesPullingTime="3.043071053s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 22:20:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 22:20:28.868507848 +0000 UTC m=+620.779960549" watchObservedRunningTime="2026-07-09 22:20:28.869192152 +0000 UTC m=+620.780644853" Jul 09 22:20:30.833066 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:20:30.832975 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-btzms" event={"ID":"29c83de2-afa8-48fa-b975-93747a089549","Type":"ContainerStarted","Data":"e1ae2647b39404d1e03f113df1e87fb6708d66f173171a7265097c12dba18275"} Jul 09 22:20:30.833544 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:20:30.833145 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-btzms" Jul 09 22:20:30.906183 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:20:30.906118 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-btzms" podStartSLOduration=2.903023449 podStartE2EDuration="6.906101248s" totalImagesPullingTime="4.003077799s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 22:20:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 22:20:30.90316824 +0000 UTC m=+622.814620940" watchObservedRunningTime="2026-07-09 22:20:30.906101248 +0000 UTC m=+622.817553978" Jul 09 22:20:33.123760 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:20:33.123732 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-84cc848bd-shdfd" Jul 09 22:20:33.258851 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:20:33.258818 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/38dc5337-f873-4426-9555-37d3d6a124b8-istio-token\") pod \"38dc5337-f873-4426-9555-37d3d6a124b8\" (UID: \"38dc5337-f873-4426-9555-37d3d6a124b8\") " Jul 09 22:20:33.259039 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:20:33.258858 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/38dc5337-f873-4426-9555-37d3d6a124b8-istio-envoy\") pod \"38dc5337-f873-4426-9555-37d3d6a124b8\" (UID: \"38dc5337-f873-4426-9555-37d3d6a124b8\") " Jul 09 22:20:33.259039 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:20:33.258911 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/38dc5337-f873-4426-9555-37d3d6a124b8-workload-socket\") pod \"38dc5337-f873-4426-9555-37d3d6a124b8\" (UID: \"38dc5337-f873-4426-9555-37d3d6a124b8\") " Jul 09 22:20:33.259039 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:20:33.258989 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/38dc5337-f873-4426-9555-37d3d6a124b8-istio-data\") pod \"38dc5337-f873-4426-9555-37d3d6a124b8\" (UID: \"38dc5337-f873-4426-9555-37d3d6a124b8\") " Jul 09 22:20:33.259165 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:20:33.259040 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/38dc5337-f873-4426-9555-37d3d6a124b8-istio-podinfo\") pod \"38dc5337-f873-4426-9555-37d3d6a124b8\" (UID: \"38dc5337-f873-4426-9555-37d3d6a124b8\") " Jul 09 22:20:33.259165 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:20:33.259081 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/38dc5337-f873-4426-9555-37d3d6a124b8-workload-certs\") pod \"38dc5337-f873-4426-9555-37d3d6a124b8\" (UID: \"38dc5337-f873-4426-9555-37d3d6a124b8\") " Jul 09 22:20:33.259165 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:20:33.259115 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cx4pg\" (UniqueName: \"kubernetes.io/projected/38dc5337-f873-4426-9555-37d3d6a124b8-kube-api-access-cx4pg\") pod \"38dc5337-f873-4426-9555-37d3d6a124b8\" (UID: \"38dc5337-f873-4426-9555-37d3d6a124b8\") " Jul 09 22:20:33.259165 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:20:33.259142 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/38dc5337-f873-4426-9555-37d3d6a124b8-istiod-ca-cert\") pod \"38dc5337-f873-4426-9555-37d3d6a124b8\" (UID: \"38dc5337-f873-4426-9555-37d3d6a124b8\") " Jul 09 22:20:33.259370 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:20:33.259175 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/38dc5337-f873-4426-9555-37d3d6a124b8-credential-socket\") pod \"38dc5337-f873-4426-9555-37d3d6a124b8\" (UID: \"38dc5337-f873-4426-9555-37d3d6a124b8\") " Jul 09 22:20:33.259370 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:20:33.259242 2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/38dc5337-f873-4426-9555-37d3d6a124b8-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "38dc5337-f873-4426-9555-37d3d6a124b8" (UID: "38dc5337-f873-4426-9555-37d3d6a124b8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 22:20:33.259370 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:20:33.259347 2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/38dc5337-f873-4426-9555-37d3d6a124b8-istio-data" (OuterVolumeSpecName: "istio-data") pod "38dc5337-f873-4426-9555-37d3d6a124b8" (UID: "38dc5337-f873-4426-9555-37d3d6a124b8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 22:20:33.259528 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:20:33.259508 2579 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/38dc5337-f873-4426-9555-37d3d6a124b8-workload-socket\") on node \"ip-10-0-136-251.ec2.internal\" DevicePath \"\"" Jul 09 22:20:33.259585 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:20:33.259535 2579 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/38dc5337-f873-4426-9555-37d3d6a124b8-istio-data\") on node \"ip-10-0-136-251.ec2.internal\" DevicePath \"\"" Jul 09 22:20:33.259585 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:20:33.259537 2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/38dc5337-f873-4426-9555-37d3d6a124b8-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "38dc5337-f873-4426-9555-37d3d6a124b8" (UID: "38dc5337-f873-4426-9555-37d3d6a124b8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 09 22:20:33.259585 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:20:33.259544 2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/38dc5337-f873-4426-9555-37d3d6a124b8-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "38dc5337-f873-4426-9555-37d3d6a124b8" (UID: "38dc5337-f873-4426-9555-37d3d6a124b8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 22:20:33.259753 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:20:33.259735 2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/38dc5337-f873-4426-9555-37d3d6a124b8-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "38dc5337-f873-4426-9555-37d3d6a124b8" (UID: "38dc5337-f873-4426-9555-37d3d6a124b8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 22:20:33.261357 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:20:33.261331 2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/38dc5337-f873-4426-9555-37d3d6a124b8-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "38dc5337-f873-4426-9555-37d3d6a124b8" (UID: "38dc5337-f873-4426-9555-37d3d6a124b8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 22:20:33.261518 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:20:33.261499 2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/38dc5337-f873-4426-9555-37d3d6a124b8-istio-token" (OuterVolumeSpecName: "istio-token") pod "38dc5337-f873-4426-9555-37d3d6a124b8" (UID: "38dc5337-f873-4426-9555-37d3d6a124b8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 22:20:33.261654 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:20:33.261636 2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/38dc5337-f873-4426-9555-37d3d6a124b8-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "38dc5337-f873-4426-9555-37d3d6a124b8" (UID: "38dc5337-f873-4426-9555-37d3d6a124b8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 09 22:20:33.261699 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:20:33.261675 2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/38dc5337-f873-4426-9555-37d3d6a124b8-kube-api-access-cx4pg" (OuterVolumeSpecName: "kube-api-access-cx4pg") pod "38dc5337-f873-4426-9555-37d3d6a124b8" (UID: "38dc5337-f873-4426-9555-37d3d6a124b8"). InnerVolumeSpecName "kube-api-access-cx4pg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 22:20:33.360225 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:20:33.360189 2579 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/38dc5337-f873-4426-9555-37d3d6a124b8-istio-podinfo\") on node \"ip-10-0-136-251.ec2.internal\" DevicePath \"\"" Jul 09 22:20:33.360225 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:20:33.360221 2579 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/38dc5337-f873-4426-9555-37d3d6a124b8-workload-certs\") on node \"ip-10-0-136-251.ec2.internal\" DevicePath \"\"" Jul 09 22:20:33.360225 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:20:33.360231 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cx4pg\" (UniqueName: \"kubernetes.io/projected/38dc5337-f873-4426-9555-37d3d6a124b8-kube-api-access-cx4pg\") on node \"ip-10-0-136-251.ec2.internal\" DevicePath \"\"" Jul 09 22:20:33.360442 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:20:33.360243 2579 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/38dc5337-f873-4426-9555-37d3d6a124b8-istiod-ca-cert\") on node \"ip-10-0-136-251.ec2.internal\" DevicePath \"\"" Jul 09 22:20:33.360442 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:20:33.360252 2579 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/38dc5337-f873-4426-9555-37d3d6a124b8-credential-socket\") on node \"ip-10-0-136-251.ec2.internal\" DevicePath \"\"" Jul 09 22:20:33.360442 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:20:33.360260 2579 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/38dc5337-f873-4426-9555-37d3d6a124b8-istio-token\") on node \"ip-10-0-136-251.ec2.internal\" DevicePath \"\"" Jul 09 22:20:33.360442 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:20:33.360268 2579 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/38dc5337-f873-4426-9555-37d3d6a124b8-istio-envoy\") on node \"ip-10-0-136-251.ec2.internal\" DevicePath \"\"" Jul 09 22:20:33.848494 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:20:33.848457 2579 generic.go:356] "Generic (PLEG): container finished" podID="38dc5337-f873-4426-9555-37d3d6a124b8" containerID="8aa3157933aff49f19b71f63f809d588a2e132bc7d38bba28bfe015f42d28f46" exitCode=0 Jul 09 22:20:33.848664 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:20:33.848509 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-84cc848bd-shdfd" Jul 09 22:20:33.848664 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:20:33.848520 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-84cc848bd-shdfd" event={"ID":"38dc5337-f873-4426-9555-37d3d6a124b8","Type":"ContainerDied","Data":"8aa3157933aff49f19b71f63f809d588a2e132bc7d38bba28bfe015f42d28f46"} Jul 09 22:20:33.848664 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:20:33.848569 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-84cc848bd-shdfd" event={"ID":"38dc5337-f873-4426-9555-37d3d6a124b8","Type":"ContainerDied","Data":"7045872a0b8a9a16be4b2ad11194b4198209bd8f2247bf82274a6ec569776dc2"} Jul 09 22:20:33.848664 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:20:33.848593 2579 scope.go:117] "RemoveContainer" containerID="8aa3157933aff49f19b71f63f809d588a2e132bc7d38bba28bfe015f42d28f46" Jul 09 22:20:33.857759 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:20:33.857740 2579 scope.go:117] "RemoveContainer" containerID="8aa3157933aff49f19b71f63f809d588a2e132bc7d38bba28bfe015f42d28f46" Jul 09 22:20:33.858081 ip-10-0-136-251 kubenswrapper[2579]: E0709 22:20:33.858064 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8aa3157933aff49f19b71f63f809d588a2e132bc7d38bba28bfe015f42d28f46\": container with ID starting with 8aa3157933aff49f19b71f63f809d588a2e132bc7d38bba28bfe015f42d28f46 not found: ID does not exist" containerID="8aa3157933aff49f19b71f63f809d588a2e132bc7d38bba28bfe015f42d28f46" Jul 09 22:20:33.858130 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:20:33.858091 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8aa3157933aff49f19b71f63f809d588a2e132bc7d38bba28bfe015f42d28f46"} err="failed to get container status \"8aa3157933aff49f19b71f63f809d588a2e132bc7d38bba28bfe015f42d28f46\": rpc error: code = NotFound desc = could not find container \"8aa3157933aff49f19b71f63f809d588a2e132bc7d38bba28bfe015f42d28f46\": container with ID starting with 8aa3157933aff49f19b71f63f809d588a2e132bc7d38bba28bfe015f42d28f46 not found: ID does not exist" Jul 09 22:20:33.869819 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:20:33.869791 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-84cc848bd-shdfd"] Jul 09 22:20:33.874364 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:20:33.874342 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-84cc848bd-shdfd"] Jul 09 22:20:34.565975 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:20:34.565905 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="38dc5337-f873-4426-9555-37d3d6a124b8" path="/var/lib/kubelet/pods/38dc5337-f873-4426-9555-37d3d6a124b8/volumes" Jul 09 22:20:39.829989 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:20:39.829960 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-76dfc7bb46-sxtvx" Jul 09 22:20:59.832302 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:20:59.832272 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-b6tv8" Jul 09 22:21:01.840169 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:21:01.840133 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-btzms" Jul 09 22:21:54.098891 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:21:54.098850 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-zm6ft"] Jul 09 22:21:54.099359 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:21:54.099215 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="38dc5337-f873-4426-9555-37d3d6a124b8" containerName="istio-proxy" Jul 09 22:21:54.099359 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:21:54.099227 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="38dc5337-f873-4426-9555-37d3d6a124b8" containerName="istio-proxy" Jul 09 22:21:54.099359 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:21:54.099304 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="38dc5337-f873-4426-9555-37d3d6a124b8" containerName="istio-proxy" Jul 09 22:21:54.102391 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:21:54.102375 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-75cc8d5cb7-zm6ft" Jul 09 22:21:54.106390 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:21:54.106367 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Jul 09 22:21:54.106390 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:21:54.106381 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Jul 09 22:21:54.107534 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:21:54.107520 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-operator-dockercfg-fdt8q\"" Jul 09 22:21:54.134768 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:21:54.134735 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-zm6ft"] Jul 09 22:21:54.138986 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:21:54.138958 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-95tdl\" (UniqueName: \"kubernetes.io/projected/c97c2fa5-5376-4f0a-bdc3-c8d69720ab75-kube-api-access-95tdl\") pod \"authorino-operator-75cc8d5cb7-zm6ft\" (UID: \"c97c2fa5-5376-4f0a-bdc3-c8d69720ab75\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-zm6ft" Jul 09 22:21:54.240038 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:21:54.239999 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-95tdl\" (UniqueName: \"kubernetes.io/projected/c97c2fa5-5376-4f0a-bdc3-c8d69720ab75-kube-api-access-95tdl\") pod \"authorino-operator-75cc8d5cb7-zm6ft\" (UID: \"c97c2fa5-5376-4f0a-bdc3-c8d69720ab75\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-zm6ft" Jul 09 22:21:54.254548 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:21:54.254512 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-95tdl\" (UniqueName: \"kubernetes.io/projected/c97c2fa5-5376-4f0a-bdc3-c8d69720ab75-kube-api-access-95tdl\") pod \"authorino-operator-75cc8d5cb7-zm6ft\" (UID: \"c97c2fa5-5376-4f0a-bdc3-c8d69720ab75\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-zm6ft" Jul 09 22:21:54.412401 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:21:54.412305 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-75cc8d5cb7-zm6ft" Jul 09 22:21:54.547864 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:21:54.547840 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-zm6ft"] Jul 09 22:21:54.550078 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:21:54.550051 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc97c2fa5_5376_4f0a_bdc3_c8d69720ab75.slice/crio-85729b9b74515bfa8308f927cdf48e11adf930eaee832fcdbb696503194d78d0 WatchSource:0}: Error finding container 85729b9b74515bfa8308f927cdf48e11adf930eaee832fcdbb696503194d78d0: Status 404 returned error can't find the container with id 85729b9b74515bfa8308f927cdf48e11adf930eaee832fcdbb696503194d78d0 Jul 09 22:21:55.212114 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:21:55.212075 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-75cc8d5cb7-zm6ft" event={"ID":"c97c2fa5-5376-4f0a-bdc3-c8d69720ab75","Type":"ContainerStarted","Data":"85729b9b74515bfa8308f927cdf48e11adf930eaee832fcdbb696503194d78d0"} Jul 09 22:21:57.224737 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:21:57.224645 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-75cc8d5cb7-zm6ft" event={"ID":"c97c2fa5-5376-4f0a-bdc3-c8d69720ab75","Type":"ContainerStarted","Data":"8ebf114916b65f5575b4a7de9e3c1f42cc0c1db62fd72b31d7e62cc4291298fa"} Jul 09 22:21:57.225115 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:21:57.224740 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/authorino-operator-75cc8d5cb7-zm6ft" Jul 09 22:21:57.249338 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:21:57.249288 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-operator-75cc8d5cb7-zm6ft" podStartSLOduration=0.929819443 podStartE2EDuration="3.249272145s" totalImagesPullingTime="2.319452702s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 22:21:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 22:21:57.24752563 +0000 UTC m=+709.158978327" watchObservedRunningTime="2026-07-09 22:21:57.249272145 +0000 UTC m=+709.160724842" Jul 09 22:21:57.355196 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:21:57.355162 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-q6sw8"] Jul 09 22:21:57.358678 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:21:57.358662 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-q6sw8" Jul 09 22:21:57.361837 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:21:57.361820 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\"" Jul 09 22:21:57.364840 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:21:57.364815 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-xshkj\"" Jul 09 22:21:57.387988 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:21:57.387961 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-q6sw8"] Jul 09 22:21:57.469272 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:21:57.469237 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8lcfz\" (UniqueName: \"kubernetes.io/projected/af0b3a7a-21c8-43ae-95d0-2f09e3dee7f4-kube-api-access-8lcfz\") pod \"dns-operator-controller-manager-864dd78c4c-q6sw8\" (UID: \"af0b3a7a-21c8-43ae-95d0-2f09e3dee7f4\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-q6sw8" Jul 09 22:21:57.570385 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:21:57.570350 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8lcfz\" (UniqueName: \"kubernetes.io/projected/af0b3a7a-21c8-43ae-95d0-2f09e3dee7f4-kube-api-access-8lcfz\") pod \"dns-operator-controller-manager-864dd78c4c-q6sw8\" (UID: \"af0b3a7a-21c8-43ae-95d0-2f09e3dee7f4\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-q6sw8" Jul 09 22:21:57.583049 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:21:57.583022 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8lcfz\" (UniqueName: \"kubernetes.io/projected/af0b3a7a-21c8-43ae-95d0-2f09e3dee7f4-kube-api-access-8lcfz\") pod \"dns-operator-controller-manager-864dd78c4c-q6sw8\" (UID: \"af0b3a7a-21c8-43ae-95d0-2f09e3dee7f4\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-q6sw8" Jul 09 22:21:57.668842 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:21:57.668805 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-q6sw8" Jul 09 22:21:57.801621 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:21:57.801598 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-q6sw8"] Jul 09 22:21:57.803757 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:21:57.803728 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaf0b3a7a_21c8_43ae_95d0_2f09e3dee7f4.slice/crio-4f0643a133593ee37b753bce81f022305f8b63dbe4b962c909512ba9cf24a592 WatchSource:0}: Error finding container 4f0643a133593ee37b753bce81f022305f8b63dbe4b962c909512ba9cf24a592: Status 404 returned error can't find the container with id 4f0643a133593ee37b753bce81f022305f8b63dbe4b962c909512ba9cf24a592 Jul 09 22:21:58.229983 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:21:58.229945 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-q6sw8" event={"ID":"af0b3a7a-21c8-43ae-95d0-2f09e3dee7f4","Type":"ContainerStarted","Data":"4f0643a133593ee37b753bce81f022305f8b63dbe4b962c909512ba9cf24a592"} Jul 09 22:22:00.241727 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:22:00.241687 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-q6sw8" event={"ID":"af0b3a7a-21c8-43ae-95d0-2f09e3dee7f4","Type":"ContainerStarted","Data":"0fbbecb09e81d37eefa24b071ea97828aafc5e3f051244621827a8781f71cbff"} Jul 09 22:22:00.242267 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:22:00.241763 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-q6sw8" Jul 09 22:22:00.263461 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:22:00.263414 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-q6sw8" podStartSLOduration=1.395262019 podStartE2EDuration="3.263400101s" totalImagesPullingTime="1.868138082s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 22:21:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 22:22:00.261528398 +0000 UTC m=+712.172981095" watchObservedRunningTime="2026-07-09 22:22:00.263400101 +0000 UTC m=+712.174852799" Jul 09 22:22:08.231910 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:22:08.231878 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/authorino-operator-75cc8d5cb7-zm6ft" Jul 09 22:22:11.248964 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:22:11.248905 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-q6sw8" Jul 09 22:22:39.529742 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:22:39.529708 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-j9x6z"] Jul 09 22:22:39.532114 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:22:39.532096 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-j9x6z" Jul 09 22:22:39.534944 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:22:39.534909 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-d4jjp\"" Jul 09 22:22:39.550043 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:22:39.550019 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-j9x6z"] Jul 09 22:22:39.637907 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:22:39.637871 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9a0d3041-24b6-42f9-9992-32cc62e3810e-workload-certs\") pod \"maas-default-gateway-openshift-default-56654dcf7-j9x6z\" (UID: \"9a0d3041-24b6-42f9-9992-32cc62e3810e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-j9x6z" Jul 09 22:22:39.638106 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:22:39.637938 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9a0d3041-24b6-42f9-9992-32cc62e3810e-istio-envoy\") pod \"maas-default-gateway-openshift-default-56654dcf7-j9x6z\" (UID: \"9a0d3041-24b6-42f9-9992-32cc62e3810e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-j9x6z" Jul 09 22:22:39.638106 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:22:39.637968 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9a0d3041-24b6-42f9-9992-32cc62e3810e-istio-data\") pod \"maas-default-gateway-openshift-default-56654dcf7-j9x6z\" (UID: \"9a0d3041-24b6-42f9-9992-32cc62e3810e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-j9x6z" Jul 09 22:22:39.638106 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:22:39.637989 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9a0d3041-24b6-42f9-9992-32cc62e3810e-istio-token\") pod \"maas-default-gateway-openshift-default-56654dcf7-j9x6z\" (UID: \"9a0d3041-24b6-42f9-9992-32cc62e3810e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-j9x6z" Jul 09 22:22:39.638106 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:22:39.638009 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x7cf2\" (UniqueName: \"kubernetes.io/projected/9a0d3041-24b6-42f9-9992-32cc62e3810e-kube-api-access-x7cf2\") pod \"maas-default-gateway-openshift-default-56654dcf7-j9x6z\" (UID: \"9a0d3041-24b6-42f9-9992-32cc62e3810e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-j9x6z" Jul 09 22:22:39.638255 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:22:39.638121 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a0d3041-24b6-42f9-9992-32cc62e3810e-credential-socket\") pod \"maas-default-gateway-openshift-default-56654dcf7-j9x6z\" (UID: \"9a0d3041-24b6-42f9-9992-32cc62e3810e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-j9x6z" Jul 09 22:22:39.638255 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:22:39.638171 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9a0d3041-24b6-42f9-9992-32cc62e3810e-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-56654dcf7-j9x6z\" (UID: \"9a0d3041-24b6-42f9-9992-32cc62e3810e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-j9x6z" Jul 09 22:22:39.638255 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:22:39.638196 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9a0d3041-24b6-42f9-9992-32cc62e3810e-istio-podinfo\") pod \"maas-default-gateway-openshift-default-56654dcf7-j9x6z\" (UID: \"9a0d3041-24b6-42f9-9992-32cc62e3810e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-j9x6z" Jul 09 22:22:39.638255 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:22:39.638215 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a0d3041-24b6-42f9-9992-32cc62e3810e-workload-socket\") pod \"maas-default-gateway-openshift-default-56654dcf7-j9x6z\" (UID: \"9a0d3041-24b6-42f9-9992-32cc62e3810e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-j9x6z" Jul 09 22:22:39.739252 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:22:39.739204 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9a0d3041-24b6-42f9-9992-32cc62e3810e-workload-certs\") pod \"maas-default-gateway-openshift-default-56654dcf7-j9x6z\" (UID: \"9a0d3041-24b6-42f9-9992-32cc62e3810e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-j9x6z" Jul 09 22:22:39.739252 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:22:39.739254 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9a0d3041-24b6-42f9-9992-32cc62e3810e-istio-envoy\") pod \"maas-default-gateway-openshift-default-56654dcf7-j9x6z\" (UID: \"9a0d3041-24b6-42f9-9992-32cc62e3810e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-j9x6z" Jul 09 22:22:39.739519 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:22:39.739271 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9a0d3041-24b6-42f9-9992-32cc62e3810e-istio-data\") pod \"maas-default-gateway-openshift-default-56654dcf7-j9x6z\" (UID: \"9a0d3041-24b6-42f9-9992-32cc62e3810e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-j9x6z" Jul 09 22:22:39.739519 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:22:39.739291 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9a0d3041-24b6-42f9-9992-32cc62e3810e-istio-token\") pod \"maas-default-gateway-openshift-default-56654dcf7-j9x6z\" (UID: \"9a0d3041-24b6-42f9-9992-32cc62e3810e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-j9x6z" Jul 09 22:22:39.739519 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:22:39.739320 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x7cf2\" (UniqueName: \"kubernetes.io/projected/9a0d3041-24b6-42f9-9992-32cc62e3810e-kube-api-access-x7cf2\") pod \"maas-default-gateway-openshift-default-56654dcf7-j9x6z\" (UID: \"9a0d3041-24b6-42f9-9992-32cc62e3810e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-j9x6z" Jul 09 22:22:39.739519 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:22:39.739390 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a0d3041-24b6-42f9-9992-32cc62e3810e-credential-socket\") pod \"maas-default-gateway-openshift-default-56654dcf7-j9x6z\" (UID: \"9a0d3041-24b6-42f9-9992-32cc62e3810e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-j9x6z" Jul 09 22:22:39.739519 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:22:39.739421 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9a0d3041-24b6-42f9-9992-32cc62e3810e-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-56654dcf7-j9x6z\" (UID: \"9a0d3041-24b6-42f9-9992-32cc62e3810e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-j9x6z" Jul 09 22:22:39.739519 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:22:39.739447 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9a0d3041-24b6-42f9-9992-32cc62e3810e-istio-podinfo\") pod \"maas-default-gateway-openshift-default-56654dcf7-j9x6z\" (UID: \"9a0d3041-24b6-42f9-9992-32cc62e3810e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-j9x6z" Jul 09 22:22:39.739519 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:22:39.739473 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a0d3041-24b6-42f9-9992-32cc62e3810e-workload-socket\") pod \"maas-default-gateway-openshift-default-56654dcf7-j9x6z\" (UID: \"9a0d3041-24b6-42f9-9992-32cc62e3810e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-j9x6z" Jul 09 22:22:39.739996 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:22:39.739678 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9a0d3041-24b6-42f9-9992-32cc62e3810e-workload-certs\") pod \"maas-default-gateway-openshift-default-56654dcf7-j9x6z\" (UID: \"9a0d3041-24b6-42f9-9992-32cc62e3810e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-j9x6z" Jul 09 22:22:39.739996 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:22:39.739838 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a0d3041-24b6-42f9-9992-32cc62e3810e-workload-socket\") pod \"maas-default-gateway-openshift-default-56654dcf7-j9x6z\" (UID: \"9a0d3041-24b6-42f9-9992-32cc62e3810e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-j9x6z" Jul 09 22:22:39.739996 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:22:39.739882 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a0d3041-24b6-42f9-9992-32cc62e3810e-credential-socket\") pod \"maas-default-gateway-openshift-default-56654dcf7-j9x6z\" (UID: \"9a0d3041-24b6-42f9-9992-32cc62e3810e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-j9x6z" Jul 09 22:22:39.739996 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:22:39.739987 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9a0d3041-24b6-42f9-9992-32cc62e3810e-istio-data\") pod \"maas-default-gateway-openshift-default-56654dcf7-j9x6z\" (UID: \"9a0d3041-24b6-42f9-9992-32cc62e3810e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-j9x6z" Jul 09 22:22:39.740199 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:22:39.740088 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9a0d3041-24b6-42f9-9992-32cc62e3810e-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-56654dcf7-j9x6z\" (UID: \"9a0d3041-24b6-42f9-9992-32cc62e3810e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-j9x6z" Jul 09 22:22:39.741641 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:22:39.741621 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9a0d3041-24b6-42f9-9992-32cc62e3810e-istio-envoy\") pod \"maas-default-gateway-openshift-default-56654dcf7-j9x6z\" (UID: \"9a0d3041-24b6-42f9-9992-32cc62e3810e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-j9x6z" Jul 09 22:22:39.741845 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:22:39.741827 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9a0d3041-24b6-42f9-9992-32cc62e3810e-istio-podinfo\") pod \"maas-default-gateway-openshift-default-56654dcf7-j9x6z\" (UID: \"9a0d3041-24b6-42f9-9992-32cc62e3810e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-j9x6z" Jul 09 22:22:39.747895 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:22:39.747865 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9a0d3041-24b6-42f9-9992-32cc62e3810e-istio-token\") pod \"maas-default-gateway-openshift-default-56654dcf7-j9x6z\" (UID: \"9a0d3041-24b6-42f9-9992-32cc62e3810e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-j9x6z" Jul 09 22:22:39.748023 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:22:39.748007 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x7cf2\" (UniqueName: \"kubernetes.io/projected/9a0d3041-24b6-42f9-9992-32cc62e3810e-kube-api-access-x7cf2\") pod \"maas-default-gateway-openshift-default-56654dcf7-j9x6z\" (UID: \"9a0d3041-24b6-42f9-9992-32cc62e3810e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-j9x6z" Jul 09 22:22:39.842254 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:22:39.842172 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-j9x6z" Jul 09 22:22:39.980948 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:22:39.980905 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-j9x6z"] Jul 09 22:22:39.983652 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:22:39.983619 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9a0d3041_24b6_42f9_9992_32cc62e3810e.slice/crio-130ff23801f0467759497f8c8e08f954d4273092d5aa513cbd26d65d1cc24bbe WatchSource:0}: Error finding container 130ff23801f0467759497f8c8e08f954d4273092d5aa513cbd26d65d1cc24bbe: Status 404 returned error can't find the container with id 130ff23801f0467759497f8c8e08f954d4273092d5aa513cbd26d65d1cc24bbe Jul 09 22:22:39.985638 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:22:39.985608 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 09 22:22:39.985735 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:22:39.985667 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 09 22:22:39.985735 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:22:39.985696 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 09 22:22:40.422602 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:22:40.422566 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-j9x6z" event={"ID":"9a0d3041-24b6-42f9-9992-32cc62e3810e","Type":"ContainerStarted","Data":"b5e285fc7ffe3721af8221349b7f3bec8bdeb82f911b284e030d3f88bd298689"} Jul 09 22:22:40.422789 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:22:40.422618 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-j9x6z" event={"ID":"9a0d3041-24b6-42f9-9992-32cc62e3810e","Type":"ContainerStarted","Data":"130ff23801f0467759497f8c8e08f954d4273092d5aa513cbd26d65d1cc24bbe"} Jul 09 22:22:40.444013 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:22:40.443963 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-j9x6z" podStartSLOduration=1.443946134 podStartE2EDuration="1.443946134s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 22:22:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 22:22:40.442720531 +0000 UTC m=+752.354173252" watchObservedRunningTime="2026-07-09 22:22:40.443946134 +0000 UTC m=+752.355398827" Jul 09 22:22:40.842574 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:22:40.842522 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-j9x6z" Jul 09 22:22:41.846868 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:22:41.846838 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-j9x6z" Jul 09 22:22:42.106680 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:22:42.106589 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-xdfrr"] Jul 09 22:22:42.109052 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:22:42.109030 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-54d6d988d5-xdfrr" Jul 09 22:22:42.111690 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:22:42.111670 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\"" Jul 09 22:22:42.111848 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:22:42.111834 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-fkpk4\"" Jul 09 22:22:42.124961 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:22:42.124906 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-xdfrr"] Jul 09 22:22:42.161502 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:22:42.161464 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/35f67383-3d5f-44fa-99d0-57f2c2a927bb-config-file\") pod \"limitador-limitador-54d6d988d5-xdfrr\" (UID: \"35f67383-3d5f-44fa-99d0-57f2c2a927bb\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-xdfrr" Jul 09 22:22:42.161502 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:22:42.161497 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n925s\" (UniqueName: \"kubernetes.io/projected/35f67383-3d5f-44fa-99d0-57f2c2a927bb-kube-api-access-n925s\") pod \"limitador-limitador-54d6d988d5-xdfrr\" (UID: \"35f67383-3d5f-44fa-99d0-57f2c2a927bb\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-xdfrr" Jul 09 22:22:42.166362 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:22:42.166329 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-xdfrr"] Jul 09 22:22:42.262807 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:22:42.262770 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/35f67383-3d5f-44fa-99d0-57f2c2a927bb-config-file\") pod \"limitador-limitador-54d6d988d5-xdfrr\" (UID: \"35f67383-3d5f-44fa-99d0-57f2c2a927bb\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-xdfrr" Jul 09 22:22:42.262807 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:22:42.262806 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n925s\" (UniqueName: \"kubernetes.io/projected/35f67383-3d5f-44fa-99d0-57f2c2a927bb-kube-api-access-n925s\") pod \"limitador-limitador-54d6d988d5-xdfrr\" (UID: \"35f67383-3d5f-44fa-99d0-57f2c2a927bb\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-xdfrr" Jul 09 22:22:42.263500 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:22:42.263479 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/35f67383-3d5f-44fa-99d0-57f2c2a927bb-config-file\") pod \"limitador-limitador-54d6d988d5-xdfrr\" (UID: \"35f67383-3d5f-44fa-99d0-57f2c2a927bb\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-xdfrr" Jul 09 22:22:42.271788 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:22:42.271753 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n925s\" (UniqueName: \"kubernetes.io/projected/35f67383-3d5f-44fa-99d0-57f2c2a927bb-kube-api-access-n925s\") pod \"limitador-limitador-54d6d988d5-xdfrr\" (UID: \"35f67383-3d5f-44fa-99d0-57f2c2a927bb\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-xdfrr" Jul 09 22:22:42.419963 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:22:42.419852 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-54d6d988d5-xdfrr" Jul 09 22:22:42.433595 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:22:42.433570 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-j9x6z" Jul 09 22:22:42.434649 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:22:42.434628 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-j9x6z" Jul 09 22:22:42.570077 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:22:42.570053 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-xdfrr"] Jul 09 22:22:42.571989 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:22:42.571960 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod35f67383_3d5f_44fa_99d0_57f2c2a927bb.slice/crio-5425ff543e62468f190ff210f46f090f6d5fb744bf0e58a736308e7071446cdd WatchSource:0}: Error finding container 5425ff543e62468f190ff210f46f090f6d5fb744bf0e58a736308e7071446cdd: Status 404 returned error can't find the container with id 5425ff543e62468f190ff210f46f090f6d5fb744bf0e58a736308e7071446cdd Jul 09 22:22:43.445094 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:22:43.445015 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-54d6d988d5-xdfrr" event={"ID":"35f67383-3d5f-44fa-99d0-57f2c2a927bb","Type":"ContainerStarted","Data":"5425ff543e62468f190ff210f46f090f6d5fb744bf0e58a736308e7071446cdd"} Jul 09 22:22:45.457665 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:22:45.457631 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-54d6d988d5-xdfrr" event={"ID":"35f67383-3d5f-44fa-99d0-57f2c2a927bb","Type":"ContainerStarted","Data":"062d55bdf6cdc07390457cb4b3a9244fbcfef79d689fae3f3f864f2103ba7dab"} Jul 09 22:22:45.458106 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:22:45.457748 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-54d6d988d5-xdfrr" Jul 09 22:22:45.477371 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:22:45.477320 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-54d6d988d5-xdfrr" podStartSLOduration=0.666821951 podStartE2EDuration="3.477300865s" totalImagesPullingTime="2.810478914s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 22:22:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 22:22:45.47503437 +0000 UTC m=+757.386487092" watchObservedRunningTime="2026-07-09 22:22:45.477300865 +0000 UTC m=+757.388753568" Jul 09 22:22:56.463425 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:22:56.463385 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-54d6d988d5-xdfrr" Jul 09 22:23:57.020258 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:23:57.020218 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-xdfrr"] Jul 09 22:24:04.829128 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:24:04.829094 2579 generic.go:356] "Generic (PLEG): container finished" podID="9a0d3041-24b6-42f9-9992-32cc62e3810e" containerID="b5e285fc7ffe3721af8221349b7f3bec8bdeb82f911b284e030d3f88bd298689" exitCode=0 Jul 09 22:24:04.829538 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:24:04.829138 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-j9x6z" event={"ID":"9a0d3041-24b6-42f9-9992-32cc62e3810e","Type":"ContainerDied","Data":"b5e285fc7ffe3721af8221349b7f3bec8bdeb82f911b284e030d3f88bd298689"} Jul 09 22:24:04.829636 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:24:04.829622 2579 scope.go:117] "RemoveContainer" containerID="b5e285fc7ffe3721af8221349b7f3bec8bdeb82f911b284e030d3f88bd298689" Jul 09 22:24:04.830753 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:24:04.830725 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 09 22:24:04.830827 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:24:04.830811 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 09 22:24:04.830874 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:24:04.830861 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 09 22:24:05.835323 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:24:05.835290 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-j9x6z" event={"ID":"9a0d3041-24b6-42f9-9992-32cc62e3810e","Type":"ContainerStarted","Data":"90d7702662adbb709ee5fe99681560bb905e05806dafe5e3eba59112657eb655"} Jul 09 22:24:05.843192 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:24:05.843160 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-j9x6z" Jul 09 22:24:05.843413 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:24:05.843389 2579 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-56654dcf7-j9x6z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.31:15021/healthz/ready\": dial tcp 10.132.0.31:15021: connect: connection refused" start-of-body= Jul 09 22:24:05.843488 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:24:05.843448 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-j9x6z" podUID="9a0d3041-24b6-42f9-9992-32cc62e3810e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.31:15021/healthz/ready\": dial tcp 10.132.0.31:15021: connect: connection refused" Jul 09 22:24:06.847065 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:24:06.847024 2579 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-56654dcf7-j9x6z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.31:15021/healthz/ready\": dial tcp 10.132.0.31:15021: connect: connection refused" start-of-body= Jul 09 22:24:06.847635 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:24:06.847601 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-j9x6z" podUID="9a0d3041-24b6-42f9-9992-32cc62e3810e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.31:15021/healthz/ready\": dial tcp 10.132.0.31:15021: connect: connection refused" Jul 09 22:24:07.843149 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:24:07.843109 2579 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-56654dcf7-j9x6z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.31:15021/healthz/ready\": dial tcp 10.132.0.31:15021: connect: connection refused" start-of-body= Jul 09 22:24:07.843361 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:24:07.843194 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-j9x6z" podUID="9a0d3041-24b6-42f9-9992-32cc62e3810e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.31:15021/healthz/ready\": dial tcp 10.132.0.31:15021: connect: connection refused" Jul 09 22:24:08.843351 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:24:08.843300 2579 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-56654dcf7-j9x6z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.31:15021/healthz/ready\": dial tcp 10.132.0.31:15021: connect: connection refused" start-of-body= Jul 09 22:24:08.843890 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:24:08.843386 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-j9x6z" podUID="9a0d3041-24b6-42f9-9992-32cc62e3810e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.31:15021/healthz/ready\": dial tcp 10.132.0.31:15021: connect: connection refused" Jul 09 22:24:09.847510 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:24:09.847479 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-j9x6z" Jul 09 22:24:09.847973 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:24:09.847741 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-j9x6z" Jul 09 22:24:09.848423 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:24:09.848404 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-j9x6z" Jul 09 22:24:21.704051 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:24:21.704014 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-xdfrr"] Jul 09 22:24:31.806710 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:24:31.806674 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-xdfrr"] Jul 09 22:24:48.316211 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:24:48.316178 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-xdfrr"] Jul 09 22:24:52.719366 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:24:52.719333 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-xdfrr"] Jul 09 22:25:15.112134 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:25:15.112097 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-xdfrr"] Jul 09 22:25:26.221841 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:25:26.221804 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-xdfrr"] Jul 09 22:26:55.718692 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:26:55.718659 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-xdfrr"] Jul 09 22:27:05.311377 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:27:05.311338 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-xdfrr"] Jul 09 22:27:15.917038 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:27:15.917006 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-xdfrr"] Jul 09 22:27:25.219165 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:27:25.219128 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-xdfrr"] Jul 09 22:27:34.933463 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:27:34.933426 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-xdfrr"] Jul 09 22:28:01.905188 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:28:01.905103 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-xdfrr"] Jul 09 22:28:13.914565 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:28:13.914507 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-xdfrr"] Jul 09 22:28:52.717024 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:28:52.716989 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-xdfrr"] Jul 09 22:29:01.811063 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:29:01.811018 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-xdfrr"] Jul 09 22:29:10.712361 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:29:10.712322 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-xdfrr"] Jul 09 22:29:19.210038 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:29:19.210001 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-xdfrr"] Jul 09 22:29:30.413054 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:29:30.413019 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-xdfrr"] Jul 09 22:29:39.520019 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:29:39.519982 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-xdfrr"] Jul 09 22:29:51.717193 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:29:51.717155 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-xdfrr"] Jul 09 22:30:00.169549 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:30:00.169514 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29727270-zzjnb"] Jul 09 22:30:00.173075 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:30:00.173056 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29727270-zzjnb" Jul 09 22:30:00.177382 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:30:00.177360 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fdzhc\"" Jul 09 22:30:00.182474 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:30:00.182454 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29727270-zzjnb" Jul 09 22:30:00.219614 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:30:00.219576 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29727270-zzjnb"] Jul 09 22:30:00.321523 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:30:00.321498 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29727270-zzjnb"] Jul 09 22:30:00.323877 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:30:00.323845 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0b468c92_33c1_40d7_81ad_d3fdeed9c632.slice/crio-fdc634f39a0f8bf81d3bf186189796e5a28dabcc1bd2fb8cd1887c5522f99c60 WatchSource:0}: Error finding container fdc634f39a0f8bf81d3bf186189796e5a28dabcc1bd2fb8cd1887c5522f99c60: Status 404 returned error can't find the container with id fdc634f39a0f8bf81d3bf186189796e5a28dabcc1bd2fb8cd1887c5522f99c60 Jul 09 22:30:00.325888 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:30:00.325870 2579 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 09 22:30:00.578875 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:30:00.578836 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29727270-zzjnb" event={"ID":"0b468c92-33c1-40d7-81ad-d3fdeed9c632","Type":"ContainerStarted","Data":"fdc634f39a0f8bf81d3bf186189796e5a28dabcc1bd2fb8cd1887c5522f99c60"} Jul 09 22:30:02.209762 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:30:02.209721 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-xdfrr"] Jul 09 22:30:03.596679 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:30:03.596591 2579 generic.go:356] "Generic (PLEG): container finished" podID="0b468c92-33c1-40d7-81ad-d3fdeed9c632" containerID="3172389ce8dbfba6dfc52f57931c0d058431d140d7f8d2b8f86b6132adf87248" exitCode=0 Jul 09 22:30:03.597053 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:30:03.596672 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29727270-zzjnb" event={"ID":"0b468c92-33c1-40d7-81ad-d3fdeed9c632","Type":"ContainerDied","Data":"3172389ce8dbfba6dfc52f57931c0d058431d140d7f8d2b8f86b6132adf87248"} Jul 09 22:30:04.742579 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:30:04.742551 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29727270-zzjnb" Jul 09 22:30:05.606906 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:30:05.606874 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29727270-zzjnb" Jul 09 22:30:05.606906 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:30:05.606891 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29727270-zzjnb" event={"ID":"0b468c92-33c1-40d7-81ad-d3fdeed9c632","Type":"ContainerDied","Data":"fdc634f39a0f8bf81d3bf186189796e5a28dabcc1bd2fb8cd1887c5522f99c60"} Jul 09 22:30:05.607143 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:30:05.606949 2579 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fdc634f39a0f8bf81d3bf186189796e5a28dabcc1bd2fb8cd1887c5522f99c60" Jul 09 22:30:23.319475 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:30:23.319435 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-xdfrr"] Jul 09 22:30:34.914120 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:30:34.914085 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-xdfrr"] Jul 09 22:30:58.554979 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:30:58.554944 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-xdfrr"] Jul 09 22:31:09.010304 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:31:09.010220 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-xdfrr"] Jul 09 22:31:33.914583 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:31:33.914550 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-xdfrr"] Jul 09 22:31:43.318577 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:31:43.318536 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-xdfrr"] Jul 09 22:31:54.714953 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:31:54.714893 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-xdfrr"] Jul 09 22:32:04.008225 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:32:04.008188 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-xdfrr"] Jul 09 22:33:51.621896 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:33:51.621851 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-xdfrr"] Jul 09 22:34:03.515114 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:34:03.515079 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-xdfrr"] Jul 09 22:34:14.220241 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:34:14.220207 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-xdfrr"] Jul 09 22:34:24.715180 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:34:24.715145 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-xdfrr"] Jul 09 22:34:36.110338 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:34:36.110298 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-xdfrr"] Jul 09 22:34:46.920749 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:34:46.920710 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-xdfrr"] Jul 09 22:34:55.920441 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:34:55.920398 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-xdfrr"] Jul 09 22:35:06.411564 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:35:06.411531 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-xdfrr"] Jul 09 22:35:15.711880 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:35:15.711836 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-xdfrr"] Jul 09 22:35:23.621587 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:35:23.621553 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-xdfrr"] Jul 09 22:35:33.014357 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:35:33.014266 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-xdfrr"] Jul 09 22:35:42.816077 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:35:42.816034 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-xdfrr"] Jul 09 22:36:03.609582 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:36:03.609541 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-xdfrr"] Jul 09 22:36:10.927854 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:36:10.927809 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-xdfrr"] Jul 09 22:36:20.520037 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:36:20.520005 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-xdfrr"] Jul 09 22:36:31.518438 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:36:31.518407 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-xdfrr"] Jul 09 22:36:40.921491 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:36:40.921454 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-xdfrr"] Jul 09 22:36:50.023106 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:36:50.023064 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-xdfrr"] Jul 09 22:37:04.014132 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:37:04.014044 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-xdfrr"] Jul 09 22:37:12.808101 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:37:12.808062 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-xdfrr"] Jul 09 22:37:21.819873 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:37:21.819835 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-xdfrr"] Jul 09 22:37:32.321693 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:37:32.321652 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-xdfrr"] Jul 09 22:37:42.115825 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:37:42.115784 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-xdfrr"] Jul 09 22:37:55.116980 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:37:55.116835 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-xdfrr"] Jul 09 22:38:04.316938 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:38:04.316884 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-xdfrr"] Jul 09 22:38:13.725377 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:38:13.725335 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-xdfrr"] Jul 09 22:38:23.866328 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:38:23.866290 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-xdfrr"] Jul 09 22:38:36.718384 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:38:36.718306 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-xdfrr"] Jul 09 22:38:44.416946 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:38:44.416881 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-xdfrr"] Jul 09 22:38:56.220232 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:38:56.220196 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-xdfrr"] Jul 09 22:39:09.112360 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:39:09.112327 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-xdfrr"] Jul 09 22:39:21.817119 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:39:21.817085 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-xdfrr"] Jul 09 22:39:35.360737 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:39:35.360704 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-xdfrr"] Jul 09 22:39:47.124735 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:39:47.124699 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-xdfrr"] Jul 09 22:39:52.644684 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:39:52.644648 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-489a4fc6-openshift-default-7d96c6c8f7-v5jwn"] Jul 09 22:39:52.645142 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:39:52.645124 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0b468c92-33c1-40d7-81ad-d3fdeed9c632" containerName="cleanup" Jul 09 22:39:52.645142 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:39:52.645141 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="0b468c92-33c1-40d7-81ad-d3fdeed9c632" containerName="cleanup" Jul 09 22:39:52.645256 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:39:52.645215 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="0b468c92-33c1-40d7-81ad-d3fdeed9c632" containerName="cleanup" Jul 09 22:39:52.648359 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:39:52.648344 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-489a4fc6-openshift-default-7d96c6c8f7-v5jwn" Jul 09 22:39:52.652263 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:39:52.652240 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-489a4fc6-openshift-default-dockercfg-8m8lw\"" Jul 09 22:39:52.683638 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:39:52.683608 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-489a4fc6-openshift-default-7d96c6c8f7-v5jwn"] Jul 09 22:39:52.786301 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:39:52.786265 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6b6b91cd-f2b7-4c38-8386-6b421efc6037-workload-certs\") pod \"e2e-ait-489a4fc6-openshift-default-7d96c6c8f7-v5jwn\" (UID: \"6b6b91cd-f2b7-4c38-8386-6b421efc6037\") " pod="openshift-ingress/e2e-ait-489a4fc6-openshift-default-7d96c6c8f7-v5jwn" Jul 09 22:39:52.786301 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:39:52.786312 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6b6b91cd-f2b7-4c38-8386-6b421efc6037-istio-data\") pod \"e2e-ait-489a4fc6-openshift-default-7d96c6c8f7-v5jwn\" (UID: \"6b6b91cd-f2b7-4c38-8386-6b421efc6037\") " pod="openshift-ingress/e2e-ait-489a4fc6-openshift-default-7d96c6c8f7-v5jwn" Jul 09 22:39:52.786516 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:39:52.786340 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6b6b91cd-f2b7-4c38-8386-6b421efc6037-istio-envoy\") pod \"e2e-ait-489a4fc6-openshift-default-7d96c6c8f7-v5jwn\" (UID: \"6b6b91cd-f2b7-4c38-8386-6b421efc6037\") " pod="openshift-ingress/e2e-ait-489a4fc6-openshift-default-7d96c6c8f7-v5jwn" Jul 09 22:39:52.786516 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:39:52.786382 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6b6b91cd-f2b7-4c38-8386-6b421efc6037-istio-token\") pod \"e2e-ait-489a4fc6-openshift-default-7d96c6c8f7-v5jwn\" (UID: \"6b6b91cd-f2b7-4c38-8386-6b421efc6037\") " pod="openshift-ingress/e2e-ait-489a4fc6-openshift-default-7d96c6c8f7-v5jwn" Jul 09 22:39:52.786516 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:39:52.786420 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b6b91cd-f2b7-4c38-8386-6b421efc6037-credential-socket\") pod \"e2e-ait-489a4fc6-openshift-default-7d96c6c8f7-v5jwn\" (UID: \"6b6b91cd-f2b7-4c38-8386-6b421efc6037\") " pod="openshift-ingress/e2e-ait-489a4fc6-openshift-default-7d96c6c8f7-v5jwn" Jul 09 22:39:52.786516 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:39:52.786440 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tdcdd\" (UniqueName: \"kubernetes.io/projected/6b6b91cd-f2b7-4c38-8386-6b421efc6037-kube-api-access-tdcdd\") pod \"e2e-ait-489a4fc6-openshift-default-7d96c6c8f7-v5jwn\" (UID: \"6b6b91cd-f2b7-4c38-8386-6b421efc6037\") " pod="openshift-ingress/e2e-ait-489a4fc6-openshift-default-7d96c6c8f7-v5jwn" Jul 09 22:39:52.786516 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:39:52.786509 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b6b91cd-f2b7-4c38-8386-6b421efc6037-workload-socket\") pod \"e2e-ait-489a4fc6-openshift-default-7d96c6c8f7-v5jwn\" (UID: \"6b6b91cd-f2b7-4c38-8386-6b421efc6037\") " pod="openshift-ingress/e2e-ait-489a4fc6-openshift-default-7d96c6c8f7-v5jwn" Jul 09 22:39:52.786669 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:39:52.786528 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6b6b91cd-f2b7-4c38-8386-6b421efc6037-istio-podinfo\") pod \"e2e-ait-489a4fc6-openshift-default-7d96c6c8f7-v5jwn\" (UID: \"6b6b91cd-f2b7-4c38-8386-6b421efc6037\") " pod="openshift-ingress/e2e-ait-489a4fc6-openshift-default-7d96c6c8f7-v5jwn" Jul 09 22:39:52.786669 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:39:52.786555 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6b6b91cd-f2b7-4c38-8386-6b421efc6037-istiod-ca-cert\") pod \"e2e-ait-489a4fc6-openshift-default-7d96c6c8f7-v5jwn\" (UID: \"6b6b91cd-f2b7-4c38-8386-6b421efc6037\") " pod="openshift-ingress/e2e-ait-489a4fc6-openshift-default-7d96c6c8f7-v5jwn" Jul 09 22:39:52.887182 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:39:52.887146 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b6b91cd-f2b7-4c38-8386-6b421efc6037-workload-socket\") pod \"e2e-ait-489a4fc6-openshift-default-7d96c6c8f7-v5jwn\" (UID: \"6b6b91cd-f2b7-4c38-8386-6b421efc6037\") " pod="openshift-ingress/e2e-ait-489a4fc6-openshift-default-7d96c6c8f7-v5jwn" Jul 09 22:39:52.887182 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:39:52.887184 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6b6b91cd-f2b7-4c38-8386-6b421efc6037-istio-podinfo\") pod \"e2e-ait-489a4fc6-openshift-default-7d96c6c8f7-v5jwn\" (UID: \"6b6b91cd-f2b7-4c38-8386-6b421efc6037\") " pod="openshift-ingress/e2e-ait-489a4fc6-openshift-default-7d96c6c8f7-v5jwn" Jul 09 22:39:52.887428 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:39:52.887216 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6b6b91cd-f2b7-4c38-8386-6b421efc6037-istiod-ca-cert\") pod \"e2e-ait-489a4fc6-openshift-default-7d96c6c8f7-v5jwn\" (UID: \"6b6b91cd-f2b7-4c38-8386-6b421efc6037\") " pod="openshift-ingress/e2e-ait-489a4fc6-openshift-default-7d96c6c8f7-v5jwn" Jul 09 22:39:52.887428 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:39:52.887247 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6b6b91cd-f2b7-4c38-8386-6b421efc6037-workload-certs\") pod \"e2e-ait-489a4fc6-openshift-default-7d96c6c8f7-v5jwn\" (UID: \"6b6b91cd-f2b7-4c38-8386-6b421efc6037\") " pod="openshift-ingress/e2e-ait-489a4fc6-openshift-default-7d96c6c8f7-v5jwn" Jul 09 22:39:52.887428 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:39:52.887282 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6b6b91cd-f2b7-4c38-8386-6b421efc6037-istio-data\") pod \"e2e-ait-489a4fc6-openshift-default-7d96c6c8f7-v5jwn\" (UID: \"6b6b91cd-f2b7-4c38-8386-6b421efc6037\") " pod="openshift-ingress/e2e-ait-489a4fc6-openshift-default-7d96c6c8f7-v5jwn" Jul 09 22:39:52.887428 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:39:52.887318 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6b6b91cd-f2b7-4c38-8386-6b421efc6037-istio-envoy\") pod \"e2e-ait-489a4fc6-openshift-default-7d96c6c8f7-v5jwn\" (UID: \"6b6b91cd-f2b7-4c38-8386-6b421efc6037\") " pod="openshift-ingress/e2e-ait-489a4fc6-openshift-default-7d96c6c8f7-v5jwn" Jul 09 22:39:52.887428 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:39:52.887350 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6b6b91cd-f2b7-4c38-8386-6b421efc6037-istio-token\") pod \"e2e-ait-489a4fc6-openshift-default-7d96c6c8f7-v5jwn\" (UID: \"6b6b91cd-f2b7-4c38-8386-6b421efc6037\") " pod="openshift-ingress/e2e-ait-489a4fc6-openshift-default-7d96c6c8f7-v5jwn" Jul 09 22:39:52.887696 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:39:52.887462 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b6b91cd-f2b7-4c38-8386-6b421efc6037-credential-socket\") pod \"e2e-ait-489a4fc6-openshift-default-7d96c6c8f7-v5jwn\" (UID: \"6b6b91cd-f2b7-4c38-8386-6b421efc6037\") " pod="openshift-ingress/e2e-ait-489a4fc6-openshift-default-7d96c6c8f7-v5jwn" Jul 09 22:39:52.887696 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:39:52.887494 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tdcdd\" (UniqueName: \"kubernetes.io/projected/6b6b91cd-f2b7-4c38-8386-6b421efc6037-kube-api-access-tdcdd\") pod \"e2e-ait-489a4fc6-openshift-default-7d96c6c8f7-v5jwn\" (UID: \"6b6b91cd-f2b7-4c38-8386-6b421efc6037\") " pod="openshift-ingress/e2e-ait-489a4fc6-openshift-default-7d96c6c8f7-v5jwn" Jul 09 22:39:52.887835 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:39:52.887690 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6b6b91cd-f2b7-4c38-8386-6b421efc6037-workload-certs\") pod \"e2e-ait-489a4fc6-openshift-default-7d96c6c8f7-v5jwn\" (UID: \"6b6b91cd-f2b7-4c38-8386-6b421efc6037\") " pod="openshift-ingress/e2e-ait-489a4fc6-openshift-default-7d96c6c8f7-v5jwn" Jul 09 22:39:52.887835 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:39:52.887716 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b6b91cd-f2b7-4c38-8386-6b421efc6037-workload-socket\") pod \"e2e-ait-489a4fc6-openshift-default-7d96c6c8f7-v5jwn\" (UID: \"6b6b91cd-f2b7-4c38-8386-6b421efc6037\") " pod="openshift-ingress/e2e-ait-489a4fc6-openshift-default-7d96c6c8f7-v5jwn" Jul 09 22:39:52.887835 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:39:52.887821 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6b6b91cd-f2b7-4c38-8386-6b421efc6037-istio-data\") pod \"e2e-ait-489a4fc6-openshift-default-7d96c6c8f7-v5jwn\" (UID: \"6b6b91cd-f2b7-4c38-8386-6b421efc6037\") " pod="openshift-ingress/e2e-ait-489a4fc6-openshift-default-7d96c6c8f7-v5jwn" Jul 09 22:39:52.887993 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:39:52.887954 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b6b91cd-f2b7-4c38-8386-6b421efc6037-credential-socket\") pod \"e2e-ait-489a4fc6-openshift-default-7d96c6c8f7-v5jwn\" (UID: \"6b6b91cd-f2b7-4c38-8386-6b421efc6037\") " pod="openshift-ingress/e2e-ait-489a4fc6-openshift-default-7d96c6c8f7-v5jwn" Jul 09 22:39:52.888111 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:39:52.888092 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6b6b91cd-f2b7-4c38-8386-6b421efc6037-istiod-ca-cert\") pod \"e2e-ait-489a4fc6-openshift-default-7d96c6c8f7-v5jwn\" (UID: \"6b6b91cd-f2b7-4c38-8386-6b421efc6037\") " pod="openshift-ingress/e2e-ait-489a4fc6-openshift-default-7d96c6c8f7-v5jwn" Jul 09 22:39:52.889599 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:39:52.889582 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6b6b91cd-f2b7-4c38-8386-6b421efc6037-istio-envoy\") pod \"e2e-ait-489a4fc6-openshift-default-7d96c6c8f7-v5jwn\" (UID: \"6b6b91cd-f2b7-4c38-8386-6b421efc6037\") " pod="openshift-ingress/e2e-ait-489a4fc6-openshift-default-7d96c6c8f7-v5jwn" Jul 09 22:39:52.890087 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:39:52.890047 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6b6b91cd-f2b7-4c38-8386-6b421efc6037-istio-podinfo\") pod \"e2e-ait-489a4fc6-openshift-default-7d96c6c8f7-v5jwn\" (UID: \"6b6b91cd-f2b7-4c38-8386-6b421efc6037\") " pod="openshift-ingress/e2e-ait-489a4fc6-openshift-default-7d96c6c8f7-v5jwn" Jul 09 22:39:52.904533 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:39:52.904505 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tdcdd\" (UniqueName: \"kubernetes.io/projected/6b6b91cd-f2b7-4c38-8386-6b421efc6037-kube-api-access-tdcdd\") pod \"e2e-ait-489a4fc6-openshift-default-7d96c6c8f7-v5jwn\" (UID: \"6b6b91cd-f2b7-4c38-8386-6b421efc6037\") " pod="openshift-ingress/e2e-ait-489a4fc6-openshift-default-7d96c6c8f7-v5jwn" Jul 09 22:39:52.904655 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:39:52.904638 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6b6b91cd-f2b7-4c38-8386-6b421efc6037-istio-token\") pod \"e2e-ait-489a4fc6-openshift-default-7d96c6c8f7-v5jwn\" (UID: \"6b6b91cd-f2b7-4c38-8386-6b421efc6037\") " pod="openshift-ingress/e2e-ait-489a4fc6-openshift-default-7d96c6c8f7-v5jwn" Jul 09 22:39:52.958749 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:39:52.958714 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-489a4fc6-openshift-default-7d96c6c8f7-v5jwn" Jul 09 22:39:53.102092 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:39:53.102068 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-489a4fc6-openshift-default-7d96c6c8f7-v5jwn"] Jul 09 22:39:53.103874 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:39:53.103848 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6b6b91cd_f2b7_4c38_8386_6b421efc6037.slice/crio-3a8f64b735d351c2836266a6091c5a7739e094f02fe78ba43dc0bbb8306ca39d WatchSource:0}: Error finding container 3a8f64b735d351c2836266a6091c5a7739e094f02fe78ba43dc0bbb8306ca39d: Status 404 returned error can't find the container with id 3a8f64b735d351c2836266a6091c5a7739e094f02fe78ba43dc0bbb8306ca39d Jul 09 22:39:53.105958 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:39:53.105906 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 09 22:39:53.106036 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:39:53.105993 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 09 22:39:53.106036 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:39:53.106020 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 09 22:39:53.484640 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:39:53.484552 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-489a4fc6-openshift-default-7d96c6c8f7-v5jwn" event={"ID":"6b6b91cd-f2b7-4c38-8386-6b421efc6037","Type":"ContainerStarted","Data":"8df796d1cbd763926ad8d75a0e4c3b510d21c71512281d898c8de974466daadc"} Jul 09 22:39:53.484640 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:39:53.484633 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-489a4fc6-openshift-default-7d96c6c8f7-v5jwn" event={"ID":"6b6b91cd-f2b7-4c38-8386-6b421efc6037","Type":"ContainerStarted","Data":"3a8f64b735d351c2836266a6091c5a7739e094f02fe78ba43dc0bbb8306ca39d"} Jul 09 22:39:53.509986 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:39:53.509912 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-489a4fc6-openshift-default-7d96c6c8f7-v5jwn" podStartSLOduration=1.509898308 podStartE2EDuration="1.509898308s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 22:39:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 22:39:53.509149375 +0000 UTC m=+1785.420602105" watchObservedRunningTime="2026-07-09 22:39:53.509898308 +0000 UTC m=+1785.421350999" Jul 09 22:39:53.959524 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:39:53.959491 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-489a4fc6-openshift-default-7d96c6c8f7-v5jwn" Jul 09 22:39:54.964569 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:39:54.964535 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-489a4fc6-openshift-default-7d96c6c8f7-v5jwn" Jul 09 22:39:55.493724 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:39:55.493691 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-489a4fc6-openshift-default-7d96c6c8f7-v5jwn" Jul 09 22:39:55.494787 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:39:55.494765 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-489a4fc6-openshift-default-7d96c6c8f7-v5jwn" Jul 09 22:39:57.111354 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:39:57.111323 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-xdfrr"] Jul 09 22:39:58.852798 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:39:58.852741 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-489a4fc6-x2wwq"] Jul 09 22:39:58.858790 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:39:58.858763 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-489a4fc6-x2wwq" Jul 09 22:39:58.866607 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:39:58.866584 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 09 22:39:58.867125 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:39:58.867106 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fdzhc\"" Jul 09 22:39:58.867557 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:39:58.867482 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-489a4fc6-x2wwq"] Jul 09 22:39:58.960437 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:39:58.960393 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f97b904d-4283-4e71-a41d-27831c412327-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-489a4fc6-x2wwq\" (UID: \"f97b904d-4283-4e71-a41d-27831c412327\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-489a4fc6-x2wwq" Jul 09 22:39:59.062512 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:39:59.062470 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f97b904d-4283-4e71-a41d-27831c412327-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-489a4fc6-x2wwq\" (UID: \"f97b904d-4283-4e71-a41d-27831c412327\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-489a4fc6-x2wwq" Jul 09 22:39:59.063663 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:39:59.063527 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f97b904d-4283-4e71-a41d-27831c412327-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-489a4fc6-x2wwq\" (UID: \"f97b904d-4283-4e71-a41d-27831c412327\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-489a4fc6-x2wwq" Jul 09 22:39:59.176578 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:39:59.176486 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-489a4fc6-x2wwq" Jul 09 22:39:59.332034 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:39:59.332006 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-489a4fc6-x2wwq"] Jul 09 22:39:59.334255 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:39:59.334227 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf97b904d_4283_4e71_a41d_27831c412327.slice/crio-a4966e7af457256a2f028ebf1f771d55c6c22926a79288f915139432f2bbc278 WatchSource:0}: Error finding container a4966e7af457256a2f028ebf1f771d55c6c22926a79288f915139432f2bbc278: Status 404 returned error can't find the container with id a4966e7af457256a2f028ebf1f771d55c6c22926a79288f915139432f2bbc278 Jul 09 22:39:59.520387 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:39:59.520348 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-489a4fc6-x2wwq" event={"ID":"f97b904d-4283-4e71-a41d-27831c412327","Type":"ContainerStarted","Data":"a4966e7af457256a2f028ebf1f771d55c6c22926a79288f915139432f2bbc278"} Jul 09 22:40:00.526772 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:40:00.526695 2579 generic.go:356] "Generic (PLEG): container finished" podID="f97b904d-4283-4e71-a41d-27831c412327" containerID="8e239efc9cee146671b2e8d83d009c3549ff8c4d2d291a29727c75868463f3e3" exitCode=7 Jul 09 22:40:00.527203 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:40:00.526797 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-489a4fc6-x2wwq" event={"ID":"f97b904d-4283-4e71-a41d-27831c412327","Type":"ContainerDied","Data":"8e239efc9cee146671b2e8d83d009c3549ff8c4d2d291a29727c75868463f3e3"} Jul 09 22:40:00.527203 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:40:00.526953 2579 scope.go:117] "RemoveContainer" containerID="8e239efc9cee146671b2e8d83d009c3549ff8c4d2d291a29727c75868463f3e3" Jul 09 22:40:01.533621 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:40:01.533584 2579 generic.go:356] "Generic (PLEG): container finished" podID="f97b904d-4283-4e71-a41d-27831c412327" containerID="3970774e47eaab7241d3eb43031b4271785f2448513234e1708a3c46b3fe9697" exitCode=7 Jul 09 22:40:01.534051 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:40:01.533667 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-489a4fc6-x2wwq" event={"ID":"f97b904d-4283-4e71-a41d-27831c412327","Type":"ContainerDied","Data":"3970774e47eaab7241d3eb43031b4271785f2448513234e1708a3c46b3fe9697"} Jul 09 22:40:01.535052 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:40:01.533742 2579 scope.go:117] "RemoveContainer" containerID="8e239efc9cee146671b2e8d83d009c3549ff8c4d2d291a29727c75868463f3e3" Jul 09 22:40:01.535150 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:40:01.535134 2579 scope.go:117] "RemoveContainer" containerID="3970774e47eaab7241d3eb43031b4271785f2448513234e1708a3c46b3fe9697" Jul 09 22:40:01.535379 ip-10-0-136-251 kubenswrapper[2579]: E0709 22:40:01.535359 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-489a4fc6-x2wwq_odh-ai-gateway-infra(f97b904d-4283-4e71-a41d-27831c412327)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-489a4fc6-x2wwq" podUID="f97b904d-4283-4e71-a41d-27831c412327" Jul 09 22:40:02.540642 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:40:02.540609 2579 scope.go:117] "RemoveContainer" containerID="3970774e47eaab7241d3eb43031b4271785f2448513234e1708a3c46b3fe9697" Jul 09 22:40:02.541114 ip-10-0-136-251 kubenswrapper[2579]: E0709 22:40:02.540869 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-489a4fc6-x2wwq_odh-ai-gateway-infra(f97b904d-4283-4e71-a41d-27831c412327)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-489a4fc6-x2wwq" podUID="f97b904d-4283-4e71-a41d-27831c412327" Jul 09 22:40:05.556703 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:40:05.556607 2579 generic.go:356] "Generic (PLEG): container finished" podID="6b6b91cd-f2b7-4c38-8386-6b421efc6037" containerID="8df796d1cbd763926ad8d75a0e4c3b510d21c71512281d898c8de974466daadc" exitCode=0 Jul 09 22:40:05.556703 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:40:05.556688 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-489a4fc6-openshift-default-7d96c6c8f7-v5jwn" event={"ID":"6b6b91cd-f2b7-4c38-8386-6b421efc6037","Type":"ContainerDied","Data":"8df796d1cbd763926ad8d75a0e4c3b510d21c71512281d898c8de974466daadc"} Jul 09 22:40:05.557307 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:40:05.557241 2579 scope.go:117] "RemoveContainer" containerID="8df796d1cbd763926ad8d75a0e4c3b510d21c71512281d898c8de974466daadc" Jul 09 22:40:05.558313 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:40:05.558283 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 09 22:40:05.558513 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:40:05.558368 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 09 22:40:05.558513 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:40:05.558442 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 09 22:40:06.571628 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:40:06.571595 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-489a4fc6-openshift-default-7d96c6c8f7-v5jwn" event={"ID":"6b6b91cd-f2b7-4c38-8386-6b421efc6037","Type":"ContainerStarted","Data":"978a2e9d77be959232e95080dddffc51563a63716629c3e29515bae6da00f807"} Jul 09 22:40:06.967151 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:40:06.967105 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-489a4fc6-openshift-default-7d96c6c8f7-v5jwn" Jul 09 22:40:06.968476 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:40:06.968445 2579 patch_prober.go:28] interesting pod/e2e-ait-489a4fc6-openshift-default-7d96c6c8f7-v5jwn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.34:15021/healthz/ready\": dial tcp 10.132.0.34:15021: connect: connection refused" start-of-body= Jul 09 22:40:06.968598 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:40:06.968498 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-489a4fc6-openshift-default-7d96c6c8f7-v5jwn" podUID="6b6b91cd-f2b7-4c38-8386-6b421efc6037" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.34:15021/healthz/ready\": dial tcp 10.132.0.34:15021: connect: connection refused" Jul 09 22:40:07.959247 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:40:07.959201 2579 patch_prober.go:28] interesting pod/e2e-ait-489a4fc6-openshift-default-7d96c6c8f7-v5jwn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.34:15021/healthz/ready\": dial tcp 10.132.0.34:15021: connect: connection refused" start-of-body= Jul 09 22:40:07.959777 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:40:07.959273 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-489a4fc6-openshift-default-7d96c6c8f7-v5jwn" podUID="6b6b91cd-f2b7-4c38-8386-6b421efc6037" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.34:15021/healthz/ready\": dial tcp 10.132.0.34:15021: connect: connection refused" Jul 09 22:40:08.959498 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:40:08.959451 2579 patch_prober.go:28] interesting pod/e2e-ait-489a4fc6-openshift-default-7d96c6c8f7-v5jwn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.34:15021/healthz/ready\": dial tcp 10.132.0.34:15021: connect: connection refused" start-of-body= Jul 09 22:40:08.960156 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:40:08.960111 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-489a4fc6-openshift-default-7d96c6c8f7-v5jwn" podUID="6b6b91cd-f2b7-4c38-8386-6b421efc6037" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.34:15021/healthz/ready\": dial tcp 10.132.0.34:15021: connect: connection refused" Jul 09 22:40:09.959992 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:40:09.959948 2579 patch_prober.go:28] interesting pod/e2e-ait-489a4fc6-openshift-default-7d96c6c8f7-v5jwn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.34:15021/healthz/ready\": dial tcp 10.132.0.34:15021: connect: connection refused" start-of-body= Jul 09 22:40:09.960520 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:40:09.960042 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-489a4fc6-openshift-default-7d96c6c8f7-v5jwn" podUID="6b6b91cd-f2b7-4c38-8386-6b421efc6037" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.34:15021/healthz/ready\": dial tcp 10.132.0.34:15021: connect: connection refused" Jul 09 22:40:10.494546 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:40:10.494506 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-489a4fc6-openshift-default-7d96c6c8f7-v5jwn" Jul 09 22:40:10.963544 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:40:10.963469 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-489a4fc6-openshift-default-7d96c6c8f7-v5jwn" Jul 09 22:40:10.964471 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:40:10.964452 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-489a4fc6-openshift-default-7d96c6c8f7-v5jwn" Jul 09 22:40:16.559097 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:40:16.559057 2579 scope.go:117] "RemoveContainer" containerID="3970774e47eaab7241d3eb43031b4271785f2448513234e1708a3c46b3fe9697" Jul 09 22:40:17.628128 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:40:17.628028 2579 generic.go:356] "Generic (PLEG): container finished" podID="f97b904d-4283-4e71-a41d-27831c412327" containerID="52697fca0d3488631f381867c2a8f5dc799058fbae9948a6f9648acf3a01a9c3" exitCode=0 Jul 09 22:40:17.628128 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:40:17.628099 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-489a4fc6-x2wwq" event={"ID":"f97b904d-4283-4e71-a41d-27831c412327","Type":"ContainerDied","Data":"52697fca0d3488631f381867c2a8f5dc799058fbae9948a6f9648acf3a01a9c3"} Jul 09 22:40:17.628604 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:40:17.628165 2579 scope.go:117] "RemoveContainer" containerID="3970774e47eaab7241d3eb43031b4271785f2448513234e1708a3c46b3fe9697" Jul 09 22:40:18.658753 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:40:18.658716 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-489a4fc6-x2wwq"] Jul 09 22:40:18.773370 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:40:18.773346 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-489a4fc6-x2wwq" Jul 09 22:40:18.846331 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:40:18.846288 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f97b904d-4283-4e71-a41d-27831c412327-maas-api-service-ca\") pod \"f97b904d-4283-4e71-a41d-27831c412327\" (UID: \"f97b904d-4283-4e71-a41d-27831c412327\") " Jul 09 22:40:18.846676 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:40:18.846652 2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f97b904d-4283-4e71-a41d-27831c412327-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "f97b904d-4283-4e71-a41d-27831c412327" (UID: "f97b904d-4283-4e71-a41d-27831c412327"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 09 22:40:18.947731 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:40:18.947642 2579 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f97b904d-4283-4e71-a41d-27831c412327-maas-api-service-ca\") on node \"ip-10-0-136-251.ec2.internal\" DevicePath \"\"" Jul 09 22:40:19.640559 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:40:19.640531 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-489a4fc6-x2wwq" Jul 09 22:40:19.640739 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:40:19.640567 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-489a4fc6-x2wwq" event={"ID":"f97b904d-4283-4e71-a41d-27831c412327","Type":"ContainerDied","Data":"a4966e7af457256a2f028ebf1f771d55c6c22926a79288f915139432f2bbc278"} Jul 09 22:40:19.640739 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:40:19.640617 2579 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a4966e7af457256a2f028ebf1f771d55c6c22926a79288f915139432f2bbc278" Jul 09 22:40:19.672448 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:40:19.672415 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-489a4fc6-x2wwq"] Jul 09 22:40:19.678618 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:40:19.678590 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-489a4fc6-x2wwq"] Jul 09 22:40:20.565107 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:40:20.565069 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f97b904d-4283-4e71-a41d-27831c412327" path="/var/lib/kubelet/pods/f97b904d-4283-4e71-a41d-27831c412327/volumes" Jul 09 22:40:58.907611 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:40:58.907574 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-489a4fc6-mt8s9"] Jul 09 22:40:58.908112 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:40:58.907956 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f97b904d-4283-4e71-a41d-27831c412327" containerName="revoke-keys" Jul 09 22:40:58.908112 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:40:58.907968 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="f97b904d-4283-4e71-a41d-27831c412327" containerName="revoke-keys" Jul 09 22:40:58.908112 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:40:58.907988 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f97b904d-4283-4e71-a41d-27831c412327" containerName="revoke-keys" Jul 09 22:40:58.908112 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:40:58.907994 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="f97b904d-4283-4e71-a41d-27831c412327" containerName="revoke-keys" Jul 09 22:40:58.908112 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:40:58.908060 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="f97b904d-4283-4e71-a41d-27831c412327" containerName="revoke-keys" Jul 09 22:40:58.908112 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:40:58.908067 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="f97b904d-4283-4e71-a41d-27831c412327" containerName="revoke-keys" Jul 09 22:40:58.908112 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:40:58.908073 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="f97b904d-4283-4e71-a41d-27831c412327" containerName="revoke-keys" Jul 09 22:40:58.911142 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:40:58.911125 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-489a4fc6-mt8s9" Jul 09 22:40:58.913691 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:40:58.913671 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 09 22:40:58.913795 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:40:58.913734 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fdzhc\"" Jul 09 22:40:58.917553 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:40:58.917524 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-489a4fc6-mt8s9"] Jul 09 22:40:58.969885 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:40:58.969852 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-489a4fc6-openshift-default-7d96c6c8f7-v5jwn"] Jul 09 22:40:58.970105 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:40:58.970079 2579 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-489a4fc6-openshift-default-7d96c6c8f7-v5jwn" podUID="6b6b91cd-f2b7-4c38-8386-6b421efc6037" containerName="istio-proxy" containerID="cri-o://978a2e9d77be959232e95080dddffc51563a63716629c3e29515bae6da00f807" gracePeriod=30 Jul 09 22:40:59.018784 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:40:59.018745 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d5613506-b584-4cfd-954e-eb6a5605350e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-489a4fc6-mt8s9\" (UID: \"d5613506-b584-4cfd-954e-eb6a5605350e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-489a4fc6-mt8s9" Jul 09 22:40:59.119642 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:40:59.119588 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d5613506-b584-4cfd-954e-eb6a5605350e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-489a4fc6-mt8s9\" (UID: \"d5613506-b584-4cfd-954e-eb6a5605350e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-489a4fc6-mt8s9" Jul 09 22:40:59.120376 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:40:59.120344 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d5613506-b584-4cfd-954e-eb6a5605350e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-489a4fc6-mt8s9\" (UID: \"d5613506-b584-4cfd-954e-eb6a5605350e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-489a4fc6-mt8s9" Jul 09 22:40:59.222640 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:40:59.222532 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-489a4fc6-mt8s9" Jul 09 22:40:59.360833 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:40:59.360805 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-489a4fc6-mt8s9"] Jul 09 22:40:59.363262 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:40:59.363232 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd5613506_b584_4cfd_954e_eb6a5605350e.slice/crio-6d9ffe678a0fb3d7e1064e957dbf9d5b4fca033e611a05f28f9be09b9b82354c WatchSource:0}: Error finding container 6d9ffe678a0fb3d7e1064e957dbf9d5b4fca033e611a05f28f9be09b9b82354c: Status 404 returned error can't find the container with id 6d9ffe678a0fb3d7e1064e957dbf9d5b4fca033e611a05f28f9be09b9b82354c Jul 09 22:40:59.843544 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:40:59.843452 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-489a4fc6-mt8s9" event={"ID":"d5613506-b584-4cfd-954e-eb6a5605350e","Type":"ContainerStarted","Data":"599e3e3edb5fa54b96e539e24e12f3a46a1b2d97efba05a362d2781b6521e27e"} Jul 09 22:40:59.843544 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:40:59.843519 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-489a4fc6-mt8s9" event={"ID":"d5613506-b584-4cfd-954e-eb6a5605350e","Type":"ContainerStarted","Data":"6d9ffe678a0fb3d7e1064e957dbf9d5b4fca033e611a05f28f9be09b9b82354c"} Jul 09 22:40:59.860558 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:40:59.860507 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-489a4fc6-mt8s9" podStartSLOduration=1.860491079 podStartE2EDuration="1.860491079s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 22:40:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 22:40:59.859440167 +0000 UTC m=+1851.770892888" watchObservedRunningTime="2026-07-09 22:40:59.860491079 +0000 UTC m=+1851.771943780" Jul 09 22:41:00.849430 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:41:00.849396 2579 generic.go:356] "Generic (PLEG): container finished" podID="d5613506-b584-4cfd-954e-eb6a5605350e" containerID="599e3e3edb5fa54b96e539e24e12f3a46a1b2d97efba05a362d2781b6521e27e" exitCode=0 Jul 09 22:41:00.849857 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:41:00.849486 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-489a4fc6-mt8s9" event={"ID":"d5613506-b584-4cfd-954e-eb6a5605350e","Type":"ContainerDied","Data":"599e3e3edb5fa54b96e539e24e12f3a46a1b2d97efba05a362d2781b6521e27e"} Jul 09 22:41:01.714478 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:41:01.714441 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-xdfrr"] Jul 09 22:41:01.986369 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:41:01.986343 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-489a4fc6-mt8s9" Jul 09 22:41:02.048160 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:41:02.048120 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d5613506-b584-4cfd-954e-eb6a5605350e-maas-api-service-ca\") pod \"d5613506-b584-4cfd-954e-eb6a5605350e\" (UID: \"d5613506-b584-4cfd-954e-eb6a5605350e\") " Jul 09 22:41:02.048539 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:41:02.048512 2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d5613506-b584-4cfd-954e-eb6a5605350e-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "d5613506-b584-4cfd-954e-eb6a5605350e" (UID: "d5613506-b584-4cfd-954e-eb6a5605350e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 09 22:41:02.148873 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:41:02.148828 2579 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d5613506-b584-4cfd-954e-eb6a5605350e-maas-api-service-ca\") on node \"ip-10-0-136-251.ec2.internal\" DevicePath \"\"" Jul 09 22:41:02.860445 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:41:02.860363 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-489a4fc6-mt8s9" Jul 09 22:41:02.860445 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:41:02.860408 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-489a4fc6-mt8s9" event={"ID":"d5613506-b584-4cfd-954e-eb6a5605350e","Type":"ContainerDied","Data":"6d9ffe678a0fb3d7e1064e957dbf9d5b4fca033e611a05f28f9be09b9b82354c"} Jul 09 22:41:02.860622 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:41:02.860456 2579 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6d9ffe678a0fb3d7e1064e957dbf9d5b4fca033e611a05f28f9be09b9b82354c" Jul 09 22:41:04.229292 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:41:04.229227 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-489a4fc6-openshift-default-7d96c6c8f7-v5jwn" Jul 09 22:41:04.268083 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:41:04.268046 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6b6b91cd-f2b7-4c38-8386-6b421efc6037-workload-certs\") pod \"6b6b91cd-f2b7-4c38-8386-6b421efc6037\" (UID: \"6b6b91cd-f2b7-4c38-8386-6b421efc6037\") " Jul 09 22:41:04.268265 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:41:04.268120 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b6b91cd-f2b7-4c38-8386-6b421efc6037-credential-socket\") pod \"6b6b91cd-f2b7-4c38-8386-6b421efc6037\" (UID: \"6b6b91cd-f2b7-4c38-8386-6b421efc6037\") " Jul 09 22:41:04.268265 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:41:04.268148 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6b6b91cd-f2b7-4c38-8386-6b421efc6037-istiod-ca-cert\") pod \"6b6b91cd-f2b7-4c38-8386-6b421efc6037\" (UID: \"6b6b91cd-f2b7-4c38-8386-6b421efc6037\") " Jul 09 22:41:04.268265 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:41:04.268185 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tdcdd\" (UniqueName: \"kubernetes.io/projected/6b6b91cd-f2b7-4c38-8386-6b421efc6037-kube-api-access-tdcdd\") pod \"6b6b91cd-f2b7-4c38-8386-6b421efc6037\" (UID: \"6b6b91cd-f2b7-4c38-8386-6b421efc6037\") " Jul 09 22:41:04.268265 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:41:04.268256 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6b6b91cd-f2b7-4c38-8386-6b421efc6037-istio-data\") pod \"6b6b91cd-f2b7-4c38-8386-6b421efc6037\" (UID: \"6b6b91cd-f2b7-4c38-8386-6b421efc6037\") " Jul 09 22:41:04.268486 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:41:04.268279 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6b6b91cd-f2b7-4c38-8386-6b421efc6037-istio-envoy\") pod \"6b6b91cd-f2b7-4c38-8386-6b421efc6037\" (UID: \"6b6b91cd-f2b7-4c38-8386-6b421efc6037\") " Jul 09 22:41:04.268486 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:41:04.268292 2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6b6b91cd-f2b7-4c38-8386-6b421efc6037-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "6b6b91cd-f2b7-4c38-8386-6b421efc6037" (UID: "6b6b91cd-f2b7-4c38-8386-6b421efc6037"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 22:41:04.268598 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:41:04.268460 2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6b6b91cd-f2b7-4c38-8386-6b421efc6037-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "6b6b91cd-f2b7-4c38-8386-6b421efc6037" (UID: "6b6b91cd-f2b7-4c38-8386-6b421efc6037"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 22:41:04.268598 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:41:04.268515 2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6b6b91cd-f2b7-4c38-8386-6b421efc6037-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "6b6b91cd-f2b7-4c38-8386-6b421efc6037" (UID: "6b6b91cd-f2b7-4c38-8386-6b421efc6037"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 09 22:41:04.268598 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:41:04.268540 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6b6b91cd-f2b7-4c38-8386-6b421efc6037-istio-podinfo\") pod \"6b6b91cd-f2b7-4c38-8386-6b421efc6037\" (UID: \"6b6b91cd-f2b7-4c38-8386-6b421efc6037\") " Jul 09 22:41:04.268598 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:41:04.268590 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6b6b91cd-f2b7-4c38-8386-6b421efc6037-istio-token\") pod \"6b6b91cd-f2b7-4c38-8386-6b421efc6037\" (UID: \"6b6b91cd-f2b7-4c38-8386-6b421efc6037\") " Jul 09 22:41:04.268806 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:41:04.268635 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b6b91cd-f2b7-4c38-8386-6b421efc6037-workload-socket\") pod \"6b6b91cd-f2b7-4c38-8386-6b421efc6037\" (UID: \"6b6b91cd-f2b7-4c38-8386-6b421efc6037\") " Jul 09 22:41:04.269223 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:41:04.268968 2579 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6b6b91cd-f2b7-4c38-8386-6b421efc6037-workload-certs\") on node \"ip-10-0-136-251.ec2.internal\" DevicePath \"\"" Jul 09 22:41:04.269223 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:41:04.268992 2579 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b6b91cd-f2b7-4c38-8386-6b421efc6037-credential-socket\") on node \"ip-10-0-136-251.ec2.internal\" DevicePath \"\"" Jul 09 22:41:04.269223 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:41:04.269006 2579 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6b6b91cd-f2b7-4c38-8386-6b421efc6037-istiod-ca-cert\") on node \"ip-10-0-136-251.ec2.internal\" DevicePath \"\"" Jul 09 22:41:04.269223 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:41:04.269171 2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6b6b91cd-f2b7-4c38-8386-6b421efc6037-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "6b6b91cd-f2b7-4c38-8386-6b421efc6037" (UID: "6b6b91cd-f2b7-4c38-8386-6b421efc6037"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 22:41:04.269223 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:41:04.269182 2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6b6b91cd-f2b7-4c38-8386-6b421efc6037-istio-data" (OuterVolumeSpecName: "istio-data") pod "6b6b91cd-f2b7-4c38-8386-6b421efc6037" (UID: "6b6b91cd-f2b7-4c38-8386-6b421efc6037"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 22:41:04.270762 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:41:04.270714 2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6b6b91cd-f2b7-4c38-8386-6b421efc6037-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "6b6b91cd-f2b7-4c38-8386-6b421efc6037" (UID: "6b6b91cd-f2b7-4c38-8386-6b421efc6037"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 09 22:41:04.270896 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:41:04.270826 2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6b6b91cd-f2b7-4c38-8386-6b421efc6037-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "6b6b91cd-f2b7-4c38-8386-6b421efc6037" (UID: "6b6b91cd-f2b7-4c38-8386-6b421efc6037"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 22:41:04.271172 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:41:04.271146 2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6b6b91cd-f2b7-4c38-8386-6b421efc6037-istio-token" (OuterVolumeSpecName: "istio-token") pod "6b6b91cd-f2b7-4c38-8386-6b421efc6037" (UID: "6b6b91cd-f2b7-4c38-8386-6b421efc6037"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 22:41:04.271240 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:41:04.271210 2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6b6b91cd-f2b7-4c38-8386-6b421efc6037-kube-api-access-tdcdd" (OuterVolumeSpecName: "kube-api-access-tdcdd") pod "6b6b91cd-f2b7-4c38-8386-6b421efc6037" (UID: "6b6b91cd-f2b7-4c38-8386-6b421efc6037"). InnerVolumeSpecName "kube-api-access-tdcdd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 22:41:04.370331 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:41:04.370289 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tdcdd\" (UniqueName: \"kubernetes.io/projected/6b6b91cd-f2b7-4c38-8386-6b421efc6037-kube-api-access-tdcdd\") on node \"ip-10-0-136-251.ec2.internal\" DevicePath \"\"" Jul 09 22:41:04.370331 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:41:04.370325 2579 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6b6b91cd-f2b7-4c38-8386-6b421efc6037-istio-data\") on node \"ip-10-0-136-251.ec2.internal\" DevicePath \"\"" Jul 09 22:41:04.370331 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:41:04.370338 2579 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6b6b91cd-f2b7-4c38-8386-6b421efc6037-istio-envoy\") on node \"ip-10-0-136-251.ec2.internal\" DevicePath \"\"" Jul 09 22:41:04.370575 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:41:04.370349 2579 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6b6b91cd-f2b7-4c38-8386-6b421efc6037-istio-podinfo\") on node \"ip-10-0-136-251.ec2.internal\" DevicePath \"\"" Jul 09 22:41:04.370575 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:41:04.370360 2579 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6b6b91cd-f2b7-4c38-8386-6b421efc6037-istio-token\") on node \"ip-10-0-136-251.ec2.internal\" DevicePath \"\"" Jul 09 22:41:04.370575 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:41:04.370374 2579 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b6b91cd-f2b7-4c38-8386-6b421efc6037-workload-socket\") on node \"ip-10-0-136-251.ec2.internal\" DevicePath \"\"" Jul 09 22:41:04.872433 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:41:04.872340 2579 generic.go:356] "Generic (PLEG): container finished" podID="6b6b91cd-f2b7-4c38-8386-6b421efc6037" containerID="978a2e9d77be959232e95080dddffc51563a63716629c3e29515bae6da00f807" exitCode=0 Jul 09 22:41:04.872433 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:41:04.872394 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-489a4fc6-openshift-default-7d96c6c8f7-v5jwn" Jul 09 22:41:04.872646 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:41:04.872423 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-489a4fc6-openshift-default-7d96c6c8f7-v5jwn" event={"ID":"6b6b91cd-f2b7-4c38-8386-6b421efc6037","Type":"ContainerDied","Data":"978a2e9d77be959232e95080dddffc51563a63716629c3e29515bae6da00f807"} Jul 09 22:41:04.872646 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:41:04.872483 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-489a4fc6-openshift-default-7d96c6c8f7-v5jwn" event={"ID":"6b6b91cd-f2b7-4c38-8386-6b421efc6037","Type":"ContainerDied","Data":"3a8f64b735d351c2836266a6091c5a7739e094f02fe78ba43dc0bbb8306ca39d"} Jul 09 22:41:04.872646 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:41:04.872505 2579 scope.go:117] "RemoveContainer" containerID="978a2e9d77be959232e95080dddffc51563a63716629c3e29515bae6da00f807" Jul 09 22:41:04.881404 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:41:04.881386 2579 scope.go:117] "RemoveContainer" containerID="8df796d1cbd763926ad8d75a0e4c3b510d21c71512281d898c8de974466daadc" Jul 09 22:41:04.889932 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:41:04.889890 2579 scope.go:117] "RemoveContainer" containerID="978a2e9d77be959232e95080dddffc51563a63716629c3e29515bae6da00f807" Jul 09 22:41:04.890276 ip-10-0-136-251 kubenswrapper[2579]: E0709 22:41:04.890250 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"978a2e9d77be959232e95080dddffc51563a63716629c3e29515bae6da00f807\": container with ID starting with 978a2e9d77be959232e95080dddffc51563a63716629c3e29515bae6da00f807 not found: ID does not exist" containerID="978a2e9d77be959232e95080dddffc51563a63716629c3e29515bae6da00f807" Jul 09 22:41:04.890363 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:41:04.890284 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"978a2e9d77be959232e95080dddffc51563a63716629c3e29515bae6da00f807"} err="failed to get container status \"978a2e9d77be959232e95080dddffc51563a63716629c3e29515bae6da00f807\": rpc error: code = NotFound desc = could not find container \"978a2e9d77be959232e95080dddffc51563a63716629c3e29515bae6da00f807\": container with ID starting with 978a2e9d77be959232e95080dddffc51563a63716629c3e29515bae6da00f807 not found: ID does not exist" Jul 09 22:41:04.890363 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:41:04.890305 2579 scope.go:117] "RemoveContainer" containerID="8df796d1cbd763926ad8d75a0e4c3b510d21c71512281d898c8de974466daadc" Jul 09 22:41:04.890573 ip-10-0-136-251 kubenswrapper[2579]: E0709 22:41:04.890558 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8df796d1cbd763926ad8d75a0e4c3b510d21c71512281d898c8de974466daadc\": container with ID starting with 8df796d1cbd763926ad8d75a0e4c3b510d21c71512281d898c8de974466daadc not found: ID does not exist" containerID="8df796d1cbd763926ad8d75a0e4c3b510d21c71512281d898c8de974466daadc" Jul 09 22:41:04.890615 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:41:04.890577 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8df796d1cbd763926ad8d75a0e4c3b510d21c71512281d898c8de974466daadc"} err="failed to get container status \"8df796d1cbd763926ad8d75a0e4c3b510d21c71512281d898c8de974466daadc\": rpc error: code = NotFound desc = could not find container \"8df796d1cbd763926ad8d75a0e4c3b510d21c71512281d898c8de974466daadc\": container with ID starting with 8df796d1cbd763926ad8d75a0e4c3b510d21c71512281d898c8de974466daadc not found: ID does not exist" Jul 09 22:41:04.891787 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:41:04.891766 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-489a4fc6-openshift-default-7d96c6c8f7-v5jwn"] Jul 09 22:41:04.898475 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:41:04.898453 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-489a4fc6-openshift-default-7d96c6c8f7-v5jwn"] Jul 09 22:41:06.565259 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:41:06.565225 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6b6b91cd-f2b7-4c38-8386-6b421efc6037" path="/var/lib/kubelet/pods/6b6b91cd-f2b7-4c38-8386-6b421efc6037/volumes" Jul 09 22:41:10.972052 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:41:10.972014 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-fdfdaecd-openshift-default-76f945c577-j9p84"] Jul 09 22:41:10.972592 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:41:10.972571 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f97b904d-4283-4e71-a41d-27831c412327" containerName="revoke-keys" Jul 09 22:41:10.972697 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:41:10.972595 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="f97b904d-4283-4e71-a41d-27831c412327" containerName="revoke-keys" Jul 09 22:41:10.972697 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:41:10.972614 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6b6b91cd-f2b7-4c38-8386-6b421efc6037" containerName="istio-proxy" Jul 09 22:41:10.972697 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:41:10.972624 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="6b6b91cd-f2b7-4c38-8386-6b421efc6037" containerName="istio-proxy" Jul 09 22:41:10.972697 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:41:10.972636 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d5613506-b584-4cfd-954e-eb6a5605350e" containerName="revoke-keys" Jul 09 22:41:10.972697 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:41:10.972644 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="d5613506-b584-4cfd-954e-eb6a5605350e" containerName="revoke-keys" Jul 09 22:41:10.972697 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:41:10.972674 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6b6b91cd-f2b7-4c38-8386-6b421efc6037" containerName="istio-proxy" Jul 09 22:41:10.972697 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:41:10.972683 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="6b6b91cd-f2b7-4c38-8386-6b421efc6037" containerName="istio-proxy" Jul 09 22:41:10.973049 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:41:10.972785 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="d5613506-b584-4cfd-954e-eb6a5605350e" containerName="revoke-keys" Jul 09 22:41:10.973049 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:41:10.972799 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="6b6b91cd-f2b7-4c38-8386-6b421efc6037" containerName="istio-proxy" Jul 09 22:41:10.973049 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:41:10.972810 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="6b6b91cd-f2b7-4c38-8386-6b421efc6037" containerName="istio-proxy" Jul 09 22:41:10.977664 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:41:10.977642 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-fdfdaecd-openshift-default-76f945c577-j9p84" Jul 09 22:41:10.981212 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:41:10.981180 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-fdfdaecd-openshift-default-dockercfg-drgdt\"" Jul 09 22:41:10.991705 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:41:10.991677 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-fdfdaecd-openshift-default-76f945c577-j9p84"] Jul 09 22:41:11.030900 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:41:11.030861 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9c62156-984b-493c-95b9-ab4b5e6a9f98-credential-socket\") pod \"e2e-ait-fdfdaecd-openshift-default-76f945c577-j9p84\" (UID: \"c9c62156-984b-493c-95b9-ab4b5e6a9f98\") " pod="openshift-ingress/e2e-ait-fdfdaecd-openshift-default-76f945c577-j9p84" Jul 09 22:41:11.030900 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:41:11.030903 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9c62156-984b-493c-95b9-ab4b5e6a9f98-workload-socket\") pod \"e2e-ait-fdfdaecd-openshift-default-76f945c577-j9p84\" (UID: \"c9c62156-984b-493c-95b9-ab4b5e6a9f98\") " pod="openshift-ingress/e2e-ait-fdfdaecd-openshift-default-76f945c577-j9p84" Jul 09 22:41:11.031148 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:41:11.030957 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c9c62156-984b-493c-95b9-ab4b5e6a9f98-istio-token\") pod \"e2e-ait-fdfdaecd-openshift-default-76f945c577-j9p84\" (UID: \"c9c62156-984b-493c-95b9-ab4b5e6a9f98\") " pod="openshift-ingress/e2e-ait-fdfdaecd-openshift-default-76f945c577-j9p84" Jul 09 22:41:11.031148 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:41:11.031004 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q94nt\" (UniqueName: \"kubernetes.io/projected/c9c62156-984b-493c-95b9-ab4b5e6a9f98-kube-api-access-q94nt\") pod \"e2e-ait-fdfdaecd-openshift-default-76f945c577-j9p84\" (UID: \"c9c62156-984b-493c-95b9-ab4b5e6a9f98\") " pod="openshift-ingress/e2e-ait-fdfdaecd-openshift-default-76f945c577-j9p84" Jul 09 22:41:11.031148 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:41:11.031027 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c9c62156-984b-493c-95b9-ab4b5e6a9f98-istio-podinfo\") pod \"e2e-ait-fdfdaecd-openshift-default-76f945c577-j9p84\" (UID: \"c9c62156-984b-493c-95b9-ab4b5e6a9f98\") " pod="openshift-ingress/e2e-ait-fdfdaecd-openshift-default-76f945c577-j9p84" Jul 09 22:41:11.031148 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:41:11.031064 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c9c62156-984b-493c-95b9-ab4b5e6a9f98-istio-data\") pod \"e2e-ait-fdfdaecd-openshift-default-76f945c577-j9p84\" (UID: \"c9c62156-984b-493c-95b9-ab4b5e6a9f98\") " pod="openshift-ingress/e2e-ait-fdfdaecd-openshift-default-76f945c577-j9p84" Jul 09 22:41:11.031148 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:41:11.031084 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c9c62156-984b-493c-95b9-ab4b5e6a9f98-istio-envoy\") pod \"e2e-ait-fdfdaecd-openshift-default-76f945c577-j9p84\" (UID: \"c9c62156-984b-493c-95b9-ab4b5e6a9f98\") " pod="openshift-ingress/e2e-ait-fdfdaecd-openshift-default-76f945c577-j9p84" Jul 09 22:41:11.031148 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:41:11.031104 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c9c62156-984b-493c-95b9-ab4b5e6a9f98-istiod-ca-cert\") pod \"e2e-ait-fdfdaecd-openshift-default-76f945c577-j9p84\" (UID: \"c9c62156-984b-493c-95b9-ab4b5e6a9f98\") " pod="openshift-ingress/e2e-ait-fdfdaecd-openshift-default-76f945c577-j9p84" Jul 09 22:41:11.031148 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:41:11.031146 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c9c62156-984b-493c-95b9-ab4b5e6a9f98-workload-certs\") pod \"e2e-ait-fdfdaecd-openshift-default-76f945c577-j9p84\" (UID: \"c9c62156-984b-493c-95b9-ab4b5e6a9f98\") " pod="openshift-ingress/e2e-ait-fdfdaecd-openshift-default-76f945c577-j9p84" Jul 09 22:41:11.132030 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:41:11.131998 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c9c62156-984b-493c-95b9-ab4b5e6a9f98-istio-token\") pod \"e2e-ait-fdfdaecd-openshift-default-76f945c577-j9p84\" (UID: \"c9c62156-984b-493c-95b9-ab4b5e6a9f98\") " pod="openshift-ingress/e2e-ait-fdfdaecd-openshift-default-76f945c577-j9p84" Jul 09 22:41:11.132030 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:41:11.132043 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q94nt\" (UniqueName: \"kubernetes.io/projected/c9c62156-984b-493c-95b9-ab4b5e6a9f98-kube-api-access-q94nt\") pod \"e2e-ait-fdfdaecd-openshift-default-76f945c577-j9p84\" (UID: \"c9c62156-984b-493c-95b9-ab4b5e6a9f98\") " pod="openshift-ingress/e2e-ait-fdfdaecd-openshift-default-76f945c577-j9p84" Jul 09 22:41:11.132304 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:41:11.132069 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c9c62156-984b-493c-95b9-ab4b5e6a9f98-istio-podinfo\") pod \"e2e-ait-fdfdaecd-openshift-default-76f945c577-j9p84\" (UID: \"c9c62156-984b-493c-95b9-ab4b5e6a9f98\") " pod="openshift-ingress/e2e-ait-fdfdaecd-openshift-default-76f945c577-j9p84" Jul 09 22:41:11.132304 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:41:11.132205 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c9c62156-984b-493c-95b9-ab4b5e6a9f98-istio-data\") pod \"e2e-ait-fdfdaecd-openshift-default-76f945c577-j9p84\" (UID: \"c9c62156-984b-493c-95b9-ab4b5e6a9f98\") " pod="openshift-ingress/e2e-ait-fdfdaecd-openshift-default-76f945c577-j9p84" Jul 09 22:41:11.132304 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:41:11.132255 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c9c62156-984b-493c-95b9-ab4b5e6a9f98-istio-envoy\") pod \"e2e-ait-fdfdaecd-openshift-default-76f945c577-j9p84\" (UID: \"c9c62156-984b-493c-95b9-ab4b5e6a9f98\") " pod="openshift-ingress/e2e-ait-fdfdaecd-openshift-default-76f945c577-j9p84" Jul 09 22:41:11.132304 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:41:11.132291 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c9c62156-984b-493c-95b9-ab4b5e6a9f98-istiod-ca-cert\") pod \"e2e-ait-fdfdaecd-openshift-default-76f945c577-j9p84\" (UID: \"c9c62156-984b-493c-95b9-ab4b5e6a9f98\") " pod="openshift-ingress/e2e-ait-fdfdaecd-openshift-default-76f945c577-j9p84" Jul 09 22:41:11.132503 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:41:11.132346 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c9c62156-984b-493c-95b9-ab4b5e6a9f98-workload-certs\") pod \"e2e-ait-fdfdaecd-openshift-default-76f945c577-j9p84\" (UID: \"c9c62156-984b-493c-95b9-ab4b5e6a9f98\") " pod="openshift-ingress/e2e-ait-fdfdaecd-openshift-default-76f945c577-j9p84" Jul 09 22:41:11.132503 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:41:11.132417 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9c62156-984b-493c-95b9-ab4b5e6a9f98-credential-socket\") pod \"e2e-ait-fdfdaecd-openshift-default-76f945c577-j9p84\" (UID: \"c9c62156-984b-493c-95b9-ab4b5e6a9f98\") " pod="openshift-ingress/e2e-ait-fdfdaecd-openshift-default-76f945c577-j9p84" Jul 09 22:41:11.132503 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:41:11.132447 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9c62156-984b-493c-95b9-ab4b5e6a9f98-workload-socket\") pod \"e2e-ait-fdfdaecd-openshift-default-76f945c577-j9p84\" (UID: \"c9c62156-984b-493c-95b9-ab4b5e6a9f98\") " pod="openshift-ingress/e2e-ait-fdfdaecd-openshift-default-76f945c577-j9p84" Jul 09 22:41:11.132679 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:41:11.132646 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c9c62156-984b-493c-95b9-ab4b5e6a9f98-istio-data\") pod \"e2e-ait-fdfdaecd-openshift-default-76f945c577-j9p84\" (UID: \"c9c62156-984b-493c-95b9-ab4b5e6a9f98\") " pod="openshift-ingress/e2e-ait-fdfdaecd-openshift-default-76f945c577-j9p84" Jul 09 22:41:11.132742 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:41:11.132679 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c9c62156-984b-493c-95b9-ab4b5e6a9f98-workload-certs\") pod \"e2e-ait-fdfdaecd-openshift-default-76f945c577-j9p84\" (UID: \"c9c62156-984b-493c-95b9-ab4b5e6a9f98\") " pod="openshift-ingress/e2e-ait-fdfdaecd-openshift-default-76f945c577-j9p84" Jul 09 22:41:11.132793 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:41:11.132757 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9c62156-984b-493c-95b9-ab4b5e6a9f98-workload-socket\") pod \"e2e-ait-fdfdaecd-openshift-default-76f945c577-j9p84\" (UID: \"c9c62156-984b-493c-95b9-ab4b5e6a9f98\") " pod="openshift-ingress/e2e-ait-fdfdaecd-openshift-default-76f945c577-j9p84" Jul 09 22:41:11.132833 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:41:11.132808 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9c62156-984b-493c-95b9-ab4b5e6a9f98-credential-socket\") pod \"e2e-ait-fdfdaecd-openshift-default-76f945c577-j9p84\" (UID: \"c9c62156-984b-493c-95b9-ab4b5e6a9f98\") " pod="openshift-ingress/e2e-ait-fdfdaecd-openshift-default-76f945c577-j9p84" Jul 09 22:41:11.133259 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:41:11.133237 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c9c62156-984b-493c-95b9-ab4b5e6a9f98-istiod-ca-cert\") pod \"e2e-ait-fdfdaecd-openshift-default-76f945c577-j9p84\" (UID: \"c9c62156-984b-493c-95b9-ab4b5e6a9f98\") " pod="openshift-ingress/e2e-ait-fdfdaecd-openshift-default-76f945c577-j9p84" Jul 09 22:41:11.134438 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:41:11.134415 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c9c62156-984b-493c-95b9-ab4b5e6a9f98-istio-envoy\") pod \"e2e-ait-fdfdaecd-openshift-default-76f945c577-j9p84\" (UID: \"c9c62156-984b-493c-95b9-ab4b5e6a9f98\") " pod="openshift-ingress/e2e-ait-fdfdaecd-openshift-default-76f945c577-j9p84" Jul 09 22:41:11.134726 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:41:11.134709 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c9c62156-984b-493c-95b9-ab4b5e6a9f98-istio-podinfo\") pod \"e2e-ait-fdfdaecd-openshift-default-76f945c577-j9p84\" (UID: \"c9c62156-984b-493c-95b9-ab4b5e6a9f98\") " pod="openshift-ingress/e2e-ait-fdfdaecd-openshift-default-76f945c577-j9p84" Jul 09 22:41:11.140823 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:41:11.140799 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c9c62156-984b-493c-95b9-ab4b5e6a9f98-istio-token\") pod \"e2e-ait-fdfdaecd-openshift-default-76f945c577-j9p84\" (UID: \"c9c62156-984b-493c-95b9-ab4b5e6a9f98\") " pod="openshift-ingress/e2e-ait-fdfdaecd-openshift-default-76f945c577-j9p84" Jul 09 22:41:11.141150 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:41:11.141133 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q94nt\" (UniqueName: \"kubernetes.io/projected/c9c62156-984b-493c-95b9-ab4b5e6a9f98-kube-api-access-q94nt\") pod \"e2e-ait-fdfdaecd-openshift-default-76f945c577-j9p84\" (UID: \"c9c62156-984b-493c-95b9-ab4b5e6a9f98\") " pod="openshift-ingress/e2e-ait-fdfdaecd-openshift-default-76f945c577-j9p84" Jul 09 22:41:11.289969 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:41:11.289914 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-fdfdaecd-openshift-default-76f945c577-j9p84" Jul 09 22:41:11.432486 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:41:11.432458 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-fdfdaecd-openshift-default-76f945c577-j9p84"] Jul 09 22:41:11.434753 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:41:11.434723 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc9c62156_984b_493c_95b9_ab4b5e6a9f98.slice/crio-0622b7d8cb2b2ba1d9ef0d7faa8b09eec587fced2c4c9e81b46ea26aaa4fec44 WatchSource:0}: Error finding container 0622b7d8cb2b2ba1d9ef0d7faa8b09eec587fced2c4c9e81b46ea26aaa4fec44: Status 404 returned error can't find the container with id 0622b7d8cb2b2ba1d9ef0d7faa8b09eec587fced2c4c9e81b46ea26aaa4fec44 Jul 09 22:41:11.437050 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:41:11.437014 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 09 22:41:11.437162 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:41:11.437088 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 09 22:41:11.437162 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:41:11.437126 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 09 22:41:11.910723 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:41:11.910685 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-fdfdaecd-openshift-default-76f945c577-j9p84" event={"ID":"c9c62156-984b-493c-95b9-ab4b5e6a9f98","Type":"ContainerStarted","Data":"82154028ecefce593fde43598fc4b12fbd1b782ae4d1b6d2ca7eae668766e19c"} Jul 09 22:41:11.910943 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:41:11.910753 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-fdfdaecd-openshift-default-76f945c577-j9p84" event={"ID":"c9c62156-984b-493c-95b9-ab4b5e6a9f98","Type":"ContainerStarted","Data":"0622b7d8cb2b2ba1d9ef0d7faa8b09eec587fced2c4c9e81b46ea26aaa4fec44"} Jul 09 22:41:11.932102 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:41:11.932041 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-fdfdaecd-openshift-default-76f945c577-j9p84" podStartSLOduration=1.932022265 podStartE2EDuration="1.932022265s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 22:41:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 22:41:11.93123303 +0000 UTC m=+1863.842685752" watchObservedRunningTime="2026-07-09 22:41:11.932022265 +0000 UTC m=+1863.843475042" Jul 09 22:41:12.290519 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:41:12.290477 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-fdfdaecd-openshift-default-76f945c577-j9p84" Jul 09 22:41:12.295622 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:41:12.295590 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-fdfdaecd-openshift-default-76f945c577-j9p84" Jul 09 22:41:12.916036 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:41:12.915999 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-fdfdaecd-openshift-default-76f945c577-j9p84" Jul 09 22:41:12.917052 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:41:12.917027 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-fdfdaecd-openshift-default-76f945c577-j9p84" Jul 09 22:41:14.722876 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:41:14.722837 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-xdfrr"] Jul 09 22:41:17.203201 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:41:17.203156 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fdfdaecd-n6thq"] Jul 09 22:41:17.206879 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:41:17.206861 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fdfdaecd-n6thq" Jul 09 22:41:17.209730 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:41:17.209708 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fdzhc\"" Jul 09 22:41:17.210159 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:41:17.210143 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 09 22:41:17.217156 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:41:17.217121 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fdfdaecd-n6thq"] Jul 09 22:41:17.295531 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:41:17.295489 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/08b884e8-79e7-4e06-98ea-b76a552d2fce-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-fdfdaecd-n6thq\" (UID: \"08b884e8-79e7-4e06-98ea-b76a552d2fce\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fdfdaecd-n6thq" Jul 09 22:41:17.396895 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:41:17.396860 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/08b884e8-79e7-4e06-98ea-b76a552d2fce-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-fdfdaecd-n6thq\" (UID: \"08b884e8-79e7-4e06-98ea-b76a552d2fce\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fdfdaecd-n6thq" Jul 09 22:41:17.397500 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:41:17.397479 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/08b884e8-79e7-4e06-98ea-b76a552d2fce-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-fdfdaecd-n6thq\" (UID: \"08b884e8-79e7-4e06-98ea-b76a552d2fce\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fdfdaecd-n6thq" Jul 09 22:41:17.517803 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:41:17.517772 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fdfdaecd-n6thq" Jul 09 22:41:17.775005 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:41:17.773780 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod08b884e8_79e7_4e06_98ea_b76a552d2fce.slice/crio-070d2203fcf520f73e3c7ab8a0340ea4c84f0278271ebe098458b4bc9d80bb44 WatchSource:0}: Error finding container 070d2203fcf520f73e3c7ab8a0340ea4c84f0278271ebe098458b4bc9d80bb44: Status 404 returned error can't find the container with id 070d2203fcf520f73e3c7ab8a0340ea4c84f0278271ebe098458b4bc9d80bb44 Jul 09 22:41:17.781466 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:41:17.779657 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fdfdaecd-n6thq"] Jul 09 22:41:17.942672 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:41:17.942623 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fdfdaecd-n6thq" event={"ID":"08b884e8-79e7-4e06-98ea-b76a552d2fce","Type":"ContainerStarted","Data":"070d2203fcf520f73e3c7ab8a0340ea4c84f0278271ebe098458b4bc9d80bb44"} Jul 09 22:41:18.953667 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:41:18.953617 2579 generic.go:356] "Generic (PLEG): container finished" podID="08b884e8-79e7-4e06-98ea-b76a552d2fce" containerID="29182852eb89a528d2d5b68f6ab5c42313bd6512092169fcf6fe3f5612802ae4" exitCode=7 Jul 09 22:41:18.954162 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:41:18.953715 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fdfdaecd-n6thq" event={"ID":"08b884e8-79e7-4e06-98ea-b76a552d2fce","Type":"ContainerDied","Data":"29182852eb89a528d2d5b68f6ab5c42313bd6512092169fcf6fe3f5612802ae4"} Jul 09 22:41:18.954162 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:41:18.954129 2579 scope.go:117] "RemoveContainer" containerID="29182852eb89a528d2d5b68f6ab5c42313bd6512092169fcf6fe3f5612802ae4" Jul 09 22:41:19.960342 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:41:19.960296 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fdfdaecd-n6thq" event={"ID":"08b884e8-79e7-4e06-98ea-b76a552d2fce","Type":"ContainerStarted","Data":"9812f0d5beabc742b1555a39db10b5296d768a2c06775ece0a61e5d76e95524a"} Jul 09 22:41:19.979606 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:41:19.979545 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fdfdaecd-n6thq" podStartSLOduration=2.979529002 podStartE2EDuration="2.979529002s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 22:41:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 22:41:19.977959147 +0000 UTC m=+1871.889411866" watchObservedRunningTime="2026-07-09 22:41:19.979529002 +0000 UTC m=+1871.890981702" Jul 09 22:41:20.968107 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:41:20.968067 2579 generic.go:356] "Generic (PLEG): container finished" podID="08b884e8-79e7-4e06-98ea-b76a552d2fce" containerID="9812f0d5beabc742b1555a39db10b5296d768a2c06775ece0a61e5d76e95524a" exitCode=0 Jul 09 22:41:20.968571 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:41:20.968148 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fdfdaecd-n6thq" event={"ID":"08b884e8-79e7-4e06-98ea-b76a552d2fce","Type":"ContainerDied","Data":"9812f0d5beabc742b1555a39db10b5296d768a2c06775ece0a61e5d76e95524a"} Jul 09 22:41:20.968571 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:41:20.968215 2579 scope.go:117] "RemoveContainer" containerID="29182852eb89a528d2d5b68f6ab5c42313bd6512092169fcf6fe3f5612802ae4" Jul 09 22:41:22.122751 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:41:22.122725 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fdfdaecd-n6thq" Jul 09 22:41:22.246034 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:41:22.245993 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/08b884e8-79e7-4e06-98ea-b76a552d2fce-maas-api-service-ca\") pod \"08b884e8-79e7-4e06-98ea-b76a552d2fce\" (UID: \"08b884e8-79e7-4e06-98ea-b76a552d2fce\") " Jul 09 22:41:22.246411 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:41:22.246383 2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/08b884e8-79e7-4e06-98ea-b76a552d2fce-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "08b884e8-79e7-4e06-98ea-b76a552d2fce" (UID: "08b884e8-79e7-4e06-98ea-b76a552d2fce"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 09 22:41:22.347188 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:41:22.347146 2579 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/08b884e8-79e7-4e06-98ea-b76a552d2fce-maas-api-service-ca\") on node \"ip-10-0-136-251.ec2.internal\" DevicePath \"\"" Jul 09 22:41:22.983049 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:41:22.982961 2579 generic.go:356] "Generic (PLEG): container finished" podID="c9c62156-984b-493c-95b9-ab4b5e6a9f98" containerID="82154028ecefce593fde43598fc4b12fbd1b782ae4d1b6d2ca7eae668766e19c" exitCode=0 Jul 09 22:41:22.983049 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:41:22.983035 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-fdfdaecd-openshift-default-76f945c577-j9p84" event={"ID":"c9c62156-984b-493c-95b9-ab4b5e6a9f98","Type":"ContainerDied","Data":"82154028ecefce593fde43598fc4b12fbd1b782ae4d1b6d2ca7eae668766e19c"} Jul 09 22:41:22.983502 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:41:22.983481 2579 scope.go:117] "RemoveContainer" containerID="82154028ecefce593fde43598fc4b12fbd1b782ae4d1b6d2ca7eae668766e19c" Jul 09 22:41:22.984515 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:41:22.984495 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fdfdaecd-n6thq" Jul 09 22:41:22.984621 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:41:22.984510 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fdfdaecd-n6thq" event={"ID":"08b884e8-79e7-4e06-98ea-b76a552d2fce","Type":"ContainerDied","Data":"070d2203fcf520f73e3c7ab8a0340ea4c84f0278271ebe098458b4bc9d80bb44"} Jul 09 22:41:22.984621 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:41:22.984566 2579 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="070d2203fcf520f73e3c7ab8a0340ea4c84f0278271ebe098458b4bc9d80bb44" Jul 09 22:41:22.984700 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:41:22.984616 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 09 22:41:22.984700 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:41:22.984678 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 09 22:41:22.984782 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:41:22.984714 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 09 22:41:23.991273 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:41:23.991237 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-fdfdaecd-openshift-default-76f945c577-j9p84" event={"ID":"c9c62156-984b-493c-95b9-ab4b5e6a9f98","Type":"ContainerStarted","Data":"23c00b6eb99a7e8b263f0f9e9fe77397f5ffa51a426dac57891a794bde1410da"} Jul 09 22:41:24.290754 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:41:24.290653 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-fdfdaecd-openshift-default-76f945c577-j9p84" Jul 09 22:41:24.291987 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:41:24.291962 2579 patch_prober.go:28] interesting pod/e2e-ait-fdfdaecd-openshift-default-76f945c577-j9p84 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.37:15021/healthz/ready\": dial tcp 10.132.0.37:15021: connect: connection refused" start-of-body= Jul 09 22:41:24.292118 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:41:24.292010 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-fdfdaecd-openshift-default-76f945c577-j9p84" podUID="c9c62156-984b-493c-95b9-ab4b5e6a9f98" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.37:15021/healthz/ready\": dial tcp 10.132.0.37:15021: connect: connection refused" Jul 09 22:41:25.291020 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:41:25.290977 2579 patch_prober.go:28] interesting pod/e2e-ait-fdfdaecd-openshift-default-76f945c577-j9p84 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.37:15021/healthz/ready\": dial tcp 10.132.0.37:15021: connect: connection refused" start-of-body= Jul 09 22:41:25.291412 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:41:25.291037 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-fdfdaecd-openshift-default-76f945c577-j9p84" podUID="c9c62156-984b-493c-95b9-ab4b5e6a9f98" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.37:15021/healthz/ready\": dial tcp 10.132.0.37:15021: connect: connection refused" Jul 09 22:41:26.290677 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:41:26.290634 2579 patch_prober.go:28] interesting pod/e2e-ait-fdfdaecd-openshift-default-76f945c577-j9p84 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.37:15021/healthz/ready\": dial tcp 10.132.0.37:15021: connect: connection refused" start-of-body= Jul 09 22:41:26.290866 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:41:26.290693 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-fdfdaecd-openshift-default-76f945c577-j9p84" podUID="c9c62156-984b-493c-95b9-ab4b5e6a9f98" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.37:15021/healthz/ready\": dial tcp 10.132.0.37:15021: connect: connection refused" Jul 09 22:41:27.291127 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:41:27.291085 2579 patch_prober.go:28] interesting pod/e2e-ait-fdfdaecd-openshift-default-76f945c577-j9p84 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.37:15021/healthz/ready\": dial tcp 10.132.0.37:15021: connect: connection refused" start-of-body= Jul 09 22:41:27.291532 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:41:27.291147 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-fdfdaecd-openshift-default-76f945c577-j9p84" podUID="c9c62156-984b-493c-95b9-ab4b5e6a9f98" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.37:15021/healthz/ready\": dial tcp 10.132.0.37:15021: connect: connection refused" Jul 09 22:41:27.916162 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:41:27.916101 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-fdfdaecd-openshift-default-76f945c577-j9p84" Jul 09 22:41:28.294843 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:41:28.294814 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-fdfdaecd-openshift-default-76f945c577-j9p84" Jul 09 22:41:28.295866 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:41:28.295851 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-fdfdaecd-openshift-default-76f945c577-j9p84" Jul 09 22:41:30.316002 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:41:30.315963 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-xdfrr"] Jul 09 22:41:43.256276 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:41:43.256241 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-fdfdaecd-openshift-default-76f945c577-j9p84"] Jul 09 22:41:43.256713 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:41:43.256527 2579 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-fdfdaecd-openshift-default-76f945c577-j9p84" podUID="c9c62156-984b-493c-95b9-ab4b5e6a9f98" containerName="istio-proxy" containerID="cri-o://23c00b6eb99a7e8b263f0f9e9fe77397f5ffa51a426dac57891a794bde1410da" gracePeriod=30 Jul 09 22:41:43.297361 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:41:43.297324 2579 patch_prober.go:28] interesting pod/e2e-ait-fdfdaecd-openshift-default-76f945c577-j9p84 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 09 22:41:43.297526 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:41:43.297404 2579 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-fdfdaecd-openshift-default-76f945c577-j9p84" podUID="c9c62156-984b-493c-95b9-ab4b5e6a9f98" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 09 22:41:43.566291 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:41:43.566213 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-derive-e08843d5-openshift-default-864445fbc9-tqhpv"] Jul 09 22:41:43.566596 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:41:43.566584 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="08b884e8-79e7-4e06-98ea-b76a552d2fce" containerName="revoke-keys" Jul 09 22:41:43.566648 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:41:43.566597 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="08b884e8-79e7-4e06-98ea-b76a552d2fce" containerName="revoke-keys" Jul 09 22:41:43.566648 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:41:43.566619 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="08b884e8-79e7-4e06-98ea-b76a552d2fce" containerName="revoke-keys" Jul 09 22:41:43.566648 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:41:43.566625 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="08b884e8-79e7-4e06-98ea-b76a552d2fce" containerName="revoke-keys" Jul 09 22:41:43.566746 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:41:43.566682 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="08b884e8-79e7-4e06-98ea-b76a552d2fce" containerName="revoke-keys" Jul 09 22:41:43.566746 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:41:43.566690 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="08b884e8-79e7-4e06-98ea-b76a552d2fce" containerName="revoke-keys" Jul 09 22:41:43.568688 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:41:43.568671 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-e08843d5-openshift-default-864445fbc9-tqhpv" Jul 09 22:41:43.571539 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:41:43.571512 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-derive-e08843d5-openshift-default-dockercfg-kmpqs\"" Jul 09 22:41:43.581723 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:41:43.581694 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-e08843d5-openshift-default-864445fbc9-tqhpv"] Jul 09 22:41:43.759787 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:41:43.759744 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/032933bd-19ac-49f0-a484-62c78b1bfc7e-istiod-ca-cert\") pod \"e2e-derive-e08843d5-openshift-default-864445fbc9-tqhpv\" (UID: \"032933bd-19ac-49f0-a484-62c78b1bfc7e\") " pod="openshift-ingress/e2e-derive-e08843d5-openshift-default-864445fbc9-tqhpv" Jul 09 22:41:43.760023 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:41:43.759794 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/032933bd-19ac-49f0-a484-62c78b1bfc7e-credential-socket\") pod \"e2e-derive-e08843d5-openshift-default-864445fbc9-tqhpv\" (UID: \"032933bd-19ac-49f0-a484-62c78b1bfc7e\") " pod="openshift-ingress/e2e-derive-e08843d5-openshift-default-864445fbc9-tqhpv" Jul 09 22:41:43.760023 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:41:43.759825 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/032933bd-19ac-49f0-a484-62c78b1bfc7e-istio-podinfo\") pod \"e2e-derive-e08843d5-openshift-default-864445fbc9-tqhpv\" (UID: \"032933bd-19ac-49f0-a484-62c78b1bfc7e\") " pod="openshift-ingress/e2e-derive-e08843d5-openshift-default-864445fbc9-tqhpv" Jul 09 22:41:43.760023 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:41:43.759859 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/032933bd-19ac-49f0-a484-62c78b1bfc7e-istio-data\") pod \"e2e-derive-e08843d5-openshift-default-864445fbc9-tqhpv\" (UID: \"032933bd-19ac-49f0-a484-62c78b1bfc7e\") " pod="openshift-ingress/e2e-derive-e08843d5-openshift-default-864445fbc9-tqhpv" Jul 09 22:41:43.760023 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:41:43.759878 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/032933bd-19ac-49f0-a484-62c78b1bfc7e-workload-certs\") pod \"e2e-derive-e08843d5-openshift-default-864445fbc9-tqhpv\" (UID: \"032933bd-19ac-49f0-a484-62c78b1bfc7e\") " pod="openshift-ingress/e2e-derive-e08843d5-openshift-default-864445fbc9-tqhpv" Jul 09 22:41:43.760247 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:41:43.760014 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/032933bd-19ac-49f0-a484-62c78b1bfc7e-istio-token\") pod \"e2e-derive-e08843d5-openshift-default-864445fbc9-tqhpv\" (UID: \"032933bd-19ac-49f0-a484-62c78b1bfc7e\") " pod="openshift-ingress/e2e-derive-e08843d5-openshift-default-864445fbc9-tqhpv" Jul 09 22:41:43.760247 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:41:43.760065 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/032933bd-19ac-49f0-a484-62c78b1bfc7e-istio-envoy\") pod \"e2e-derive-e08843d5-openshift-default-864445fbc9-tqhpv\" (UID: \"032933bd-19ac-49f0-a484-62c78b1bfc7e\") " pod="openshift-ingress/e2e-derive-e08843d5-openshift-default-864445fbc9-tqhpv" Jul 09 22:41:43.760247 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:41:43.760086 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/032933bd-19ac-49f0-a484-62c78b1bfc7e-workload-socket\") pod \"e2e-derive-e08843d5-openshift-default-864445fbc9-tqhpv\" (UID: \"032933bd-19ac-49f0-a484-62c78b1bfc7e\") " pod="openshift-ingress/e2e-derive-e08843d5-openshift-default-864445fbc9-tqhpv" Jul 09 22:41:43.760247 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:41:43.760109 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nd24l\" (UniqueName: \"kubernetes.io/projected/032933bd-19ac-49f0-a484-62c78b1bfc7e-kube-api-access-nd24l\") pod \"e2e-derive-e08843d5-openshift-default-864445fbc9-tqhpv\" (UID: \"032933bd-19ac-49f0-a484-62c78b1bfc7e\") " pod="openshift-ingress/e2e-derive-e08843d5-openshift-default-864445fbc9-tqhpv" Jul 09 22:41:43.861138 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:41:43.861050 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/032933bd-19ac-49f0-a484-62c78b1bfc7e-istio-data\") pod \"e2e-derive-e08843d5-openshift-default-864445fbc9-tqhpv\" (UID: \"032933bd-19ac-49f0-a484-62c78b1bfc7e\") " pod="openshift-ingress/e2e-derive-e08843d5-openshift-default-864445fbc9-tqhpv" Jul 09 22:41:43.861138 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:41:43.861086 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/032933bd-19ac-49f0-a484-62c78b1bfc7e-workload-certs\") pod \"e2e-derive-e08843d5-openshift-default-864445fbc9-tqhpv\" (UID: \"032933bd-19ac-49f0-a484-62c78b1bfc7e\") " pod="openshift-ingress/e2e-derive-e08843d5-openshift-default-864445fbc9-tqhpv" Jul 09 22:41:43.861349 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:41:43.861213 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/032933bd-19ac-49f0-a484-62c78b1bfc7e-istio-token\") pod \"e2e-derive-e08843d5-openshift-default-864445fbc9-tqhpv\" (UID: \"032933bd-19ac-49f0-a484-62c78b1bfc7e\") " pod="openshift-ingress/e2e-derive-e08843d5-openshift-default-864445fbc9-tqhpv" Jul 09 22:41:43.861349 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:41:43.861253 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/032933bd-19ac-49f0-a484-62c78b1bfc7e-istio-envoy\") pod \"e2e-derive-e08843d5-openshift-default-864445fbc9-tqhpv\" (UID: \"032933bd-19ac-49f0-a484-62c78b1bfc7e\") " pod="openshift-ingress/e2e-derive-e08843d5-openshift-default-864445fbc9-tqhpv" Jul 09 22:41:43.861349 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:41:43.861280 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/032933bd-19ac-49f0-a484-62c78b1bfc7e-workload-socket\") pod \"e2e-derive-e08843d5-openshift-default-864445fbc9-tqhpv\" (UID: \"032933bd-19ac-49f0-a484-62c78b1bfc7e\") " pod="openshift-ingress/e2e-derive-e08843d5-openshift-default-864445fbc9-tqhpv" Jul 09 22:41:43.861349 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:41:43.861315 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nd24l\" (UniqueName: \"kubernetes.io/projected/032933bd-19ac-49f0-a484-62c78b1bfc7e-kube-api-access-nd24l\") pod \"e2e-derive-e08843d5-openshift-default-864445fbc9-tqhpv\" (UID: \"032933bd-19ac-49f0-a484-62c78b1bfc7e\") " pod="openshift-ingress/e2e-derive-e08843d5-openshift-default-864445fbc9-tqhpv" Jul 09 22:41:43.861571 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:41:43.861401 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/032933bd-19ac-49f0-a484-62c78b1bfc7e-istiod-ca-cert\") pod \"e2e-derive-e08843d5-openshift-default-864445fbc9-tqhpv\" (UID: \"032933bd-19ac-49f0-a484-62c78b1bfc7e\") " pod="openshift-ingress/e2e-derive-e08843d5-openshift-default-864445fbc9-tqhpv" Jul 09 22:41:43.861571 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:41:43.861440 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/032933bd-19ac-49f0-a484-62c78b1bfc7e-credential-socket\") pod \"e2e-derive-e08843d5-openshift-default-864445fbc9-tqhpv\" (UID: \"032933bd-19ac-49f0-a484-62c78b1bfc7e\") " pod="openshift-ingress/e2e-derive-e08843d5-openshift-default-864445fbc9-tqhpv" Jul 09 22:41:43.861571 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:41:43.861455 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/032933bd-19ac-49f0-a484-62c78b1bfc7e-istio-data\") pod \"e2e-derive-e08843d5-openshift-default-864445fbc9-tqhpv\" (UID: \"032933bd-19ac-49f0-a484-62c78b1bfc7e\") " pod="openshift-ingress/e2e-derive-e08843d5-openshift-default-864445fbc9-tqhpv" Jul 09 22:41:43.861571 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:41:43.861470 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/032933bd-19ac-49f0-a484-62c78b1bfc7e-istio-podinfo\") pod \"e2e-derive-e08843d5-openshift-default-864445fbc9-tqhpv\" (UID: \"032933bd-19ac-49f0-a484-62c78b1bfc7e\") " pod="openshift-ingress/e2e-derive-e08843d5-openshift-default-864445fbc9-tqhpv" Jul 09 22:41:43.861571 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:41:43.861503 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/032933bd-19ac-49f0-a484-62c78b1bfc7e-workload-certs\") pod \"e2e-derive-e08843d5-openshift-default-864445fbc9-tqhpv\" (UID: \"032933bd-19ac-49f0-a484-62c78b1bfc7e\") " pod="openshift-ingress/e2e-derive-e08843d5-openshift-default-864445fbc9-tqhpv" Jul 09 22:41:43.861821 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:41:43.861799 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/032933bd-19ac-49f0-a484-62c78b1bfc7e-credential-socket\") pod \"e2e-derive-e08843d5-openshift-default-864445fbc9-tqhpv\" (UID: \"032933bd-19ac-49f0-a484-62c78b1bfc7e\") " pod="openshift-ingress/e2e-derive-e08843d5-openshift-default-864445fbc9-tqhpv" Jul 09 22:41:43.861876 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:41:43.861832 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/032933bd-19ac-49f0-a484-62c78b1bfc7e-workload-socket\") pod \"e2e-derive-e08843d5-openshift-default-864445fbc9-tqhpv\" (UID: \"032933bd-19ac-49f0-a484-62c78b1bfc7e\") " pod="openshift-ingress/e2e-derive-e08843d5-openshift-default-864445fbc9-tqhpv" Jul 09 22:41:43.862159 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:41:43.862137 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/032933bd-19ac-49f0-a484-62c78b1bfc7e-istiod-ca-cert\") pod \"e2e-derive-e08843d5-openshift-default-864445fbc9-tqhpv\" (UID: \"032933bd-19ac-49f0-a484-62c78b1bfc7e\") " pod="openshift-ingress/e2e-derive-e08843d5-openshift-default-864445fbc9-tqhpv" Jul 09 22:41:43.863479 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:41:43.863459 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/032933bd-19ac-49f0-a484-62c78b1bfc7e-istio-envoy\") pod \"e2e-derive-e08843d5-openshift-default-864445fbc9-tqhpv\" (UID: \"032933bd-19ac-49f0-a484-62c78b1bfc7e\") " pod="openshift-ingress/e2e-derive-e08843d5-openshift-default-864445fbc9-tqhpv" Jul 09 22:41:43.863763 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:41:43.863746 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/032933bd-19ac-49f0-a484-62c78b1bfc7e-istio-podinfo\") pod \"e2e-derive-e08843d5-openshift-default-864445fbc9-tqhpv\" (UID: \"032933bd-19ac-49f0-a484-62c78b1bfc7e\") " pod="openshift-ingress/e2e-derive-e08843d5-openshift-default-864445fbc9-tqhpv" Jul 09 22:41:43.870012 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:41:43.869985 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/032933bd-19ac-49f0-a484-62c78b1bfc7e-istio-token\") pod \"e2e-derive-e08843d5-openshift-default-864445fbc9-tqhpv\" (UID: \"032933bd-19ac-49f0-a484-62c78b1bfc7e\") " pod="openshift-ingress/e2e-derive-e08843d5-openshift-default-864445fbc9-tqhpv" Jul 09 22:41:43.870131 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:41:43.870076 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nd24l\" (UniqueName: \"kubernetes.io/projected/032933bd-19ac-49f0-a484-62c78b1bfc7e-kube-api-access-nd24l\") pod \"e2e-derive-e08843d5-openshift-default-864445fbc9-tqhpv\" (UID: \"032933bd-19ac-49f0-a484-62c78b1bfc7e\") " pod="openshift-ingress/e2e-derive-e08843d5-openshift-default-864445fbc9-tqhpv" Jul 09 22:41:43.880545 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:41:43.880516 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-e08843d5-openshift-default-864445fbc9-tqhpv" Jul 09 22:41:44.022523 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:41:44.022491 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-e08843d5-openshift-default-864445fbc9-tqhpv"] Jul 09 22:41:44.024253 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:41:44.024220 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod032933bd_19ac_49f0_a484_62c78b1bfc7e.slice/crio-cbbb4faec2dbe8ecb8c32e2b749e9b5a4768564fb3a5f820b1c0dd56443ca4fe WatchSource:0}: Error finding container cbbb4faec2dbe8ecb8c32e2b749e9b5a4768564fb3a5f820b1c0dd56443ca4fe: Status 404 returned error can't find the container with id cbbb4faec2dbe8ecb8c32e2b749e9b5a4768564fb3a5f820b1c0dd56443ca4fe Jul 09 22:41:44.026621 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:41:44.026588 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 09 22:41:44.026843 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:41:44.026651 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 09 22:41:44.026843 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:41:44.026681 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 09 22:41:44.093809 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:41:44.093759 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-e08843d5-openshift-default-864445fbc9-tqhpv" event={"ID":"032933bd-19ac-49f0-a484-62c78b1bfc7e","Type":"ContainerStarted","Data":"42b5296eab5b632399c7c565157c5958b842bb0fe2b55e7d6a71456b4e39ab54"} Jul 09 22:41:44.093986 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:41:44.093831 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-e08843d5-openshift-default-864445fbc9-tqhpv" event={"ID":"032933bd-19ac-49f0-a484-62c78b1bfc7e","Type":"ContainerStarted","Data":"cbbb4faec2dbe8ecb8c32e2b749e9b5a4768564fb3a5f820b1c0dd56443ca4fe"} Jul 09 22:41:44.115708 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:41:44.115578 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-derive-e08843d5-openshift-default-864445fbc9-tqhpv" podStartSLOduration=1.115559098 podStartE2EDuration="1.115559098s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 22:41:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 22:41:44.113578554 +0000 UTC m=+1896.025031256" watchObservedRunningTime="2026-07-09 22:41:44.115559098 +0000 UTC m=+1896.027011797" Jul 09 22:41:44.881121 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:41:44.881075 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-e08843d5-openshift-default-864445fbc9-tqhpv" Jul 09 22:41:45.887586 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:41:45.887552 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-e08843d5-openshift-default-864445fbc9-tqhpv" Jul 09 22:41:46.107894 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:41:46.107864 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-e08843d5-openshift-default-864445fbc9-tqhpv" Jul 09 22:41:46.108889 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:41:46.108867 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-e08843d5-openshift-default-864445fbc9-tqhpv" Jul 09 22:41:47.410619 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:41:47.410581 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-xdfrr"] Jul 09 22:41:48.539148 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:41:48.539121 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-fdfdaecd-openshift-default-76f945c577-j9p84" Jul 09 22:41:48.603343 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:41:48.603307 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c9c62156-984b-493c-95b9-ab4b5e6a9f98-istiod-ca-cert\") pod \"c9c62156-984b-493c-95b9-ab4b5e6a9f98\" (UID: \"c9c62156-984b-493c-95b9-ab4b5e6a9f98\") " Jul 09 22:41:48.603542 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:41:48.603350 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9c62156-984b-493c-95b9-ab4b5e6a9f98-credential-socket\") pod \"c9c62156-984b-493c-95b9-ab4b5e6a9f98\" (UID: \"c9c62156-984b-493c-95b9-ab4b5e6a9f98\") " Jul 09 22:41:48.603542 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:41:48.603376 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c9c62156-984b-493c-95b9-ab4b5e6a9f98-workload-certs\") pod \"c9c62156-984b-493c-95b9-ab4b5e6a9f98\" (UID: \"c9c62156-984b-493c-95b9-ab4b5e6a9f98\") " Jul 09 22:41:48.603542 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:41:48.603414 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9c62156-984b-493c-95b9-ab4b5e6a9f98-workload-socket\") pod \"c9c62156-984b-493c-95b9-ab4b5e6a9f98\" (UID: \"c9c62156-984b-493c-95b9-ab4b5e6a9f98\") " Jul 09 22:41:48.603542 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:41:48.603473 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c9c62156-984b-493c-95b9-ab4b5e6a9f98-istio-token\") pod \"c9c62156-984b-493c-95b9-ab4b5e6a9f98\" (UID: \"c9c62156-984b-493c-95b9-ab4b5e6a9f98\") " Jul 09 22:41:48.603542 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:41:48.603499 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-q94nt\" (UniqueName: \"kubernetes.io/projected/c9c62156-984b-493c-95b9-ab4b5e6a9f98-kube-api-access-q94nt\") pod \"c9c62156-984b-493c-95b9-ab4b5e6a9f98\" (UID: \"c9c62156-984b-493c-95b9-ab4b5e6a9f98\") " Jul 09 22:41:48.603542 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:41:48.603535 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c9c62156-984b-493c-95b9-ab4b5e6a9f98-istio-podinfo\") pod \"c9c62156-984b-493c-95b9-ab4b5e6a9f98\" (UID: \"c9c62156-984b-493c-95b9-ab4b5e6a9f98\") " Jul 09 22:41:48.603869 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:41:48.603564 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c9c62156-984b-493c-95b9-ab4b5e6a9f98-istio-envoy\") pod \"c9c62156-984b-493c-95b9-ab4b5e6a9f98\" (UID: \"c9c62156-984b-493c-95b9-ab4b5e6a9f98\") " Jul 09 22:41:48.603869 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:41:48.603604 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c9c62156-984b-493c-95b9-ab4b5e6a9f98-istio-data\") pod \"c9c62156-984b-493c-95b9-ab4b5e6a9f98\" (UID: \"c9c62156-984b-493c-95b9-ab4b5e6a9f98\") " Jul 09 22:41:48.603869 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:41:48.603641 2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c9c62156-984b-493c-95b9-ab4b5e6a9f98-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "c9c62156-984b-493c-95b9-ab4b5e6a9f98" (UID: "c9c62156-984b-493c-95b9-ab4b5e6a9f98"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 22:41:48.603869 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:41:48.603841 2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c9c62156-984b-493c-95b9-ab4b5e6a9f98-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "c9c62156-984b-493c-95b9-ab4b5e6a9f98" (UID: "c9c62156-984b-493c-95b9-ab4b5e6a9f98"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 22:41:48.604189 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:41:48.603968 2579 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9c62156-984b-493c-95b9-ab4b5e6a9f98-credential-socket\") on node \"ip-10-0-136-251.ec2.internal\" DevicePath \"\"" Jul 09 22:41:48.604189 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:41:48.603987 2579 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9c62156-984b-493c-95b9-ab4b5e6a9f98-workload-socket\") on node \"ip-10-0-136-251.ec2.internal\" DevicePath \"\"" Jul 09 22:41:48.604944 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:41:48.604495 2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c9c62156-984b-493c-95b9-ab4b5e6a9f98-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "c9c62156-984b-493c-95b9-ab4b5e6a9f98" (UID: "c9c62156-984b-493c-95b9-ab4b5e6a9f98"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 22:41:48.604944 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:41:48.604782 2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c9c62156-984b-493c-95b9-ab4b5e6a9f98-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "c9c62156-984b-493c-95b9-ab4b5e6a9f98" (UID: "c9c62156-984b-493c-95b9-ab4b5e6a9f98"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 09 22:41:48.605114 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:41:48.605015 2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c9c62156-984b-493c-95b9-ab4b5e6a9f98-istio-data" (OuterVolumeSpecName: "istio-data") pod "c9c62156-984b-493c-95b9-ab4b5e6a9f98" (UID: "c9c62156-984b-493c-95b9-ab4b5e6a9f98"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 22:41:48.606133 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:41:48.606099 2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c9c62156-984b-493c-95b9-ab4b5e6a9f98-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "c9c62156-984b-493c-95b9-ab4b5e6a9f98" (UID: "c9c62156-984b-493c-95b9-ab4b5e6a9f98"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 09 22:41:48.606360 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:41:48.606336 2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c9c62156-984b-493c-95b9-ab4b5e6a9f98-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "c9c62156-984b-493c-95b9-ab4b5e6a9f98" (UID: "c9c62156-984b-493c-95b9-ab4b5e6a9f98"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 22:41:48.606628 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:41:48.606606 2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c9c62156-984b-493c-95b9-ab4b5e6a9f98-kube-api-access-q94nt" (OuterVolumeSpecName: "kube-api-access-q94nt") pod "c9c62156-984b-493c-95b9-ab4b5e6a9f98" (UID: "c9c62156-984b-493c-95b9-ab4b5e6a9f98"). InnerVolumeSpecName "kube-api-access-q94nt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 22:41:48.606808 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:41:48.606783 2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c9c62156-984b-493c-95b9-ab4b5e6a9f98-istio-token" (OuterVolumeSpecName: "istio-token") pod "c9c62156-984b-493c-95b9-ab4b5e6a9f98" (UID: "c9c62156-984b-493c-95b9-ab4b5e6a9f98"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 22:41:48.704880 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:41:48.704838 2579 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c9c62156-984b-493c-95b9-ab4b5e6a9f98-istio-token\") on node \"ip-10-0-136-251.ec2.internal\" DevicePath \"\"" Jul 09 22:41:48.704880 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:41:48.704873 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q94nt\" (UniqueName: \"kubernetes.io/projected/c9c62156-984b-493c-95b9-ab4b5e6a9f98-kube-api-access-q94nt\") on node \"ip-10-0-136-251.ec2.internal\" DevicePath \"\"" Jul 09 22:41:48.704880 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:41:48.704883 2579 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c9c62156-984b-493c-95b9-ab4b5e6a9f98-istio-podinfo\") on node \"ip-10-0-136-251.ec2.internal\" DevicePath \"\"" Jul 09 22:41:48.704880 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:41:48.704892 2579 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c9c62156-984b-493c-95b9-ab4b5e6a9f98-istio-envoy\") on node \"ip-10-0-136-251.ec2.internal\" DevicePath \"\"" Jul 09 22:41:48.705195 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:41:48.704900 2579 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c9c62156-984b-493c-95b9-ab4b5e6a9f98-istio-data\") on node \"ip-10-0-136-251.ec2.internal\" DevicePath \"\"" Jul 09 22:41:48.705195 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:41:48.704908 2579 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c9c62156-984b-493c-95b9-ab4b5e6a9f98-istiod-ca-cert\") on node \"ip-10-0-136-251.ec2.internal\" DevicePath \"\"" Jul 09 22:41:48.705195 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:41:48.704947 2579 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c9c62156-984b-493c-95b9-ab4b5e6a9f98-workload-certs\") on node \"ip-10-0-136-251.ec2.internal\" DevicePath \"\"" Jul 09 22:41:49.097939 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:41:49.097874 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-e08843d5-5jkb5"] Jul 09 22:41:49.098380 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:41:49.098361 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c9c62156-984b-493c-95b9-ab4b5e6a9f98" containerName="istio-proxy" Jul 09 22:41:49.098475 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:41:49.098382 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="c9c62156-984b-493c-95b9-ab4b5e6a9f98" containerName="istio-proxy" Jul 09 22:41:49.098475 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:41:49.098403 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c9c62156-984b-493c-95b9-ab4b5e6a9f98" containerName="istio-proxy" Jul 09 22:41:49.098475 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:41:49.098410 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="c9c62156-984b-493c-95b9-ab4b5e6a9f98" containerName="istio-proxy" Jul 09 22:41:49.098625 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:41:49.098509 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="c9c62156-984b-493c-95b9-ab4b5e6a9f98" containerName="istio-proxy" Jul 09 22:41:49.101121 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:41:49.101101 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-e08843d5-5jkb5" Jul 09 22:41:49.103941 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:41:49.103894 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fdzhc\"" Jul 09 22:41:49.104071 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:41:49.103945 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 09 22:41:49.107867 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:41:49.107835 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4a1a24ae-c92f-45dc-91f8-2abb2c3ea1f1-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-e08843d5-5jkb5\" (UID: \"4a1a24ae-c92f-45dc-91f8-2abb2c3ea1f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-e08843d5-5jkb5" Jul 09 22:41:49.111119 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:41:49.111084 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-e08843d5-5jkb5"] Jul 09 22:41:49.124507 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:41:49.124475 2579 generic.go:356] "Generic (PLEG): container finished" podID="c9c62156-984b-493c-95b9-ab4b5e6a9f98" containerID="23c00b6eb99a7e8b263f0f9e9fe77397f5ffa51a426dac57891a794bde1410da" exitCode=0 Jul 09 22:41:49.124682 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:41:49.124531 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-fdfdaecd-openshift-default-76f945c577-j9p84" Jul 09 22:41:49.124682 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:41:49.124548 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-fdfdaecd-openshift-default-76f945c577-j9p84" event={"ID":"c9c62156-984b-493c-95b9-ab4b5e6a9f98","Type":"ContainerDied","Data":"23c00b6eb99a7e8b263f0f9e9fe77397f5ffa51a426dac57891a794bde1410da"} Jul 09 22:41:49.124682 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:41:49.124600 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-fdfdaecd-openshift-default-76f945c577-j9p84" event={"ID":"c9c62156-984b-493c-95b9-ab4b5e6a9f98","Type":"ContainerDied","Data":"0622b7d8cb2b2ba1d9ef0d7faa8b09eec587fced2c4c9e81b46ea26aaa4fec44"} Jul 09 22:41:49.124682 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:41:49.124622 2579 scope.go:117] "RemoveContainer" containerID="23c00b6eb99a7e8b263f0f9e9fe77397f5ffa51a426dac57891a794bde1410da" Jul 09 22:41:49.134560 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:41:49.134541 2579 scope.go:117] "RemoveContainer" containerID="82154028ecefce593fde43598fc4b12fbd1b782ae4d1b6d2ca7eae668766e19c" Jul 09 22:41:49.161970 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:41:49.158673 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-fdfdaecd-openshift-default-76f945c577-j9p84"] Jul 09 22:41:49.167966 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:41:49.165112 2579 scope.go:117] "RemoveContainer" containerID="23c00b6eb99a7e8b263f0f9e9fe77397f5ffa51a426dac57891a794bde1410da" Jul 09 22:41:49.167966 ip-10-0-136-251 kubenswrapper[2579]: E0709 22:41:49.165529 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"23c00b6eb99a7e8b263f0f9e9fe77397f5ffa51a426dac57891a794bde1410da\": container with ID starting with 23c00b6eb99a7e8b263f0f9e9fe77397f5ffa51a426dac57891a794bde1410da not found: ID does not exist" containerID="23c00b6eb99a7e8b263f0f9e9fe77397f5ffa51a426dac57891a794bde1410da" Jul 09 22:41:49.167966 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:41:49.165566 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"23c00b6eb99a7e8b263f0f9e9fe77397f5ffa51a426dac57891a794bde1410da"} err="failed to get container status \"23c00b6eb99a7e8b263f0f9e9fe77397f5ffa51a426dac57891a794bde1410da\": rpc error: code = NotFound desc = could not find container \"23c00b6eb99a7e8b263f0f9e9fe77397f5ffa51a426dac57891a794bde1410da\": container with ID starting with 23c00b6eb99a7e8b263f0f9e9fe77397f5ffa51a426dac57891a794bde1410da not found: ID does not exist" Jul 09 22:41:49.167966 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:41:49.165592 2579 scope.go:117] "RemoveContainer" containerID="82154028ecefce593fde43598fc4b12fbd1b782ae4d1b6d2ca7eae668766e19c" Jul 09 22:41:49.167966 ip-10-0-136-251 kubenswrapper[2579]: E0709 22:41:49.165843 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"82154028ecefce593fde43598fc4b12fbd1b782ae4d1b6d2ca7eae668766e19c\": container with ID starting with 82154028ecefce593fde43598fc4b12fbd1b782ae4d1b6d2ca7eae668766e19c not found: ID does not exist" containerID="82154028ecefce593fde43598fc4b12fbd1b782ae4d1b6d2ca7eae668766e19c" Jul 09 22:41:49.167966 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:41:49.165867 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"82154028ecefce593fde43598fc4b12fbd1b782ae4d1b6d2ca7eae668766e19c"} err="failed to get container status \"82154028ecefce593fde43598fc4b12fbd1b782ae4d1b6d2ca7eae668766e19c\": rpc error: code = NotFound desc = could not find container \"82154028ecefce593fde43598fc4b12fbd1b782ae4d1b6d2ca7eae668766e19c\": container with ID starting with 82154028ecefce593fde43598fc4b12fbd1b782ae4d1b6d2ca7eae668766e19c not found: ID does not exist" Jul 09 22:41:49.169708 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:41:49.169673 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-fdfdaecd-openshift-default-76f945c577-j9p84"] Jul 09 22:41:49.209131 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:41:49.209094 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4a1a24ae-c92f-45dc-91f8-2abb2c3ea1f1-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-e08843d5-5jkb5\" (UID: \"4a1a24ae-c92f-45dc-91f8-2abb2c3ea1f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-e08843d5-5jkb5" Jul 09 22:41:49.209800 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:41:49.209779 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4a1a24ae-c92f-45dc-91f8-2abb2c3ea1f1-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-e08843d5-5jkb5\" (UID: \"4a1a24ae-c92f-45dc-91f8-2abb2c3ea1f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-e08843d5-5jkb5" Jul 09 22:41:49.413429 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:41:49.413330 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-e08843d5-5jkb5" Jul 09 22:41:49.598290 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:41:49.598261 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-e08843d5-5jkb5"] Jul 09 22:41:49.600389 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:41:49.600365 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4a1a24ae_c92f_45dc_91f8_2abb2c3ea1f1.slice/crio-644d0c50e8ff627b21bcaddb3764a4fb870f6f57ed9ade245beba907295bcbe3 WatchSource:0}: Error finding container 644d0c50e8ff627b21bcaddb3764a4fb870f6f57ed9ade245beba907295bcbe3: Status 404 returned error can't find the container with id 644d0c50e8ff627b21bcaddb3764a4fb870f6f57ed9ade245beba907295bcbe3 Jul 09 22:41:50.132714 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:41:50.132616 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-e08843d5-5jkb5" event={"ID":"4a1a24ae-c92f-45dc-91f8-2abb2c3ea1f1","Type":"ContainerStarted","Data":"55b29170ac3d008b9dc9e96dfa5cfaf73a4b77f3fab56f78cceba1c7b0945413"} Jul 09 22:41:50.132714 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:41:50.132677 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-e08843d5-5jkb5" event={"ID":"4a1a24ae-c92f-45dc-91f8-2abb2c3ea1f1","Type":"ContainerStarted","Data":"644d0c50e8ff627b21bcaddb3764a4fb870f6f57ed9ade245beba907295bcbe3"} Jul 09 22:41:50.151727 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:41:50.151638 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-e08843d5-5jkb5" podStartSLOduration=1.151604294 podStartE2EDuration="1.151604294s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 22:41:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 22:41:50.149220183 +0000 UTC m=+1902.060672883" watchObservedRunningTime="2026-07-09 22:41:50.151604294 +0000 UTC m=+1902.063056994" Jul 09 22:41:50.568837 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:41:50.568792 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c9c62156-984b-493c-95b9-ab4b5e6a9f98" path="/var/lib/kubelet/pods/c9c62156-984b-493c-95b9-ab4b5e6a9f98/volumes" Jul 09 22:41:51.139477 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:41:51.139440 2579 generic.go:356] "Generic (PLEG): container finished" podID="4a1a24ae-c92f-45dc-91f8-2abb2c3ea1f1" containerID="55b29170ac3d008b9dc9e96dfa5cfaf73a4b77f3fab56f78cceba1c7b0945413" exitCode=7 Jul 09 22:41:51.139984 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:41:51.139527 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-e08843d5-5jkb5" event={"ID":"4a1a24ae-c92f-45dc-91f8-2abb2c3ea1f1","Type":"ContainerDied","Data":"55b29170ac3d008b9dc9e96dfa5cfaf73a4b77f3fab56f78cceba1c7b0945413"} Jul 09 22:41:51.139984 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:41:51.139746 2579 scope.go:117] "RemoveContainer" containerID="55b29170ac3d008b9dc9e96dfa5cfaf73a4b77f3fab56f78cceba1c7b0945413" Jul 09 22:41:52.149578 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:41:52.148671 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-e08843d5-5jkb5" event={"ID":"4a1a24ae-c92f-45dc-91f8-2abb2c3ea1f1","Type":"ContainerStarted","Data":"13ef2b6187d485eae6bd99114d4e73a5d779d17130e84b4618079c9805369abd"} Jul 09 22:41:53.157261 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:41:53.157164 2579 generic.go:356] "Generic (PLEG): container finished" podID="4a1a24ae-c92f-45dc-91f8-2abb2c3ea1f1" containerID="13ef2b6187d485eae6bd99114d4e73a5d779d17130e84b4618079c9805369abd" exitCode=7 Jul 09 22:41:53.157261 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:41:53.157239 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-e08843d5-5jkb5" event={"ID":"4a1a24ae-c92f-45dc-91f8-2abb2c3ea1f1","Type":"ContainerDied","Data":"13ef2b6187d485eae6bd99114d4e73a5d779d17130e84b4618079c9805369abd"} Jul 09 22:41:53.157851 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:41:53.157297 2579 scope.go:117] "RemoveContainer" containerID="55b29170ac3d008b9dc9e96dfa5cfaf73a4b77f3fab56f78cceba1c7b0945413" Jul 09 22:41:53.157851 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:41:53.157483 2579 scope.go:117] "RemoveContainer" containerID="13ef2b6187d485eae6bd99114d4e73a5d779d17130e84b4618079c9805369abd" Jul 09 22:41:53.157851 ip-10-0-136-251 kubenswrapper[2579]: E0709 22:41:53.157758 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-e08843d5-5jkb5_odh-ai-gateway-infra(4a1a24ae-c92f-45dc-91f8-2abb2c3ea1f1)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-e08843d5-5jkb5" podUID="4a1a24ae-c92f-45dc-91f8-2abb2c3ea1f1" Jul 09 22:41:54.165092 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:41:54.165060 2579 scope.go:117] "RemoveContainer" containerID="13ef2b6187d485eae6bd99114d4e73a5d779d17130e84b4618079c9805369abd" Jul 09 22:41:54.165598 ip-10-0-136-251 kubenswrapper[2579]: E0709 22:41:54.165242 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-e08843d5-5jkb5_odh-ai-gateway-infra(4a1a24ae-c92f-45dc-91f8-2abb2c3ea1f1)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-e08843d5-5jkb5" podUID="4a1a24ae-c92f-45dc-91f8-2abb2c3ea1f1" Jul 09 22:41:56.176606 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:41:56.176573 2579 generic.go:356] "Generic (PLEG): container finished" podID="032933bd-19ac-49f0-a484-62c78b1bfc7e" containerID="42b5296eab5b632399c7c565157c5958b842bb0fe2b55e7d6a71456b4e39ab54" exitCode=0 Jul 09 22:41:56.177074 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:41:56.176651 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-e08843d5-openshift-default-864445fbc9-tqhpv" event={"ID":"032933bd-19ac-49f0-a484-62c78b1bfc7e","Type":"ContainerDied","Data":"42b5296eab5b632399c7c565157c5958b842bb0fe2b55e7d6a71456b4e39ab54"} Jul 09 22:41:56.177129 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:41:56.177079 2579 scope.go:117] "RemoveContainer" containerID="42b5296eab5b632399c7c565157c5958b842bb0fe2b55e7d6a71456b4e39ab54" Jul 09 22:41:56.178030 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:41:56.178006 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 09 22:41:56.178092 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:41:56.178078 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 09 22:41:56.178137 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:41:56.178113 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 09 22:41:57.184895 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:41:57.184862 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-e08843d5-openshift-default-864445fbc9-tqhpv" event={"ID":"032933bd-19ac-49f0-a484-62c78b1bfc7e","Type":"ContainerStarted","Data":"3d8cdc691c7f80f32aac8d0a33b5863fdad23eb754ca6724ce97baa2c111d41b"} Jul 09 22:41:57.881792 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:41:57.881753 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-e08843d5-openshift-default-864445fbc9-tqhpv" Jul 09 22:41:57.883090 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:41:57.883061 2579 patch_prober.go:28] interesting pod/e2e-derive-e08843d5-openshift-default-864445fbc9-tqhpv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.39:15021/healthz/ready\": dial tcp 10.132.0.39:15021: connect: connection refused" start-of-body= Jul 09 22:41:57.883218 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:41:57.883117 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-e08843d5-openshift-default-864445fbc9-tqhpv" podUID="032933bd-19ac-49f0-a484-62c78b1bfc7e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.39:15021/healthz/ready\": dial tcp 10.132.0.39:15021: connect: connection refused" Jul 09 22:41:58.882012 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:41:58.881967 2579 patch_prober.go:28] interesting pod/e2e-derive-e08843d5-openshift-default-864445fbc9-tqhpv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.39:15021/healthz/ready\": dial tcp 10.132.0.39:15021: connect: connection refused" start-of-body= Jul 09 22:41:58.882441 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:41:58.882049 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-e08843d5-openshift-default-864445fbc9-tqhpv" podUID="032933bd-19ac-49f0-a484-62c78b1bfc7e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.39:15021/healthz/ready\": dial tcp 10.132.0.39:15021: connect: connection refused" Jul 09 22:41:59.882009 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:41:59.881967 2579 patch_prober.go:28] interesting pod/e2e-derive-e08843d5-openshift-default-864445fbc9-tqhpv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.39:15021/healthz/ready\": dial tcp 10.132.0.39:15021: connect: connection refused" start-of-body= Jul 09 22:41:59.882211 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:41:59.882056 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-e08843d5-openshift-default-864445fbc9-tqhpv" podUID="032933bd-19ac-49f0-a484-62c78b1bfc7e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.39:15021/healthz/ready\": dial tcp 10.132.0.39:15021: connect: connection refused" Jul 09 22:42:00.881565 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:42:00.881534 2579 patch_prober.go:28] interesting pod/e2e-derive-e08843d5-openshift-default-864445fbc9-tqhpv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.39:15021/healthz/ready\": dial tcp 10.132.0.39:15021: connect: connection refused" start-of-body= Jul 09 22:42:00.881899 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:42:00.881638 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-e08843d5-openshift-default-864445fbc9-tqhpv" podUID="032933bd-19ac-49f0-a484-62c78b1bfc7e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.39:15021/healthz/ready\": dial tcp 10.132.0.39:15021: connect: connection refused" Jul 09 22:42:01.108036 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:42:01.107989 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-e08843d5-openshift-default-864445fbc9-tqhpv" Jul 09 22:42:01.885504 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:42:01.885466 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-e08843d5-openshift-default-864445fbc9-tqhpv" Jul 09 22:42:01.886441 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:42:01.886419 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-e08843d5-openshift-default-864445fbc9-tqhpv" Jul 09 22:42:05.558820 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:42:05.558791 2579 scope.go:117] "RemoveContainer" containerID="13ef2b6187d485eae6bd99114d4e73a5d779d17130e84b4618079c9805369abd" Jul 09 22:42:06.233791 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:42:06.233700 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-e08843d5-5jkb5" event={"ID":"4a1a24ae-c92f-45dc-91f8-2abb2c3ea1f1","Type":"ContainerStarted","Data":"c75c9c19a5df4b5ba10b010bd2d9622dd3f193e8df38e80d0f19bf10a6c9efaa"} Jul 09 22:42:07.240826 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:42:07.240794 2579 generic.go:356] "Generic (PLEG): container finished" podID="4a1a24ae-c92f-45dc-91f8-2abb2c3ea1f1" containerID="c75c9c19a5df4b5ba10b010bd2d9622dd3f193e8df38e80d0f19bf10a6c9efaa" exitCode=0 Jul 09 22:42:07.241341 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:42:07.240878 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-e08843d5-5jkb5" event={"ID":"4a1a24ae-c92f-45dc-91f8-2abb2c3ea1f1","Type":"ContainerDied","Data":"c75c9c19a5df4b5ba10b010bd2d9622dd3f193e8df38e80d0f19bf10a6c9efaa"} Jul 09 22:42:07.241341 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:42:07.240967 2579 scope.go:117] "RemoveContainer" containerID="13ef2b6187d485eae6bd99114d4e73a5d779d17130e84b4618079c9805369abd" Jul 09 22:42:07.277807 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:42:07.277769 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-e08843d5-5jkb5"] Jul 09 22:42:08.382634 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:42:08.382608 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-e08843d5-5jkb5" Jul 09 22:42:08.489771 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:42:08.489733 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4a1a24ae-c92f-45dc-91f8-2abb2c3ea1f1-maas-api-service-ca\") pod \"4a1a24ae-c92f-45dc-91f8-2abb2c3ea1f1\" (UID: \"4a1a24ae-c92f-45dc-91f8-2abb2c3ea1f1\") " Jul 09 22:42:08.490202 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:42:08.490159 2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4a1a24ae-c92f-45dc-91f8-2abb2c3ea1f1-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "4a1a24ae-c92f-45dc-91f8-2abb2c3ea1f1" (UID: "4a1a24ae-c92f-45dc-91f8-2abb2c3ea1f1"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 09 22:42:08.591449 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:42:08.591411 2579 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4a1a24ae-c92f-45dc-91f8-2abb2c3ea1f1-maas-api-service-ca\") on node \"ip-10-0-136-251.ec2.internal\" DevicePath \"\"" Jul 09 22:42:09.252856 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:42:09.252821 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-e08843d5-5jkb5" Jul 09 22:42:09.253061 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:42:09.252856 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-e08843d5-5jkb5" event={"ID":"4a1a24ae-c92f-45dc-91f8-2abb2c3ea1f1","Type":"ContainerDied","Data":"644d0c50e8ff627b21bcaddb3764a4fb870f6f57ed9ade245beba907295bcbe3"} Jul 09 22:42:09.253061 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:42:09.252963 2579 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="644d0c50e8ff627b21bcaddb3764a4fb870f6f57ed9ade245beba907295bcbe3" Jul 09 22:42:09.282058 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:42:09.282017 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-e08843d5-5jkb5"] Jul 09 22:42:09.285875 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:42:09.285846 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-e08843d5-5jkb5"] Jul 09 22:42:10.565195 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:42:10.565157 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4a1a24ae-c92f-45dc-91f8-2abb2c3ea1f1" path="/var/lib/kubelet/pods/4a1a24ae-c92f-45dc-91f8-2abb2c3ea1f1/volumes" Jul 09 22:42:49.160031 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:42:49.159993 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-e08843d5-czkpv"] Jul 09 22:42:49.160534 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:42:49.160377 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4a1a24ae-c92f-45dc-91f8-2abb2c3ea1f1" containerName="revoke-keys" Jul 09 22:42:49.160534 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:42:49.160388 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="4a1a24ae-c92f-45dc-91f8-2abb2c3ea1f1" containerName="revoke-keys" Jul 09 22:42:49.160534 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:42:49.160401 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4a1a24ae-c92f-45dc-91f8-2abb2c3ea1f1" containerName="revoke-keys" Jul 09 22:42:49.160534 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:42:49.160406 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="4a1a24ae-c92f-45dc-91f8-2abb2c3ea1f1" containerName="revoke-keys" Jul 09 22:42:49.160534 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:42:49.160474 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="4a1a24ae-c92f-45dc-91f8-2abb2c3ea1f1" containerName="revoke-keys" Jul 09 22:42:49.160534 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:42:49.160484 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="4a1a24ae-c92f-45dc-91f8-2abb2c3ea1f1" containerName="revoke-keys" Jul 09 22:42:49.160534 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:42:49.160491 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="c9c62156-984b-493c-95b9-ab4b5e6a9f98" containerName="istio-proxy" Jul 09 22:42:49.170576 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:42:49.170540 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-e08843d5-czkpv" Jul 09 22:42:49.171975 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:42:49.171942 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-e08843d5-czkpv"] Jul 09 22:42:49.173475 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:42:49.173451 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fdzhc\"" Jul 09 22:42:49.173608 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:42:49.173450 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 09 22:42:49.217450 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:42:49.217415 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-e08843d5-openshift-default-864445fbc9-tqhpv"] Jul 09 22:42:49.217653 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:42:49.217617 2579 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-ingress/e2e-derive-e08843d5-openshift-default-864445fbc9-tqhpv" podUID="032933bd-19ac-49f0-a484-62c78b1bfc7e" containerName="istio-proxy" containerID="cri-o://3d8cdc691c7f80f32aac8d0a33b5863fdad23eb754ca6724ce97baa2c111d41b" gracePeriod=30 Jul 09 22:42:49.255365 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:42:49.255329 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d6bc0f38-299a-485a-8484-aafc7f04fca9-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-e08843d5-czkpv\" (UID: \"d6bc0f38-299a-485a-8484-aafc7f04fca9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-e08843d5-czkpv" Jul 09 22:42:49.356823 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:42:49.356789 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d6bc0f38-299a-485a-8484-aafc7f04fca9-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-e08843d5-czkpv\" (UID: \"d6bc0f38-299a-485a-8484-aafc7f04fca9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-e08843d5-czkpv" Jul 09 22:42:49.357450 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:42:49.357430 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d6bc0f38-299a-485a-8484-aafc7f04fca9-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-e08843d5-czkpv\" (UID: \"d6bc0f38-299a-485a-8484-aafc7f04fca9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-e08843d5-czkpv" Jul 09 22:42:49.481835 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:42:49.481743 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-e08843d5-czkpv" Jul 09 22:42:49.620242 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:42:49.620207 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-e08843d5-czkpv"] Jul 09 22:42:49.621547 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:42:49.621515 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd6bc0f38_299a_485a_8484_aafc7f04fca9.slice/crio-1b38a943bd92fa3486deccff173384f8db6d39b7b7f3ccc1478993b4da7622d5 WatchSource:0}: Error finding container 1b38a943bd92fa3486deccff173384f8db6d39b7b7f3ccc1478993b4da7622d5: Status 404 returned error can't find the container with id 1b38a943bd92fa3486deccff173384f8db6d39b7b7f3ccc1478993b4da7622d5 Jul 09 22:42:50.462025 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:42:50.461992 2579 generic.go:356] "Generic (PLEG): container finished" podID="d6bc0f38-299a-485a-8484-aafc7f04fca9" containerID="fa0ebe2f4455052c0b0d33fab730d9a63bcb10d7ee487d47f20b79637c733d7c" exitCode=0 Jul 09 22:42:50.462434 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:42:50.462033 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-e08843d5-czkpv" event={"ID":"d6bc0f38-299a-485a-8484-aafc7f04fca9","Type":"ContainerDied","Data":"fa0ebe2f4455052c0b0d33fab730d9a63bcb10d7ee487d47f20b79637c733d7c"} Jul 09 22:42:50.462434 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:42:50.462088 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-e08843d5-czkpv" event={"ID":"d6bc0f38-299a-485a-8484-aafc7f04fca9","Type":"ContainerStarted","Data":"1b38a943bd92fa3486deccff173384f8db6d39b7b7f3ccc1478993b4da7622d5"} Jul 09 22:42:51.602894 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:42:51.602865 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-e08843d5-czkpv" Jul 09 22:42:51.676714 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:42:51.676681 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d6bc0f38-299a-485a-8484-aafc7f04fca9-maas-api-service-ca\") pod \"d6bc0f38-299a-485a-8484-aafc7f04fca9\" (UID: \"d6bc0f38-299a-485a-8484-aafc7f04fca9\") " Jul 09 22:42:51.677033 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:42:51.677010 2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d6bc0f38-299a-485a-8484-aafc7f04fca9-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "d6bc0f38-299a-485a-8484-aafc7f04fca9" (UID: "d6bc0f38-299a-485a-8484-aafc7f04fca9"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 09 22:42:51.778264 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:42:51.778225 2579 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d6bc0f38-299a-485a-8484-aafc7f04fca9-maas-api-service-ca\") on node \"ip-10-0-136-251.ec2.internal\" DevicePath \"\"" Jul 09 22:42:52.316603 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:42:52.316568 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-xdfrr"] Jul 09 22:42:52.473887 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:42:52.473852 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-e08843d5-czkpv" event={"ID":"d6bc0f38-299a-485a-8484-aafc7f04fca9","Type":"ContainerDied","Data":"1b38a943bd92fa3486deccff173384f8db6d39b7b7f3ccc1478993b4da7622d5"} Jul 09 22:42:52.474088 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:42:52.473905 2579 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1b38a943bd92fa3486deccff173384f8db6d39b7b7f3ccc1478993b4da7622d5" Jul 09 22:42:52.474088 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:42:52.473860 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-e08843d5-czkpv" Jul 09 22:42:54.479209 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:42:54.479183 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-e08843d5-openshift-default-864445fbc9-tqhpv" Jul 09 22:42:54.485118 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:42:54.485036 2579 generic.go:356] "Generic (PLEG): container finished" podID="032933bd-19ac-49f0-a484-62c78b1bfc7e" containerID="3d8cdc691c7f80f32aac8d0a33b5863fdad23eb754ca6724ce97baa2c111d41b" exitCode=0 Jul 09 22:42:54.485118 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:42:54.485084 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-e08843d5-openshift-default-864445fbc9-tqhpv" Jul 09 22:42:54.485334 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:42:54.485114 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-e08843d5-openshift-default-864445fbc9-tqhpv" event={"ID":"032933bd-19ac-49f0-a484-62c78b1bfc7e","Type":"ContainerDied","Data":"3d8cdc691c7f80f32aac8d0a33b5863fdad23eb754ca6724ce97baa2c111d41b"} Jul 09 22:42:54.485334 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:42:54.485220 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-e08843d5-openshift-default-864445fbc9-tqhpv" event={"ID":"032933bd-19ac-49f0-a484-62c78b1bfc7e","Type":"ContainerDied","Data":"cbbb4faec2dbe8ecb8c32e2b749e9b5a4768564fb3a5f820b1c0dd56443ca4fe"} Jul 09 22:42:54.485334 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:42:54.485256 2579 scope.go:117] "RemoveContainer" containerID="3d8cdc691c7f80f32aac8d0a33b5863fdad23eb754ca6724ce97baa2c111d41b" Jul 09 22:42:54.494256 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:42:54.494235 2579 scope.go:117] "RemoveContainer" containerID="42b5296eab5b632399c7c565157c5958b842bb0fe2b55e7d6a71456b4e39ab54" Jul 09 22:42:54.503053 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:42:54.503028 2579 scope.go:117] "RemoveContainer" containerID="3d8cdc691c7f80f32aac8d0a33b5863fdad23eb754ca6724ce97baa2c111d41b" Jul 09 22:42:54.503446 ip-10-0-136-251 kubenswrapper[2579]: E0709 22:42:54.503405 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3d8cdc691c7f80f32aac8d0a33b5863fdad23eb754ca6724ce97baa2c111d41b\": container with ID starting with 3d8cdc691c7f80f32aac8d0a33b5863fdad23eb754ca6724ce97baa2c111d41b not found: ID does not exist" containerID="3d8cdc691c7f80f32aac8d0a33b5863fdad23eb754ca6724ce97baa2c111d41b" Jul 09 22:42:54.503577 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:42:54.503451 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3d8cdc691c7f80f32aac8d0a33b5863fdad23eb754ca6724ce97baa2c111d41b"} err="failed to get container status \"3d8cdc691c7f80f32aac8d0a33b5863fdad23eb754ca6724ce97baa2c111d41b\": rpc error: code = NotFound desc = could not find container \"3d8cdc691c7f80f32aac8d0a33b5863fdad23eb754ca6724ce97baa2c111d41b\": container with ID starting with 3d8cdc691c7f80f32aac8d0a33b5863fdad23eb754ca6724ce97baa2c111d41b not found: ID does not exist" Jul 09 22:42:54.503577 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:42:54.503476 2579 scope.go:117] "RemoveContainer" containerID="42b5296eab5b632399c7c565157c5958b842bb0fe2b55e7d6a71456b4e39ab54" Jul 09 22:42:54.503826 ip-10-0-136-251 kubenswrapper[2579]: E0709 22:42:54.503799 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"42b5296eab5b632399c7c565157c5958b842bb0fe2b55e7d6a71456b4e39ab54\": container with ID starting with 42b5296eab5b632399c7c565157c5958b842bb0fe2b55e7d6a71456b4e39ab54 not found: ID does not exist" containerID="42b5296eab5b632399c7c565157c5958b842bb0fe2b55e7d6a71456b4e39ab54" Jul 09 22:42:54.503951 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:42:54.503836 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"42b5296eab5b632399c7c565157c5958b842bb0fe2b55e7d6a71456b4e39ab54"} err="failed to get container status \"42b5296eab5b632399c7c565157c5958b842bb0fe2b55e7d6a71456b4e39ab54\": rpc error: code = NotFound desc = could not find container \"42b5296eab5b632399c7c565157c5958b842bb0fe2b55e7d6a71456b4e39ab54\": container with ID starting with 42b5296eab5b632399c7c565157c5958b842bb0fe2b55e7d6a71456b4e39ab54 not found: ID does not exist" Jul 09 22:42:54.602074 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:42:54.602043 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/032933bd-19ac-49f0-a484-62c78b1bfc7e-workload-certs\") pod \"032933bd-19ac-49f0-a484-62c78b1bfc7e\" (UID: \"032933bd-19ac-49f0-a484-62c78b1bfc7e\") " Jul 09 22:42:54.602074 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:42:54.602082 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/032933bd-19ac-49f0-a484-62c78b1bfc7e-credential-socket\") pod \"032933bd-19ac-49f0-a484-62c78b1bfc7e\" (UID: \"032933bd-19ac-49f0-a484-62c78b1bfc7e\") " Jul 09 22:42:54.602338 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:42:54.602123 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/032933bd-19ac-49f0-a484-62c78b1bfc7e-istio-podinfo\") pod \"032933bd-19ac-49f0-a484-62c78b1bfc7e\" (UID: \"032933bd-19ac-49f0-a484-62c78b1bfc7e\") " Jul 09 22:42:54.602338 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:42:54.602167 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/032933bd-19ac-49f0-a484-62c78b1bfc7e-istio-data\") pod \"032933bd-19ac-49f0-a484-62c78b1bfc7e\" (UID: \"032933bd-19ac-49f0-a484-62c78b1bfc7e\") " Jul 09 22:42:54.602338 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:42:54.602202 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/032933bd-19ac-49f0-a484-62c78b1bfc7e-istiod-ca-cert\") pod \"032933bd-19ac-49f0-a484-62c78b1bfc7e\" (UID: \"032933bd-19ac-49f0-a484-62c78b1bfc7e\") " Jul 09 22:42:54.602338 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:42:54.602218 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/032933bd-19ac-49f0-a484-62c78b1bfc7e-istio-envoy\") pod \"032933bd-19ac-49f0-a484-62c78b1bfc7e\" (UID: \"032933bd-19ac-49f0-a484-62c78b1bfc7e\") " Jul 09 22:42:54.602551 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:42:54.602361 2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/032933bd-19ac-49f0-a484-62c78b1bfc7e-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "032933bd-19ac-49f0-a484-62c78b1bfc7e" (UID: "032933bd-19ac-49f0-a484-62c78b1bfc7e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 22:42:54.602551 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:42:54.602401 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/032933bd-19ac-49f0-a484-62c78b1bfc7e-workload-socket\") pod \"032933bd-19ac-49f0-a484-62c78b1bfc7e\" (UID: \"032933bd-19ac-49f0-a484-62c78b1bfc7e\") " Jul 09 22:42:54.602551 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:42:54.602435 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nd24l\" (UniqueName: \"kubernetes.io/projected/032933bd-19ac-49f0-a484-62c78b1bfc7e-kube-api-access-nd24l\") pod \"032933bd-19ac-49f0-a484-62c78b1bfc7e\" (UID: \"032933bd-19ac-49f0-a484-62c78b1bfc7e\") " Jul 09 22:42:54.602551 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:42:54.602472 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/032933bd-19ac-49f0-a484-62c78b1bfc7e-istio-token\") pod \"032933bd-19ac-49f0-a484-62c78b1bfc7e\" (UID: \"032933bd-19ac-49f0-a484-62c78b1bfc7e\") " Jul 09 22:42:54.602551 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:42:54.602501 2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/032933bd-19ac-49f0-a484-62c78b1bfc7e-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "032933bd-19ac-49f0-a484-62c78b1bfc7e" (UID: "032933bd-19ac-49f0-a484-62c78b1bfc7e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 22:42:54.602822 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:42:54.602713 2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/032933bd-19ac-49f0-a484-62c78b1bfc7e-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "032933bd-19ac-49f0-a484-62c78b1bfc7e" (UID: "032933bd-19ac-49f0-a484-62c78b1bfc7e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 09 22:42:54.602890 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:42:54.602848 2579 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/032933bd-19ac-49f0-a484-62c78b1bfc7e-istiod-ca-cert\") on node \"ip-10-0-136-251.ec2.internal\" DevicePath \"\"" Jul 09 22:42:54.602890 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:42:54.602868 2579 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/032933bd-19ac-49f0-a484-62c78b1bfc7e-workload-certs\") on node \"ip-10-0-136-251.ec2.internal\" DevicePath \"\"" Jul 09 22:42:54.602890 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:42:54.602882 2579 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/032933bd-19ac-49f0-a484-62c78b1bfc7e-credential-socket\") on node \"ip-10-0-136-251.ec2.internal\" DevicePath \"\"" Jul 09 22:42:54.602890 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:42:54.602883 2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/032933bd-19ac-49f0-a484-62c78b1bfc7e-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "032933bd-19ac-49f0-a484-62c78b1bfc7e" (UID: "032933bd-19ac-49f0-a484-62c78b1bfc7e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 22:42:54.603166 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:42:54.603124 2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/032933bd-19ac-49f0-a484-62c78b1bfc7e-istio-data" (OuterVolumeSpecName: "istio-data") pod "032933bd-19ac-49f0-a484-62c78b1bfc7e" (UID: "032933bd-19ac-49f0-a484-62c78b1bfc7e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 22:42:54.604566 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:42:54.604538 2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/032933bd-19ac-49f0-a484-62c78b1bfc7e-istio-token" (OuterVolumeSpecName: "istio-token") pod "032933bd-19ac-49f0-a484-62c78b1bfc7e" (UID: "032933bd-19ac-49f0-a484-62c78b1bfc7e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 22:42:54.604669 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:42:54.604588 2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/032933bd-19ac-49f0-a484-62c78b1bfc7e-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "032933bd-19ac-49f0-a484-62c78b1bfc7e" (UID: "032933bd-19ac-49f0-a484-62c78b1bfc7e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 22:42:54.604792 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:42:54.604777 2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/032933bd-19ac-49f0-a484-62c78b1bfc7e-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "032933bd-19ac-49f0-a484-62c78b1bfc7e" (UID: "032933bd-19ac-49f0-a484-62c78b1bfc7e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 09 22:42:54.604981 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:42:54.604963 2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/032933bd-19ac-49f0-a484-62c78b1bfc7e-kube-api-access-nd24l" (OuterVolumeSpecName: "kube-api-access-nd24l") pod "032933bd-19ac-49f0-a484-62c78b1bfc7e" (UID: "032933bd-19ac-49f0-a484-62c78b1bfc7e"). InnerVolumeSpecName "kube-api-access-nd24l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 22:42:54.703829 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:42:54.703787 2579 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/032933bd-19ac-49f0-a484-62c78b1bfc7e-istio-podinfo\") on node \"ip-10-0-136-251.ec2.internal\" DevicePath \"\"" Jul 09 22:42:54.703829 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:42:54.703822 2579 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/032933bd-19ac-49f0-a484-62c78b1bfc7e-istio-data\") on node \"ip-10-0-136-251.ec2.internal\" DevicePath \"\"" Jul 09 22:42:54.703829 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:42:54.703832 2579 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/032933bd-19ac-49f0-a484-62c78b1bfc7e-istio-envoy\") on node \"ip-10-0-136-251.ec2.internal\" DevicePath \"\"" Jul 09 22:42:54.703829 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:42:54.703840 2579 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/032933bd-19ac-49f0-a484-62c78b1bfc7e-workload-socket\") on node \"ip-10-0-136-251.ec2.internal\" DevicePath \"\"" Jul 09 22:42:54.704144 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:42:54.703849 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nd24l\" (UniqueName: \"kubernetes.io/projected/032933bd-19ac-49f0-a484-62c78b1bfc7e-kube-api-access-nd24l\") on node \"ip-10-0-136-251.ec2.internal\" DevicePath \"\"" Jul 09 22:42:54.704144 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:42:54.703858 2579 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/032933bd-19ac-49f0-a484-62c78b1bfc7e-istio-token\") on node \"ip-10-0-136-251.ec2.internal\" DevicePath \"\"" Jul 09 22:42:54.807737 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:42:54.807705 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-e08843d5-openshift-default-864445fbc9-tqhpv"] Jul 09 22:42:54.813633 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:42:54.813602 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-derive-e08843d5-openshift-default-864445fbc9-tqhpv"] Jul 09 22:42:56.564289 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:42:56.564255 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="032933bd-19ac-49f0-a484-62c78b1bfc7e" path="/var/lib/kubelet/pods/032933bd-19ac-49f0-a484-62c78b1bfc7e/volumes" Jul 09 22:44:25.312414 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:44:25.312377 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-xdfrr"] Jul 09 22:44:32.819542 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:44:32.819454 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-xdfrr"] Jul 09 22:44:38.940589 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:44:38.940556 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-67cb547f-openshift-default-5d9648c9ff-5dwtm"] Jul 09 22:44:38.940993 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:44:38.940940 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d6bc0f38-299a-485a-8484-aafc7f04fca9" containerName="revoke-keys" Jul 09 22:44:38.940993 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:44:38.940951 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="d6bc0f38-299a-485a-8484-aafc7f04fca9" containerName="revoke-keys" Jul 09 22:44:38.940993 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:44:38.940968 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="032933bd-19ac-49f0-a484-62c78b1bfc7e" containerName="istio-proxy" Jul 09 22:44:38.940993 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:44:38.940973 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="032933bd-19ac-49f0-a484-62c78b1bfc7e" containerName="istio-proxy" Jul 09 22:44:38.940993 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:44:38.940988 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="032933bd-19ac-49f0-a484-62c78b1bfc7e" containerName="istio-proxy" Jul 09 22:44:38.940993 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:44:38.940994 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="032933bd-19ac-49f0-a484-62c78b1bfc7e" containerName="istio-proxy" Jul 09 22:44:38.941190 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:44:38.941007 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4a1a24ae-c92f-45dc-91f8-2abb2c3ea1f1" containerName="revoke-keys" Jul 09 22:44:38.941190 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:44:38.941012 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="4a1a24ae-c92f-45dc-91f8-2abb2c3ea1f1" containerName="revoke-keys" Jul 09 22:44:38.941190 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:44:38.941070 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="4a1a24ae-c92f-45dc-91f8-2abb2c3ea1f1" containerName="revoke-keys" Jul 09 22:44:38.941190 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:44:38.941079 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="032933bd-19ac-49f0-a484-62c78b1bfc7e" containerName="istio-proxy" Jul 09 22:44:38.941190 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:44:38.941088 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="d6bc0f38-299a-485a-8484-aafc7f04fca9" containerName="revoke-keys" Jul 09 22:44:38.944271 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:44:38.944248 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-67cb547f-openshift-default-5d9648c9ff-5dwtm" Jul 09 22:44:38.947069 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:44:38.947047 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-67cb547f-openshift-default-dockercfg-q29cb\"" Jul 09 22:44:38.968850 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:44:38.968818 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-67cb547f-openshift-default-5d9648c9ff-5dwtm"] Jul 09 22:44:39.023591 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:44:39.023556 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/018127de-0b90-4c23-932d-dd1291e30903-istio-token\") pod \"e2e-mt-67cb547f-openshift-default-5d9648c9ff-5dwtm\" (UID: \"018127de-0b90-4c23-932d-dd1291e30903\") " pod="openshift-ingress/e2e-mt-67cb547f-openshift-default-5d9648c9ff-5dwtm" Jul 09 22:44:39.023591 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:44:39.023592 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/018127de-0b90-4c23-932d-dd1291e30903-istiod-ca-cert\") pod \"e2e-mt-67cb547f-openshift-default-5d9648c9ff-5dwtm\" (UID: \"018127de-0b90-4c23-932d-dd1291e30903\") " pod="openshift-ingress/e2e-mt-67cb547f-openshift-default-5d9648c9ff-5dwtm" Jul 09 22:44:39.023841 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:44:39.023613 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jnzx7\" (UniqueName: \"kubernetes.io/projected/018127de-0b90-4c23-932d-dd1291e30903-kube-api-access-jnzx7\") pod \"e2e-mt-67cb547f-openshift-default-5d9648c9ff-5dwtm\" (UID: \"018127de-0b90-4c23-932d-dd1291e30903\") " pod="openshift-ingress/e2e-mt-67cb547f-openshift-default-5d9648c9ff-5dwtm" Jul 09 22:44:39.023841 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:44:39.023758 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/018127de-0b90-4c23-932d-dd1291e30903-workload-socket\") pod \"e2e-mt-67cb547f-openshift-default-5d9648c9ff-5dwtm\" (UID: \"018127de-0b90-4c23-932d-dd1291e30903\") " pod="openshift-ingress/e2e-mt-67cb547f-openshift-default-5d9648c9ff-5dwtm" Jul 09 22:44:39.023841 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:44:39.023824 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/018127de-0b90-4c23-932d-dd1291e30903-credential-socket\") pod \"e2e-mt-67cb547f-openshift-default-5d9648c9ff-5dwtm\" (UID: \"018127de-0b90-4c23-932d-dd1291e30903\") " pod="openshift-ingress/e2e-mt-67cb547f-openshift-default-5d9648c9ff-5dwtm" Jul 09 22:44:39.023988 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:44:39.023853 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/018127de-0b90-4c23-932d-dd1291e30903-istio-data\") pod \"e2e-mt-67cb547f-openshift-default-5d9648c9ff-5dwtm\" (UID: \"018127de-0b90-4c23-932d-dd1291e30903\") " pod="openshift-ingress/e2e-mt-67cb547f-openshift-default-5d9648c9ff-5dwtm" Jul 09 22:44:39.023988 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:44:39.023948 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/018127de-0b90-4c23-932d-dd1291e30903-workload-certs\") pod \"e2e-mt-67cb547f-openshift-default-5d9648c9ff-5dwtm\" (UID: \"018127de-0b90-4c23-932d-dd1291e30903\") " pod="openshift-ingress/e2e-mt-67cb547f-openshift-default-5d9648c9ff-5dwtm" Jul 09 22:44:39.023988 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:44:39.023979 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/018127de-0b90-4c23-932d-dd1291e30903-istio-envoy\") pod \"e2e-mt-67cb547f-openshift-default-5d9648c9ff-5dwtm\" (UID: \"018127de-0b90-4c23-932d-dd1291e30903\") " pod="openshift-ingress/e2e-mt-67cb547f-openshift-default-5d9648c9ff-5dwtm" Jul 09 22:44:39.024106 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:44:39.024013 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/018127de-0b90-4c23-932d-dd1291e30903-istio-podinfo\") pod \"e2e-mt-67cb547f-openshift-default-5d9648c9ff-5dwtm\" (UID: \"018127de-0b90-4c23-932d-dd1291e30903\") " pod="openshift-ingress/e2e-mt-67cb547f-openshift-default-5d9648c9ff-5dwtm" Jul 09 22:44:39.125222 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:44:39.125174 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/018127de-0b90-4c23-932d-dd1291e30903-workload-certs\") pod \"e2e-mt-67cb547f-openshift-default-5d9648c9ff-5dwtm\" (UID: \"018127de-0b90-4c23-932d-dd1291e30903\") " pod="openshift-ingress/e2e-mt-67cb547f-openshift-default-5d9648c9ff-5dwtm" Jul 09 22:44:39.125222 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:44:39.125213 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/018127de-0b90-4c23-932d-dd1291e30903-istio-envoy\") pod \"e2e-mt-67cb547f-openshift-default-5d9648c9ff-5dwtm\" (UID: \"018127de-0b90-4c23-932d-dd1291e30903\") " pod="openshift-ingress/e2e-mt-67cb547f-openshift-default-5d9648c9ff-5dwtm" Jul 09 22:44:39.125490 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:44:39.125242 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/018127de-0b90-4c23-932d-dd1291e30903-istio-podinfo\") pod \"e2e-mt-67cb547f-openshift-default-5d9648c9ff-5dwtm\" (UID: \"018127de-0b90-4c23-932d-dd1291e30903\") " pod="openshift-ingress/e2e-mt-67cb547f-openshift-default-5d9648c9ff-5dwtm" Jul 09 22:44:39.125490 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:44:39.125265 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/018127de-0b90-4c23-932d-dd1291e30903-istio-token\") pod \"e2e-mt-67cb547f-openshift-default-5d9648c9ff-5dwtm\" (UID: \"018127de-0b90-4c23-932d-dd1291e30903\") " pod="openshift-ingress/e2e-mt-67cb547f-openshift-default-5d9648c9ff-5dwtm" Jul 09 22:44:39.125490 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:44:39.125284 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/018127de-0b90-4c23-932d-dd1291e30903-istiod-ca-cert\") pod \"e2e-mt-67cb547f-openshift-default-5d9648c9ff-5dwtm\" (UID: \"018127de-0b90-4c23-932d-dd1291e30903\") " pod="openshift-ingress/e2e-mt-67cb547f-openshift-default-5d9648c9ff-5dwtm" Jul 09 22:44:39.125490 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:44:39.125300 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jnzx7\" (UniqueName: \"kubernetes.io/projected/018127de-0b90-4c23-932d-dd1291e30903-kube-api-access-jnzx7\") pod \"e2e-mt-67cb547f-openshift-default-5d9648c9ff-5dwtm\" (UID: \"018127de-0b90-4c23-932d-dd1291e30903\") " pod="openshift-ingress/e2e-mt-67cb547f-openshift-default-5d9648c9ff-5dwtm" Jul 09 22:44:39.125490 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:44:39.125348 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/018127de-0b90-4c23-932d-dd1291e30903-workload-socket\") pod \"e2e-mt-67cb547f-openshift-default-5d9648c9ff-5dwtm\" (UID: \"018127de-0b90-4c23-932d-dd1291e30903\") " pod="openshift-ingress/e2e-mt-67cb547f-openshift-default-5d9648c9ff-5dwtm" Jul 09 22:44:39.125490 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:44:39.125379 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/018127de-0b90-4c23-932d-dd1291e30903-credential-socket\") pod \"e2e-mt-67cb547f-openshift-default-5d9648c9ff-5dwtm\" (UID: \"018127de-0b90-4c23-932d-dd1291e30903\") " pod="openshift-ingress/e2e-mt-67cb547f-openshift-default-5d9648c9ff-5dwtm" Jul 09 22:44:39.125490 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:44:39.125401 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/018127de-0b90-4c23-932d-dd1291e30903-istio-data\") pod \"e2e-mt-67cb547f-openshift-default-5d9648c9ff-5dwtm\" (UID: \"018127de-0b90-4c23-932d-dd1291e30903\") " pod="openshift-ingress/e2e-mt-67cb547f-openshift-default-5d9648c9ff-5dwtm" Jul 09 22:44:39.125831 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:44:39.125753 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/018127de-0b90-4c23-932d-dd1291e30903-credential-socket\") pod \"e2e-mt-67cb547f-openshift-default-5d9648c9ff-5dwtm\" (UID: \"018127de-0b90-4c23-932d-dd1291e30903\") " pod="openshift-ingress/e2e-mt-67cb547f-openshift-default-5d9648c9ff-5dwtm" Jul 09 22:44:39.125831 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:44:39.125780 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/018127de-0b90-4c23-932d-dd1291e30903-workload-socket\") pod \"e2e-mt-67cb547f-openshift-default-5d9648c9ff-5dwtm\" (UID: \"018127de-0b90-4c23-932d-dd1291e30903\") " pod="openshift-ingress/e2e-mt-67cb547f-openshift-default-5d9648c9ff-5dwtm" Jul 09 22:44:39.125987 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:44:39.125851 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/018127de-0b90-4c23-932d-dd1291e30903-istio-data\") pod \"e2e-mt-67cb547f-openshift-default-5d9648c9ff-5dwtm\" (UID: \"018127de-0b90-4c23-932d-dd1291e30903\") " pod="openshift-ingress/e2e-mt-67cb547f-openshift-default-5d9648c9ff-5dwtm" Jul 09 22:44:39.125987 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:44:39.125864 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/018127de-0b90-4c23-932d-dd1291e30903-workload-certs\") pod \"e2e-mt-67cb547f-openshift-default-5d9648c9ff-5dwtm\" (UID: \"018127de-0b90-4c23-932d-dd1291e30903\") " pod="openshift-ingress/e2e-mt-67cb547f-openshift-default-5d9648c9ff-5dwtm" Jul 09 22:44:39.126217 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:44:39.126196 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/018127de-0b90-4c23-932d-dd1291e30903-istiod-ca-cert\") pod \"e2e-mt-67cb547f-openshift-default-5d9648c9ff-5dwtm\" (UID: \"018127de-0b90-4c23-932d-dd1291e30903\") " pod="openshift-ingress/e2e-mt-67cb547f-openshift-default-5d9648c9ff-5dwtm" Jul 09 22:44:39.128241 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:44:39.128212 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/018127de-0b90-4c23-932d-dd1291e30903-istio-envoy\") pod \"e2e-mt-67cb547f-openshift-default-5d9648c9ff-5dwtm\" (UID: \"018127de-0b90-4c23-932d-dd1291e30903\") " pod="openshift-ingress/e2e-mt-67cb547f-openshift-default-5d9648c9ff-5dwtm" Jul 09 22:44:39.128378 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:44:39.128361 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/018127de-0b90-4c23-932d-dd1291e30903-istio-podinfo\") pod \"e2e-mt-67cb547f-openshift-default-5d9648c9ff-5dwtm\" (UID: \"018127de-0b90-4c23-932d-dd1291e30903\") " pod="openshift-ingress/e2e-mt-67cb547f-openshift-default-5d9648c9ff-5dwtm" Jul 09 22:44:39.133872 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:44:39.133845 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jnzx7\" (UniqueName: \"kubernetes.io/projected/018127de-0b90-4c23-932d-dd1291e30903-kube-api-access-jnzx7\") pod \"e2e-mt-67cb547f-openshift-default-5d9648c9ff-5dwtm\" (UID: \"018127de-0b90-4c23-932d-dd1291e30903\") " pod="openshift-ingress/e2e-mt-67cb547f-openshift-default-5d9648c9ff-5dwtm" Jul 09 22:44:39.133872 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:44:39.133860 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/018127de-0b90-4c23-932d-dd1291e30903-istio-token\") pod \"e2e-mt-67cb547f-openshift-default-5d9648c9ff-5dwtm\" (UID: \"018127de-0b90-4c23-932d-dd1291e30903\") " pod="openshift-ingress/e2e-mt-67cb547f-openshift-default-5d9648c9ff-5dwtm" Jul 09 22:44:39.255668 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:44:39.255631 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-67cb547f-openshift-default-5d9648c9ff-5dwtm" Jul 09 22:44:39.398782 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:44:39.398753 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-67cb547f-openshift-default-5d9648c9ff-5dwtm"] Jul 09 22:44:39.401099 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:44:39.401065 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod018127de_0b90_4c23_932d_dd1291e30903.slice/crio-f837c750c37c6f9f07ff4bdb86f6de27703fcd420c2e0ca6255237ec80147f82 WatchSource:0}: Error finding container f837c750c37c6f9f07ff4bdb86f6de27703fcd420c2e0ca6255237ec80147f82: Status 404 returned error can't find the container with id f837c750c37c6f9f07ff4bdb86f6de27703fcd420c2e0ca6255237ec80147f82 Jul 09 22:44:39.404037 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:44:39.403999 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 09 22:44:39.404183 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:44:39.404088 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 09 22:44:39.404183 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:44:39.404129 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 09 22:44:40.037746 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:44:40.037707 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-67cb547f-openshift-default-5d9648c9ff-5dwtm" event={"ID":"018127de-0b90-4c23-932d-dd1291e30903","Type":"ContainerStarted","Data":"f29d6b76bbb4f5ad77fefb36072ef31d6ac5aa873addb118ca2f30d88a5b5ec0"} Jul 09 22:44:40.038175 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:44:40.037773 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-67cb547f-openshift-default-5d9648c9ff-5dwtm" event={"ID":"018127de-0b90-4c23-932d-dd1291e30903","Type":"ContainerStarted","Data":"f837c750c37c6f9f07ff4bdb86f6de27703fcd420c2e0ca6255237ec80147f82"} Jul 09 22:44:40.074143 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:44:40.074077 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-67cb547f-openshift-default-5d9648c9ff-5dwtm" podStartSLOduration=2.074054743 podStartE2EDuration="2.074054743s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 22:44:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 22:44:40.07400254 +0000 UTC m=+2071.985455265" watchObservedRunningTime="2026-07-09 22:44:40.074054743 +0000 UTC m=+2071.985507444" Jul 09 22:44:40.255984 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:44:40.255956 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-67cb547f-openshift-default-5d9648c9ff-5dwtm" Jul 09 22:44:40.260907 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:44:40.260882 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-67cb547f-openshift-default-5d9648c9ff-5dwtm" Jul 09 22:44:41.042631 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:44:41.042591 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-67cb547f-openshift-default-5d9648c9ff-5dwtm" Jul 09 22:44:41.043884 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:44:41.043853 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-67cb547f-openshift-default-5d9648c9ff-5dwtm" Jul 09 22:44:41.621343 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:44:41.621305 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-xdfrr"] Jul 09 22:44:51.607883 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:44:51.607844 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-xdfrr"] Jul 09 22:44:54.154836 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:44:54.154804 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-67cb547f-q8dqt"] Jul 09 22:44:54.155275 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:44:54.155261 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="032933bd-19ac-49f0-a484-62c78b1bfc7e" containerName="istio-proxy" Jul 09 22:44:54.159143 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:44:54.159124 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-67cb547f-q8dqt" Jul 09 22:44:54.161678 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:44:54.161653 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 09 22:44:54.161799 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:44:54.161701 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fdzhc\"" Jul 09 22:44:54.166148 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:44:54.166122 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-67cb547f-q8dqt"] Jul 09 22:44:54.267662 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:44:54.267614 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f9be0827-40f8-4eef-be83-3abc36f47b7f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-67cb547f-q8dqt\" (UID: \"f9be0827-40f8-4eef-be83-3abc36f47b7f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-67cb547f-q8dqt" Jul 09 22:44:54.368993 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:44:54.368949 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f9be0827-40f8-4eef-be83-3abc36f47b7f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-67cb547f-q8dqt\" (UID: \"f9be0827-40f8-4eef-be83-3abc36f47b7f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-67cb547f-q8dqt" Jul 09 22:44:54.369607 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:44:54.369588 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f9be0827-40f8-4eef-be83-3abc36f47b7f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-67cb547f-q8dqt\" (UID: \"f9be0827-40f8-4eef-be83-3abc36f47b7f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-67cb547f-q8dqt" Jul 09 22:44:54.472400 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:44:54.472313 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-67cb547f-q8dqt" Jul 09 22:44:54.616047 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:44:54.616013 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-67cb547f-q8dqt"] Jul 09 22:44:54.618266 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:44:54.618232 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf9be0827_40f8_4eef_be83_3abc36f47b7f.slice/crio-7e487a43722f0d5ff5a3c1a36669bfe6c7e4a647d8e18043b19dbdc31023e871 WatchSource:0}: Error finding container 7e487a43722f0d5ff5a3c1a36669bfe6c7e4a647d8e18043b19dbdc31023e871: Status 404 returned error can't find the container with id 7e487a43722f0d5ff5a3c1a36669bfe6c7e4a647d8e18043b19dbdc31023e871 Jul 09 22:44:55.114808 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:44:55.114725 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-67cb547f-q8dqt" event={"ID":"f9be0827-40f8-4eef-be83-3abc36f47b7f","Type":"ContainerStarted","Data":"ea76edf88149247d122849a521a1b84192fcef2e65acc8edac47401cab1ff236"} Jul 09 22:44:55.114808 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:44:55.114777 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-67cb547f-q8dqt" event={"ID":"f9be0827-40f8-4eef-be83-3abc36f47b7f","Type":"ContainerStarted","Data":"7e487a43722f0d5ff5a3c1a36669bfe6c7e4a647d8e18043b19dbdc31023e871"} Jul 09 22:44:55.131179 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:44:55.131124 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-67cb547f-q8dqt" podStartSLOduration=1.1311022099999999 podStartE2EDuration="1.13110221s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 22:44:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 22:44:55.130838612 +0000 UTC m=+2087.042291314" watchObservedRunningTime="2026-07-09 22:44:55.13110221 +0000 UTC m=+2087.042554911" Jul 09 22:44:56.043912 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:44:56.043875 2579 patch_prober.go:28] interesting pod/e2e-mt-67cb547f-openshift-default-5d9648c9ff-5dwtm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.42:15021/healthz/ready\": dial tcp 10.132.0.42:15021: connect: connection refused" start-of-body= Jul 09 22:44:56.044455 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:44:56.043947 2579 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-67cb547f-openshift-default-5d9648c9ff-5dwtm" podUID="018127de-0b90-4c23-932d-dd1291e30903" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.42:15021/healthz/ready\": dial tcp 10.132.0.42:15021: connect: connection refused" Jul 09 22:44:56.121759 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:44:56.121725 2579 generic.go:356] "Generic (PLEG): container finished" podID="018127de-0b90-4c23-932d-dd1291e30903" containerID="f29d6b76bbb4f5ad77fefb36072ef31d6ac5aa873addb118ca2f30d88a5b5ec0" exitCode=0 Jul 09 22:44:56.121970 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:44:56.121803 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-67cb547f-openshift-default-5d9648c9ff-5dwtm" event={"ID":"018127de-0b90-4c23-932d-dd1291e30903","Type":"ContainerDied","Data":"f29d6b76bbb4f5ad77fefb36072ef31d6ac5aa873addb118ca2f30d88a5b5ec0"} Jul 09 22:44:56.122345 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:44:56.122326 2579 scope.go:117] "RemoveContainer" containerID="f29d6b76bbb4f5ad77fefb36072ef31d6ac5aa873addb118ca2f30d88a5b5ec0" Jul 09 22:44:56.123442 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:44:56.123417 2579 generic.go:356] "Generic (PLEG): container finished" podID="f9be0827-40f8-4eef-be83-3abc36f47b7f" containerID="ea76edf88149247d122849a521a1b84192fcef2e65acc8edac47401cab1ff236" exitCode=0 Jul 09 22:44:56.123557 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:44:56.123448 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-67cb547f-q8dqt" event={"ID":"f9be0827-40f8-4eef-be83-3abc36f47b7f","Type":"ContainerDied","Data":"ea76edf88149247d122849a521a1b84192fcef2e65acc8edac47401cab1ff236"} Jul 09 22:44:56.123557 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:44:56.123414 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 09 22:44:56.123675 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:44:56.123585 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 09 22:44:56.123675 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:44:56.123642 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 09 22:44:57.131002 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:44:57.130963 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-67cb547f-openshift-default-5d9648c9ff-5dwtm" event={"ID":"018127de-0b90-4c23-932d-dd1291e30903","Type":"ContainerStarted","Data":"a77a239c9b852b217f27ce626dd0d74cd31562292a0f9d917f04babe82882b40"} Jul 09 22:44:57.264456 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:44:57.264368 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-67cb547f-openshift-default-5d9648c9ff-5dwtm" Jul 09 22:44:57.265114 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:44:57.264739 2579 patch_prober.go:28] interesting pod/e2e-mt-67cb547f-openshift-default-5d9648c9ff-5dwtm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.42:15021/healthz/ready\": dial tcp 10.132.0.42:15021: connect: connection refused" start-of-body= Jul 09 22:44:57.265114 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:44:57.264796 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-67cb547f-openshift-default-5d9648c9ff-5dwtm" podUID="018127de-0b90-4c23-932d-dd1291e30903" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.42:15021/healthz/ready\": dial tcp 10.132.0.42:15021: connect: connection refused" Jul 09 22:44:57.298064 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:44:57.298037 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-67cb547f-q8dqt" Jul 09 22:44:57.398748 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:44:57.398646 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f9be0827-40f8-4eef-be83-3abc36f47b7f-maas-api-service-ca\") pod \"f9be0827-40f8-4eef-be83-3abc36f47b7f\" (UID: \"f9be0827-40f8-4eef-be83-3abc36f47b7f\") " Jul 09 22:44:57.399049 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:44:57.399024 2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f9be0827-40f8-4eef-be83-3abc36f47b7f-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "f9be0827-40f8-4eef-be83-3abc36f47b7f" (UID: "f9be0827-40f8-4eef-be83-3abc36f47b7f"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 09 22:44:57.500073 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:44:57.500035 2579 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f9be0827-40f8-4eef-be83-3abc36f47b7f-maas-api-service-ca\") on node \"ip-10-0-136-251.ec2.internal\" DevicePath \"\"" Jul 09 22:44:58.137990 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:44:58.137946 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-67cb547f-q8dqt" event={"ID":"f9be0827-40f8-4eef-be83-3abc36f47b7f","Type":"ContainerDied","Data":"7e487a43722f0d5ff5a3c1a36669bfe6c7e4a647d8e18043b19dbdc31023e871"} Jul 09 22:44:58.138487 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:44:58.137995 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-67cb547f-q8dqt" Jul 09 22:44:58.138487 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:44:58.138040 2579 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7e487a43722f0d5ff5a3c1a36669bfe6c7e4a647d8e18043b19dbdc31023e871" Jul 09 22:44:58.260945 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:44:58.260365 2579 patch_prober.go:28] interesting pod/e2e-mt-67cb547f-openshift-default-5d9648c9ff-5dwtm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.42:15021/healthz/ready\": dial tcp 10.132.0.42:15021: connect: connection refused" start-of-body= Jul 09 22:44:58.260945 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:44:58.260433 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-67cb547f-openshift-default-5d9648c9ff-5dwtm" podUID="018127de-0b90-4c23-932d-dd1291e30903" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.42:15021/healthz/ready\": dial tcp 10.132.0.42:15021: connect: connection refused" Jul 09 22:44:59.256602 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:44:59.256556 2579 patch_prober.go:28] interesting pod/e2e-mt-67cb547f-openshift-default-5d9648c9ff-5dwtm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.42:15021/healthz/ready\": dial tcp 10.132.0.42:15021: connect: connection refused" start-of-body= Jul 09 22:44:59.257281 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:44:59.257241 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-67cb547f-openshift-default-5d9648c9ff-5dwtm" podUID="018127de-0b90-4c23-932d-dd1291e30903" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.42:15021/healthz/ready\": dial tcp 10.132.0.42:15021: connect: connection refused" Jul 09 22:45:00.132769 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:45:00.132732 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-67cb547f-29727285-ndqg9"] Jul 09 22:45:00.133194 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:45:00.133179 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f9be0827-40f8-4eef-be83-3abc36f47b7f" containerName="revoke-keys" Jul 09 22:45:00.133249 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:45:00.133196 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="f9be0827-40f8-4eef-be83-3abc36f47b7f" containerName="revoke-keys" Jul 09 22:45:00.133283 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:45:00.133272 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="f9be0827-40f8-4eef-be83-3abc36f47b7f" containerName="revoke-keys" Jul 09 22:45:00.136740 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:45:00.136708 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-67cb547f-29727285-ndqg9" Jul 09 22:45:00.139667 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:45:00.139637 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fdzhc\"" Jul 09 22:45:00.141688 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:45:00.141667 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29727285-grs86"] Jul 09 22:45:00.146079 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:45:00.146054 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29727285-grs86" Jul 09 22:45:00.150118 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:45:00.150083 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-67cb547f-29727285-ndqg9"] Jul 09 22:45:00.153517 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:45:00.153497 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-67cb547f-29727285-ndqg9" Jul 09 22:45:00.154510 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:45:00.154483 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29727285-grs86"] Jul 09 22:45:00.166531 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:45:00.166507 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29727285-grs86" Jul 09 22:45:00.260942 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:45:00.260485 2579 patch_prober.go:28] interesting pod/e2e-mt-67cb547f-openshift-default-5d9648c9ff-5dwtm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.42:15021/healthz/ready\": dial tcp 10.132.0.42:15021: connect: connection refused" start-of-body= Jul 09 22:45:00.260942 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:45:00.260557 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-67cb547f-openshift-default-5d9648c9ff-5dwtm" podUID="018127de-0b90-4c23-932d-dd1291e30903" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.42:15021/healthz/ready\": dial tcp 10.132.0.42:15021: connect: connection refused" Jul 09 22:45:00.369977 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:45:00.362694 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-67cb547f-29727285-ndqg9"] Jul 09 22:45:00.375117 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:45:00.375082 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeeab6ee4_1801_464d_b625_7ecfc69c9bd6.slice/crio-45d4d9b34e12f844575642f2bdbb286c5e3c839d2ed5b6872713f8de288b5288 WatchSource:0}: Error finding container 45d4d9b34e12f844575642f2bdbb286c5e3c839d2ed5b6872713f8de288b5288: Status 404 returned error can't find the container with id 45d4d9b34e12f844575642f2bdbb286c5e3c839d2ed5b6872713f8de288b5288 Jul 09 22:45:00.609177 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:45:00.609145 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29727285-grs86"] Jul 09 22:45:00.610658 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:45:00.610629 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod413b1426_ad61_40dd_98f1_289859e64232.slice/crio-0acf5b3aea65609becab3bcf60c53422c39de1208383946944a07d0bcd903b27 WatchSource:0}: Error finding container 0acf5b3aea65609becab3bcf60c53422c39de1208383946944a07d0bcd903b27: Status 404 returned error can't find the container with id 0acf5b3aea65609becab3bcf60c53422c39de1208383946944a07d0bcd903b27 Jul 09 22:45:01.158671 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:45:01.158631 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29727285-grs86" event={"ID":"413b1426-ad61-40dd-98f1-289859e64232","Type":"ContainerStarted","Data":"1044c25d12d88963e8f176c2fd8dfb2a78acdb7e9d818896770b0b9cb7da2ec4"} Jul 09 22:45:01.158870 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:45:01.158698 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29727285-grs86" event={"ID":"413b1426-ad61-40dd-98f1-289859e64232","Type":"ContainerStarted","Data":"0acf5b3aea65609becab3bcf60c53422c39de1208383946944a07d0bcd903b27"} Jul 09 22:45:01.160153 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:45:01.160126 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-67cb547f-29727285-ndqg9" event={"ID":"eeab6ee4-1801-464d-b625-7ecfc69c9bd6","Type":"ContainerStarted","Data":"f8147c174f041ea94faa77544648ea9772412b3a2c230683d2837eca76976a8a"} Jul 09 22:45:01.160269 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:45:01.160175 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-67cb547f-29727285-ndqg9" event={"ID":"eeab6ee4-1801-464d-b625-7ecfc69c9bd6","Type":"ContainerStarted","Data":"45d4d9b34e12f844575642f2bdbb286c5e3c839d2ed5b6872713f8de288b5288"} Jul 09 22:45:01.177105 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:45:01.177055 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29727285-grs86" podStartSLOduration=1.17703958 podStartE2EDuration="1.17703958s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 22:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 22:45:01.175578514 +0000 UTC m=+2093.087031211" watchObservedRunningTime="2026-07-09 22:45:01.17703958 +0000 UTC m=+2093.088492280" Jul 09 22:45:01.192609 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:45:01.192542 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-67cb547f-29727285-ndqg9" podStartSLOduration=1.192526279 podStartE2EDuration="1.192526279s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 22:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 22:45:01.191040299 +0000 UTC m=+2093.102493000" watchObservedRunningTime="2026-07-09 22:45:01.192526279 +0000 UTC m=+2093.103978979" Jul 09 22:45:01.260363 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:45:01.260336 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-67cb547f-openshift-default-5d9648c9ff-5dwtm" Jul 09 22:45:01.260749 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:45:01.260728 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-67cb547f-openshift-default-5d9648c9ff-5dwtm" Jul 09 22:45:01.261514 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:45:01.261490 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-67cb547f-openshift-default-5d9648c9ff-5dwtm" Jul 09 22:45:02.166642 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:45:02.166601 2579 generic.go:356] "Generic (PLEG): container finished" podID="413b1426-ad61-40dd-98f1-289859e64232" containerID="1044c25d12d88963e8f176c2fd8dfb2a78acdb7e9d818896770b0b9cb7da2ec4" exitCode=0 Jul 09 22:45:02.166838 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:45:02.166695 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29727285-grs86" event={"ID":"413b1426-ad61-40dd-98f1-289859e64232","Type":"ContainerDied","Data":"1044c25d12d88963e8f176c2fd8dfb2a78acdb7e9d818896770b0b9cb7da2ec4"} Jul 09 22:45:02.168188 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:45:02.168158 2579 generic.go:356] "Generic (PLEG): container finished" podID="eeab6ee4-1801-464d-b625-7ecfc69c9bd6" containerID="f8147c174f041ea94faa77544648ea9772412b3a2c230683d2837eca76976a8a" exitCode=0 Jul 09 22:45:02.168300 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:45:02.168244 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-67cb547f-29727285-ndqg9" event={"ID":"eeab6ee4-1801-464d-b625-7ecfc69c9bd6","Type":"ContainerDied","Data":"f8147c174f041ea94faa77544648ea9772412b3a2c230683d2837eca76976a8a"} Jul 09 22:45:03.322834 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:45:03.322812 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-67cb547f-29727285-ndqg9" Jul 09 22:45:03.343368 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:45:03.343338 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29727285-grs86" Jul 09 22:45:04.179569 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:45:04.179539 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29727285-grs86" Jul 09 22:45:04.179873 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:45:04.179563 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29727285-grs86" event={"ID":"413b1426-ad61-40dd-98f1-289859e64232","Type":"ContainerDied","Data":"0acf5b3aea65609becab3bcf60c53422c39de1208383946944a07d0bcd903b27"} Jul 09 22:45:04.179873 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:45:04.179650 2579 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0acf5b3aea65609becab3bcf60c53422c39de1208383946944a07d0bcd903b27" Jul 09 22:45:04.181040 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:45:04.181021 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-67cb547f-29727285-ndqg9" Jul 09 22:45:04.181167 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:45:04.181056 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-67cb547f-29727285-ndqg9" event={"ID":"eeab6ee4-1801-464d-b625-7ecfc69c9bd6","Type":"ContainerDied","Data":"45d4d9b34e12f844575642f2bdbb286c5e3c839d2ed5b6872713f8de288b5288"} Jul 09 22:45:04.181167 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:45:04.181109 2579 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="45d4d9b34e12f844575642f2bdbb286c5e3c839d2ed5b6872713f8de288b5288" Jul 09 22:45:04.200521 ip-10-0-136-251 kubenswrapper[2579]: E0709 22:45:04.200488 2579 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeeab6ee4_1801_464d_b625_7ecfc69c9bd6.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod413b1426_ad61_40dd_98f1_289859e64232.slice\": RecentStats: unable to find data in memory cache]" Jul 09 22:45:04.693211 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:45:04.693171 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-67cb547f-29727285-ndqg9"] Jul 09 22:45:04.698885 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:45:04.698854 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-67cb547f-29727285-ndqg9"] Jul 09 22:45:06.564282 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:45:06.564247 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="eeab6ee4-1801-464d-b625-7ecfc69c9bd6" path="/var/lib/kubelet/pods/eeab6ee4-1801-464d-b625-7ecfc69c9bd6/volumes" Jul 09 22:45:07.519042 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:45:07.518999 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-xdfrr"] Jul 09 22:45:19.885666 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:45:19.885632 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-67cb547f-openshift-default-5d9648c9ff-5dwtm"] Jul 09 22:45:19.886117 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:45:19.885871 2579 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-67cb547f-openshift-default-5d9648c9ff-5dwtm" podUID="018127de-0b90-4c23-932d-dd1291e30903" containerName="istio-proxy" containerID="cri-o://a77a239c9b852b217f27ce626dd0d74cd31562292a0f9d917f04babe82882b40" gracePeriod=30 Jul 09 22:45:25.145899 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:45:25.145865 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-67cb547f-openshift-default-5d9648c9ff-5dwtm" Jul 09 22:45:25.246902 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:45:25.246862 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/018127de-0b90-4c23-932d-dd1291e30903-istio-data\") pod \"018127de-0b90-4c23-932d-dd1291e30903\" (UID: \"018127de-0b90-4c23-932d-dd1291e30903\") " Jul 09 22:45:25.247133 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:45:25.246911 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/018127de-0b90-4c23-932d-dd1291e30903-istio-token\") pod \"018127de-0b90-4c23-932d-dd1291e30903\" (UID: \"018127de-0b90-4c23-932d-dd1291e30903\") " Jul 09 22:45:25.247133 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:45:25.246963 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/018127de-0b90-4c23-932d-dd1291e30903-workload-socket\") pod \"018127de-0b90-4c23-932d-dd1291e30903\" (UID: \"018127de-0b90-4c23-932d-dd1291e30903\") " Jul 09 22:45:25.247133 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:45:25.247027 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jnzx7\" (UniqueName: \"kubernetes.io/projected/018127de-0b90-4c23-932d-dd1291e30903-kube-api-access-jnzx7\") pod \"018127de-0b90-4c23-932d-dd1291e30903\" (UID: \"018127de-0b90-4c23-932d-dd1291e30903\") " Jul 09 22:45:25.247133 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:45:25.247052 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/018127de-0b90-4c23-932d-dd1291e30903-istiod-ca-cert\") pod \"018127de-0b90-4c23-932d-dd1291e30903\" (UID: \"018127de-0b90-4c23-932d-dd1291e30903\") " Jul 09 22:45:25.247133 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:45:25.247079 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/018127de-0b90-4c23-932d-dd1291e30903-workload-certs\") pod \"018127de-0b90-4c23-932d-dd1291e30903\" (UID: \"018127de-0b90-4c23-932d-dd1291e30903\") " Jul 09 22:45:25.247133 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:45:25.247123 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/018127de-0b90-4c23-932d-dd1291e30903-credential-socket\") pod \"018127de-0b90-4c23-932d-dd1291e30903\" (UID: \"018127de-0b90-4c23-932d-dd1291e30903\") " Jul 09 22:45:25.247453 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:45:25.247161 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/018127de-0b90-4c23-932d-dd1291e30903-istio-envoy\") pod \"018127de-0b90-4c23-932d-dd1291e30903\" (UID: \"018127de-0b90-4c23-932d-dd1291e30903\") " Jul 09 22:45:25.247453 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:45:25.247209 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/018127de-0b90-4c23-932d-dd1291e30903-istio-podinfo\") pod \"018127de-0b90-4c23-932d-dd1291e30903\" (UID: \"018127de-0b90-4c23-932d-dd1291e30903\") " Jul 09 22:45:25.247453 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:45:25.247255 2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/018127de-0b90-4c23-932d-dd1291e30903-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "018127de-0b90-4c23-932d-dd1291e30903" (UID: "018127de-0b90-4c23-932d-dd1291e30903"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 22:45:25.247626 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:45:25.247452 2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/018127de-0b90-4c23-932d-dd1291e30903-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "018127de-0b90-4c23-932d-dd1291e30903" (UID: "018127de-0b90-4c23-932d-dd1291e30903"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 22:45:25.247626 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:45:25.247554 2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/018127de-0b90-4c23-932d-dd1291e30903-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "018127de-0b90-4c23-932d-dd1291e30903" (UID: "018127de-0b90-4c23-932d-dd1291e30903"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 09 22:45:25.247740 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:45:25.247647 2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/018127de-0b90-4c23-932d-dd1291e30903-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "018127de-0b90-4c23-932d-dd1291e30903" (UID: "018127de-0b90-4c23-932d-dd1291e30903"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 22:45:25.247845 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:45:25.247822 2579 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/018127de-0b90-4c23-932d-dd1291e30903-workload-socket\") on node \"ip-10-0-136-251.ec2.internal\" DevicePath \"\"" Jul 09 22:45:25.247909 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:45:25.247850 2579 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/018127de-0b90-4c23-932d-dd1291e30903-istiod-ca-cert\") on node \"ip-10-0-136-251.ec2.internal\" DevicePath \"\"" Jul 09 22:45:25.247909 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:45:25.247857 2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/018127de-0b90-4c23-932d-dd1291e30903-istio-data" (OuterVolumeSpecName: "istio-data") pod "018127de-0b90-4c23-932d-dd1291e30903" (UID: "018127de-0b90-4c23-932d-dd1291e30903"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 22:45:25.247909 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:45:25.247864 2579 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/018127de-0b90-4c23-932d-dd1291e30903-workload-certs\") on node \"ip-10-0-136-251.ec2.internal\" DevicePath \"\"" Jul 09 22:45:25.247909 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:45:25.247906 2579 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/018127de-0b90-4c23-932d-dd1291e30903-credential-socket\") on node \"ip-10-0-136-251.ec2.internal\" DevicePath \"\"" Jul 09 22:45:25.249361 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:45:25.249336 2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/018127de-0b90-4c23-932d-dd1291e30903-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "018127de-0b90-4c23-932d-dd1291e30903" (UID: "018127de-0b90-4c23-932d-dd1291e30903"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 09 22:45:25.249492 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:45:25.249347 2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/018127de-0b90-4c23-932d-dd1291e30903-istio-token" (OuterVolumeSpecName: "istio-token") pod "018127de-0b90-4c23-932d-dd1291e30903" (UID: "018127de-0b90-4c23-932d-dd1291e30903"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 22:45:25.249492 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:45:25.249386 2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/018127de-0b90-4c23-932d-dd1291e30903-kube-api-access-jnzx7" (OuterVolumeSpecName: "kube-api-access-jnzx7") pod "018127de-0b90-4c23-932d-dd1291e30903" (UID: "018127de-0b90-4c23-932d-dd1291e30903"). InnerVolumeSpecName "kube-api-access-jnzx7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 22:45:25.249569 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:45:25.249485 2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/018127de-0b90-4c23-932d-dd1291e30903-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "018127de-0b90-4c23-932d-dd1291e30903" (UID: "018127de-0b90-4c23-932d-dd1291e30903"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 22:45:25.297630 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:45:25.297600 2579 generic.go:356] "Generic (PLEG): container finished" podID="018127de-0b90-4c23-932d-dd1291e30903" containerID="a77a239c9b852b217f27ce626dd0d74cd31562292a0f9d917f04babe82882b40" exitCode=0 Jul 09 22:45:25.297842 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:45:25.297653 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-67cb547f-openshift-default-5d9648c9ff-5dwtm" Jul 09 22:45:25.297842 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:45:25.297689 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-67cb547f-openshift-default-5d9648c9ff-5dwtm" event={"ID":"018127de-0b90-4c23-932d-dd1291e30903","Type":"ContainerDied","Data":"a77a239c9b852b217f27ce626dd0d74cd31562292a0f9d917f04babe82882b40"} Jul 09 22:45:25.297842 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:45:25.297757 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-67cb547f-openshift-default-5d9648c9ff-5dwtm" event={"ID":"018127de-0b90-4c23-932d-dd1291e30903","Type":"ContainerDied","Data":"f837c750c37c6f9f07ff4bdb86f6de27703fcd420c2e0ca6255237ec80147f82"} Jul 09 22:45:25.297842 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:45:25.297788 2579 scope.go:117] "RemoveContainer" containerID="a77a239c9b852b217f27ce626dd0d74cd31562292a0f9d917f04babe82882b40" Jul 09 22:45:25.307785 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:45:25.307767 2579 scope.go:117] "RemoveContainer" containerID="f29d6b76bbb4f5ad77fefb36072ef31d6ac5aa873addb118ca2f30d88a5b5ec0" Jul 09 22:45:25.316364 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:45:25.316343 2579 scope.go:117] "RemoveContainer" containerID="a77a239c9b852b217f27ce626dd0d74cd31562292a0f9d917f04babe82882b40" Jul 09 22:45:25.316655 ip-10-0-136-251 kubenswrapper[2579]: E0709 22:45:25.316635 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a77a239c9b852b217f27ce626dd0d74cd31562292a0f9d917f04babe82882b40\": container with ID starting with a77a239c9b852b217f27ce626dd0d74cd31562292a0f9d917f04babe82882b40 not found: ID does not exist" containerID="a77a239c9b852b217f27ce626dd0d74cd31562292a0f9d917f04babe82882b40" Jul 09 22:45:25.316715 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:45:25.316664 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a77a239c9b852b217f27ce626dd0d74cd31562292a0f9d917f04babe82882b40"} err="failed to get container status \"a77a239c9b852b217f27ce626dd0d74cd31562292a0f9d917f04babe82882b40\": rpc error: code = NotFound desc = could not find container \"a77a239c9b852b217f27ce626dd0d74cd31562292a0f9d917f04babe82882b40\": container with ID starting with a77a239c9b852b217f27ce626dd0d74cd31562292a0f9d917f04babe82882b40 not found: ID does not exist" Jul 09 22:45:25.316715 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:45:25.316681 2579 scope.go:117] "RemoveContainer" containerID="f29d6b76bbb4f5ad77fefb36072ef31d6ac5aa873addb118ca2f30d88a5b5ec0" Jul 09 22:45:25.316941 ip-10-0-136-251 kubenswrapper[2579]: E0709 22:45:25.316910 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f29d6b76bbb4f5ad77fefb36072ef31d6ac5aa873addb118ca2f30d88a5b5ec0\": container with ID starting with f29d6b76bbb4f5ad77fefb36072ef31d6ac5aa873addb118ca2f30d88a5b5ec0 not found: ID does not exist" containerID="f29d6b76bbb4f5ad77fefb36072ef31d6ac5aa873addb118ca2f30d88a5b5ec0" Jul 09 22:45:25.317004 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:45:25.316951 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f29d6b76bbb4f5ad77fefb36072ef31d6ac5aa873addb118ca2f30d88a5b5ec0"} err="failed to get container status \"f29d6b76bbb4f5ad77fefb36072ef31d6ac5aa873addb118ca2f30d88a5b5ec0\": rpc error: code = NotFound desc = could not find container \"f29d6b76bbb4f5ad77fefb36072ef31d6ac5aa873addb118ca2f30d88a5b5ec0\": container with ID starting with f29d6b76bbb4f5ad77fefb36072ef31d6ac5aa873addb118ca2f30d88a5b5ec0 not found: ID does not exist" Jul 09 22:45:25.321759 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:45:25.321736 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-67cb547f-openshift-default-5d9648c9ff-5dwtm"] Jul 09 22:45:25.329708 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:45:25.329675 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-67cb547f-openshift-default-5d9648c9ff-5dwtm"] Jul 09 22:45:25.348972 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:45:25.348940 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jnzx7\" (UniqueName: \"kubernetes.io/projected/018127de-0b90-4c23-932d-dd1291e30903-kube-api-access-jnzx7\") on node \"ip-10-0-136-251.ec2.internal\" DevicePath \"\"" Jul 09 22:45:25.348972 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:45:25.348973 2579 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/018127de-0b90-4c23-932d-dd1291e30903-istio-envoy\") on node \"ip-10-0-136-251.ec2.internal\" DevicePath \"\"" Jul 09 22:45:25.349151 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:45:25.348989 2579 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/018127de-0b90-4c23-932d-dd1291e30903-istio-podinfo\") on node \"ip-10-0-136-251.ec2.internal\" DevicePath \"\"" Jul 09 22:45:25.349151 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:45:25.348998 2579 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/018127de-0b90-4c23-932d-dd1291e30903-istio-data\") on node \"ip-10-0-136-251.ec2.internal\" DevicePath \"\"" Jul 09 22:45:25.349151 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:45:25.349006 2579 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/018127de-0b90-4c23-932d-dd1291e30903-istio-token\") on node \"ip-10-0-136-251.ec2.internal\" DevicePath \"\"" Jul 09 22:45:26.565161 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:45:26.565127 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="018127de-0b90-4c23-932d-dd1291e30903" path="/var/lib/kubelet/pods/018127de-0b90-4c23-932d-dd1291e30903/volumes" Jul 09 22:45:39.424465 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:45:39.424431 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-eb6719bd-openshift-default-5cf45475f-ksshj"] Jul 09 22:45:39.424964 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:45:39.424800 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="413b1426-ad61-40dd-98f1-289859e64232" containerName="cleanup" Jul 09 22:45:39.424964 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:45:39.424811 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="413b1426-ad61-40dd-98f1-289859e64232" containerName="cleanup" Jul 09 22:45:39.424964 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:45:39.424828 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="018127de-0b90-4c23-932d-dd1291e30903" containerName="istio-proxy" Jul 09 22:45:39.424964 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:45:39.424834 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="018127de-0b90-4c23-932d-dd1291e30903" containerName="istio-proxy" Jul 09 22:45:39.424964 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:45:39.424841 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="018127de-0b90-4c23-932d-dd1291e30903" containerName="istio-proxy" Jul 09 22:45:39.424964 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:45:39.424847 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="018127de-0b90-4c23-932d-dd1291e30903" containerName="istio-proxy" Jul 09 22:45:39.424964 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:45:39.424866 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eeab6ee4-1801-464d-b625-7ecfc69c9bd6" containerName="cleanup" Jul 09 22:45:39.424964 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:45:39.424872 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="eeab6ee4-1801-464d-b625-7ecfc69c9bd6" containerName="cleanup" Jul 09 22:45:39.424964 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:45:39.424949 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="eeab6ee4-1801-464d-b625-7ecfc69c9bd6" containerName="cleanup" Jul 09 22:45:39.424964 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:45:39.424960 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="018127de-0b90-4c23-932d-dd1291e30903" containerName="istio-proxy" Jul 09 22:45:39.424964 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:45:39.424969 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="018127de-0b90-4c23-932d-dd1291e30903" containerName="istio-proxy" Jul 09 22:45:39.425317 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:45:39.424978 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="413b1426-ad61-40dd-98f1-289859e64232" containerName="cleanup" Jul 09 22:45:39.429306 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:45:39.429285 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-eb6719bd-openshift-default-5cf45475f-ksshj" Jul 09 22:45:39.432131 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:45:39.432106 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-eb6719bd-openshift-default-dockercfg-rhzz4\"" Jul 09 22:45:39.441833 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:45:39.441805 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-eb6719bd-openshift-default-5cf45475f-ksshj"] Jul 09 22:45:39.476705 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:45:39.476665 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9190ba20-80d6-463a-98fc-3ce89ba9179d-istiod-ca-cert\") pod \"e2e-mt-eb6719bd-openshift-default-5cf45475f-ksshj\" (UID: \"9190ba20-80d6-463a-98fc-3ce89ba9179d\") " pod="openshift-ingress/e2e-mt-eb6719bd-openshift-default-5cf45475f-ksshj" Jul 09 22:45:39.476893 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:45:39.476735 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9190ba20-80d6-463a-98fc-3ce89ba9179d-credential-socket\") pod \"e2e-mt-eb6719bd-openshift-default-5cf45475f-ksshj\" (UID: \"9190ba20-80d6-463a-98fc-3ce89ba9179d\") " pod="openshift-ingress/e2e-mt-eb6719bd-openshift-default-5cf45475f-ksshj" Jul 09 22:45:39.476893 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:45:39.476799 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9190ba20-80d6-463a-98fc-3ce89ba9179d-istio-podinfo\") pod \"e2e-mt-eb6719bd-openshift-default-5cf45475f-ksshj\" (UID: \"9190ba20-80d6-463a-98fc-3ce89ba9179d\") " pod="openshift-ingress/e2e-mt-eb6719bd-openshift-default-5cf45475f-ksshj" Jul 09 22:45:39.476893 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:45:39.476827 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9190ba20-80d6-463a-98fc-3ce89ba9179d-istio-token\") pod \"e2e-mt-eb6719bd-openshift-default-5cf45475f-ksshj\" (UID: \"9190ba20-80d6-463a-98fc-3ce89ba9179d\") " pod="openshift-ingress/e2e-mt-eb6719bd-openshift-default-5cf45475f-ksshj" Jul 09 22:45:39.477167 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:45:39.476950 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9190ba20-80d6-463a-98fc-3ce89ba9179d-istio-envoy\") pod \"e2e-mt-eb6719bd-openshift-default-5cf45475f-ksshj\" (UID: \"9190ba20-80d6-463a-98fc-3ce89ba9179d\") " pod="openshift-ingress/e2e-mt-eb6719bd-openshift-default-5cf45475f-ksshj" Jul 09 22:45:39.477167 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:45:39.476986 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4fkgp\" (UniqueName: \"kubernetes.io/projected/9190ba20-80d6-463a-98fc-3ce89ba9179d-kube-api-access-4fkgp\") pod \"e2e-mt-eb6719bd-openshift-default-5cf45475f-ksshj\" (UID: \"9190ba20-80d6-463a-98fc-3ce89ba9179d\") " pod="openshift-ingress/e2e-mt-eb6719bd-openshift-default-5cf45475f-ksshj" Jul 09 22:45:39.477167 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:45:39.477037 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9190ba20-80d6-463a-98fc-3ce89ba9179d-istio-data\") pod \"e2e-mt-eb6719bd-openshift-default-5cf45475f-ksshj\" (UID: \"9190ba20-80d6-463a-98fc-3ce89ba9179d\") " pod="openshift-ingress/e2e-mt-eb6719bd-openshift-default-5cf45475f-ksshj" Jul 09 22:45:39.477167 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:45:39.477088 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9190ba20-80d6-463a-98fc-3ce89ba9179d-workload-socket\") pod \"e2e-mt-eb6719bd-openshift-default-5cf45475f-ksshj\" (UID: \"9190ba20-80d6-463a-98fc-3ce89ba9179d\") " pod="openshift-ingress/e2e-mt-eb6719bd-openshift-default-5cf45475f-ksshj" Jul 09 22:45:39.477167 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:45:39.477115 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9190ba20-80d6-463a-98fc-3ce89ba9179d-workload-certs\") pod \"e2e-mt-eb6719bd-openshift-default-5cf45475f-ksshj\" (UID: \"9190ba20-80d6-463a-98fc-3ce89ba9179d\") " pod="openshift-ingress/e2e-mt-eb6719bd-openshift-default-5cf45475f-ksshj" Jul 09 22:45:39.577643 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:45:39.577597 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9190ba20-80d6-463a-98fc-3ce89ba9179d-credential-socket\") pod \"e2e-mt-eb6719bd-openshift-default-5cf45475f-ksshj\" (UID: \"9190ba20-80d6-463a-98fc-3ce89ba9179d\") " pod="openshift-ingress/e2e-mt-eb6719bd-openshift-default-5cf45475f-ksshj" Jul 09 22:45:39.577828 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:45:39.577688 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9190ba20-80d6-463a-98fc-3ce89ba9179d-istio-podinfo\") pod \"e2e-mt-eb6719bd-openshift-default-5cf45475f-ksshj\" (UID: \"9190ba20-80d6-463a-98fc-3ce89ba9179d\") " pod="openshift-ingress/e2e-mt-eb6719bd-openshift-default-5cf45475f-ksshj" Jul 09 22:45:39.577828 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:45:39.577717 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9190ba20-80d6-463a-98fc-3ce89ba9179d-istio-token\") pod \"e2e-mt-eb6719bd-openshift-default-5cf45475f-ksshj\" (UID: \"9190ba20-80d6-463a-98fc-3ce89ba9179d\") " pod="openshift-ingress/e2e-mt-eb6719bd-openshift-default-5cf45475f-ksshj" Jul 09 22:45:39.577828 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:45:39.577762 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9190ba20-80d6-463a-98fc-3ce89ba9179d-istio-envoy\") pod \"e2e-mt-eb6719bd-openshift-default-5cf45475f-ksshj\" (UID: \"9190ba20-80d6-463a-98fc-3ce89ba9179d\") " pod="openshift-ingress/e2e-mt-eb6719bd-openshift-default-5cf45475f-ksshj" Jul 09 22:45:39.577828 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:45:39.577785 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4fkgp\" (UniqueName: \"kubernetes.io/projected/9190ba20-80d6-463a-98fc-3ce89ba9179d-kube-api-access-4fkgp\") pod \"e2e-mt-eb6719bd-openshift-default-5cf45475f-ksshj\" (UID: \"9190ba20-80d6-463a-98fc-3ce89ba9179d\") " pod="openshift-ingress/e2e-mt-eb6719bd-openshift-default-5cf45475f-ksshj" Jul 09 22:45:39.577828 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:45:39.577810 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9190ba20-80d6-463a-98fc-3ce89ba9179d-istio-data\") pod \"e2e-mt-eb6719bd-openshift-default-5cf45475f-ksshj\" (UID: \"9190ba20-80d6-463a-98fc-3ce89ba9179d\") " pod="openshift-ingress/e2e-mt-eb6719bd-openshift-default-5cf45475f-ksshj" Jul 09 22:45:39.578127 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:45:39.577856 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9190ba20-80d6-463a-98fc-3ce89ba9179d-workload-socket\") pod \"e2e-mt-eb6719bd-openshift-default-5cf45475f-ksshj\" (UID: \"9190ba20-80d6-463a-98fc-3ce89ba9179d\") " pod="openshift-ingress/e2e-mt-eb6719bd-openshift-default-5cf45475f-ksshj" Jul 09 22:45:39.578127 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:45:39.577881 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9190ba20-80d6-463a-98fc-3ce89ba9179d-workload-certs\") pod \"e2e-mt-eb6719bd-openshift-default-5cf45475f-ksshj\" (UID: \"9190ba20-80d6-463a-98fc-3ce89ba9179d\") " pod="openshift-ingress/e2e-mt-eb6719bd-openshift-default-5cf45475f-ksshj" Jul 09 22:45:39.578127 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:45:39.577942 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9190ba20-80d6-463a-98fc-3ce89ba9179d-istiod-ca-cert\") pod \"e2e-mt-eb6719bd-openshift-default-5cf45475f-ksshj\" (UID: \"9190ba20-80d6-463a-98fc-3ce89ba9179d\") " pod="openshift-ingress/e2e-mt-eb6719bd-openshift-default-5cf45475f-ksshj" Jul 09 22:45:39.578309 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:45:39.578235 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9190ba20-80d6-463a-98fc-3ce89ba9179d-credential-socket\") pod \"e2e-mt-eb6719bd-openshift-default-5cf45475f-ksshj\" (UID: \"9190ba20-80d6-463a-98fc-3ce89ba9179d\") " pod="openshift-ingress/e2e-mt-eb6719bd-openshift-default-5cf45475f-ksshj" Jul 09 22:45:39.578309 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:45:39.578266 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9190ba20-80d6-463a-98fc-3ce89ba9179d-workload-socket\") pod \"e2e-mt-eb6719bd-openshift-default-5cf45475f-ksshj\" (UID: \"9190ba20-80d6-463a-98fc-3ce89ba9179d\") " pod="openshift-ingress/e2e-mt-eb6719bd-openshift-default-5cf45475f-ksshj" Jul 09 22:45:39.578412 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:45:39.578352 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9190ba20-80d6-463a-98fc-3ce89ba9179d-istio-data\") pod \"e2e-mt-eb6719bd-openshift-default-5cf45475f-ksshj\" (UID: \"9190ba20-80d6-463a-98fc-3ce89ba9179d\") " pod="openshift-ingress/e2e-mt-eb6719bd-openshift-default-5cf45475f-ksshj" Jul 09 22:45:39.578453 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:45:39.578440 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9190ba20-80d6-463a-98fc-3ce89ba9179d-workload-certs\") pod \"e2e-mt-eb6719bd-openshift-default-5cf45475f-ksshj\" (UID: \"9190ba20-80d6-463a-98fc-3ce89ba9179d\") " pod="openshift-ingress/e2e-mt-eb6719bd-openshift-default-5cf45475f-ksshj" Jul 09 22:45:39.578668 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:45:39.578653 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9190ba20-80d6-463a-98fc-3ce89ba9179d-istiod-ca-cert\") pod \"e2e-mt-eb6719bd-openshift-default-5cf45475f-ksshj\" (UID: \"9190ba20-80d6-463a-98fc-3ce89ba9179d\") " pod="openshift-ingress/e2e-mt-eb6719bd-openshift-default-5cf45475f-ksshj" Jul 09 22:45:39.580098 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:45:39.580079 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9190ba20-80d6-463a-98fc-3ce89ba9179d-istio-envoy\") pod \"e2e-mt-eb6719bd-openshift-default-5cf45475f-ksshj\" (UID: \"9190ba20-80d6-463a-98fc-3ce89ba9179d\") " pod="openshift-ingress/e2e-mt-eb6719bd-openshift-default-5cf45475f-ksshj" Jul 09 22:45:39.580393 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:45:39.580373 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9190ba20-80d6-463a-98fc-3ce89ba9179d-istio-podinfo\") pod \"e2e-mt-eb6719bd-openshift-default-5cf45475f-ksshj\" (UID: \"9190ba20-80d6-463a-98fc-3ce89ba9179d\") " pod="openshift-ingress/e2e-mt-eb6719bd-openshift-default-5cf45475f-ksshj" Jul 09 22:45:39.586716 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:45:39.586692 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9190ba20-80d6-463a-98fc-3ce89ba9179d-istio-token\") pod \"e2e-mt-eb6719bd-openshift-default-5cf45475f-ksshj\" (UID: \"9190ba20-80d6-463a-98fc-3ce89ba9179d\") " pod="openshift-ingress/e2e-mt-eb6719bd-openshift-default-5cf45475f-ksshj" Jul 09 22:45:39.586874 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:45:39.586858 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4fkgp\" (UniqueName: \"kubernetes.io/projected/9190ba20-80d6-463a-98fc-3ce89ba9179d-kube-api-access-4fkgp\") pod \"e2e-mt-eb6719bd-openshift-default-5cf45475f-ksshj\" (UID: \"9190ba20-80d6-463a-98fc-3ce89ba9179d\") " pod="openshift-ingress/e2e-mt-eb6719bd-openshift-default-5cf45475f-ksshj" Jul 09 22:45:39.740084 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:45:39.740050 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-eb6719bd-openshift-default-5cf45475f-ksshj" Jul 09 22:45:39.888839 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:45:39.888813 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-eb6719bd-openshift-default-5cf45475f-ksshj"] Jul 09 22:45:39.890668 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:45:39.890639 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9190ba20_80d6_463a_98fc_3ce89ba9179d.slice/crio-bd6479a90ee22bb170181e478abd8882a22bab12b8948ef9a46e050b874ed3e5 WatchSource:0}: Error finding container bd6479a90ee22bb170181e478abd8882a22bab12b8948ef9a46e050b874ed3e5: Status 404 returned error can't find the container with id bd6479a90ee22bb170181e478abd8882a22bab12b8948ef9a46e050b874ed3e5 Jul 09 22:45:39.893418 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:45:39.893376 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 09 22:45:39.893508 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:45:39.893468 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 09 22:45:39.893547 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:45:39.893509 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 09 22:45:40.375398 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:45:40.375362 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-eb6719bd-openshift-default-5cf45475f-ksshj" event={"ID":"9190ba20-80d6-463a-98fc-3ce89ba9179d","Type":"ContainerStarted","Data":"87b1e798c3c21bf90b58d4ff26afa713a03fc91fb27a2317515b898b63b234c6"} Jul 09 22:45:40.375576 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:45:40.375416 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-eb6719bd-openshift-default-5cf45475f-ksshj" event={"ID":"9190ba20-80d6-463a-98fc-3ce89ba9179d","Type":"ContainerStarted","Data":"bd6479a90ee22bb170181e478abd8882a22bab12b8948ef9a46e050b874ed3e5"} Jul 09 22:45:40.401608 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:45:40.401539 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-eb6719bd-openshift-default-5cf45475f-ksshj" podStartSLOduration=1.401517705 podStartE2EDuration="1.401517705s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 22:45:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 22:45:40.401155631 +0000 UTC m=+2132.312608353" watchObservedRunningTime="2026-07-09 22:45:40.401517705 +0000 UTC m=+2132.312970406" Jul 09 22:45:40.740815 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:45:40.740781 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-eb6719bd-openshift-default-5cf45475f-ksshj" Jul 09 22:45:40.746153 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:45:40.746125 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-eb6719bd-openshift-default-5cf45475f-ksshj" Jul 09 22:45:41.381264 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:45:41.381231 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-eb6719bd-openshift-default-5cf45475f-ksshj" Jul 09 22:45:41.382287 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:45:41.382266 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-eb6719bd-openshift-default-5cf45475f-ksshj" Jul 09 22:45:48.008277 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:45:48.008241 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-xdfrr"] Jul 09 22:45:56.383602 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:45:56.383565 2579 patch_prober.go:28] interesting pod/e2e-mt-eb6719bd-openshift-default-5cf45475f-ksshj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused" start-of-body= Jul 09 22:45:56.384072 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:45:56.383625 2579 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-eb6719bd-openshift-default-5cf45475f-ksshj" podUID="9190ba20-80d6-463a-98fc-3ce89ba9179d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused" Jul 09 22:45:56.466637 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:45:56.466600 2579 generic.go:356] "Generic (PLEG): container finished" podID="9190ba20-80d6-463a-98fc-3ce89ba9179d" containerID="87b1e798c3c21bf90b58d4ff26afa713a03fc91fb27a2317515b898b63b234c6" exitCode=0 Jul 09 22:45:56.466798 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:45:56.466669 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-eb6719bd-openshift-default-5cf45475f-ksshj" event={"ID":"9190ba20-80d6-463a-98fc-3ce89ba9179d","Type":"ContainerDied","Data":"87b1e798c3c21bf90b58d4ff26afa713a03fc91fb27a2317515b898b63b234c6"} Jul 09 22:45:56.467066 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:45:56.467052 2579 scope.go:117] "RemoveContainer" containerID="87b1e798c3c21bf90b58d4ff26afa713a03fc91fb27a2317515b898b63b234c6" Jul 09 22:45:56.468042 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:45:56.468020 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 09 22:45:56.468104 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:45:56.468082 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 09 22:45:56.468143 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:45:56.468118 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 09 22:45:57.475419 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:45:57.475383 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-eb6719bd-openshift-default-5cf45475f-ksshj" event={"ID":"9190ba20-80d6-463a-98fc-3ce89ba9179d","Type":"ContainerStarted","Data":"2da7553edf1d8ff4b15b2248687bbca3a66fcc81ef4b1960a45a1ed87b336142"} Jul 09 22:45:57.740389 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:45:57.740354 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-eb6719bd-openshift-default-5cf45475f-ksshj" Jul 09 22:45:57.740643 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:45:57.740617 2579 patch_prober.go:28] interesting pod/e2e-mt-eb6719bd-openshift-default-5cf45475f-ksshj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused" start-of-body= Jul 09 22:45:57.740702 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:45:57.740670 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-eb6719bd-openshift-default-5cf45475f-ksshj" podUID="9190ba20-80d6-463a-98fc-3ce89ba9179d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused" Jul 09 22:45:58.741080 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:45:58.741042 2579 patch_prober.go:28] interesting pod/e2e-mt-eb6719bd-openshift-default-5cf45475f-ksshj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused" start-of-body= Jul 09 22:45:58.741515 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:45:58.741098 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-eb6719bd-openshift-default-5cf45475f-ksshj" podUID="9190ba20-80d6-463a-98fc-3ce89ba9179d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused" Jul 09 22:45:59.740500 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:45:59.740459 2579 patch_prober.go:28] interesting pod/e2e-mt-eb6719bd-openshift-default-5cf45475f-ksshj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused" start-of-body= Jul 09 22:45:59.740686 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:45:59.740521 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-eb6719bd-openshift-default-5cf45475f-ksshj" podUID="9190ba20-80d6-463a-98fc-3ce89ba9179d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused" Jul 09 22:46:00.741373 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:46:00.741264 2579 patch_prober.go:28] interesting pod/e2e-mt-eb6719bd-openshift-default-5cf45475f-ksshj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused" start-of-body= Jul 09 22:46:00.741373 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:46:00.741348 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-eb6719bd-openshift-default-5cf45475f-ksshj" podUID="9190ba20-80d6-463a-98fc-3ce89ba9179d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused" Jul 09 22:46:01.745904 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:46:01.745870 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-eb6719bd-openshift-default-5cf45475f-ksshj" Jul 09 22:46:01.746340 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:46:01.746234 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-eb6719bd-openshift-default-5cf45475f-ksshj" Jul 09 22:46:01.747088 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:46:01.747064 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-eb6719bd-openshift-default-5cf45475f-ksshj" Jul 09 22:46:22.620241 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:46:22.620205 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-xdfrr"] Jul 09 22:46:47.219267 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:46:47.219226 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-xdfrr"] Jul 09 22:46:50.636813 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:46:50.636780 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-eb6719bd-722zs"] Jul 09 22:46:50.640427 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:46:50.640410 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-eb6719bd-722zs" Jul 09 22:46:50.642912 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:46:50.642886 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 09 22:46:50.643048 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:46:50.642886 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fdzhc\"" Jul 09 22:46:50.647444 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:46:50.647414 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-eb6719bd-722zs"] Jul 09 22:46:50.653365 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:46:50.653338 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5e177509-1d82-4fee-89b3-c306b771165e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-eb6719bd-722zs\" (UID: \"5e177509-1d82-4fee-89b3-c306b771165e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-eb6719bd-722zs" Jul 09 22:46:50.754878 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:46:50.754838 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5e177509-1d82-4fee-89b3-c306b771165e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-eb6719bd-722zs\" (UID: \"5e177509-1d82-4fee-89b3-c306b771165e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-eb6719bd-722zs" Jul 09 22:46:50.755541 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:46:50.755517 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5e177509-1d82-4fee-89b3-c306b771165e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-eb6719bd-722zs\" (UID: \"5e177509-1d82-4fee-89b3-c306b771165e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-eb6719bd-722zs" Jul 09 22:46:50.951393 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:46:50.951300 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-eb6719bd-722zs" Jul 09 22:46:51.100559 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:46:51.100526 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-eb6719bd-722zs"] Jul 09 22:46:51.101724 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:46:51.101696 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5e177509_1d82_4fee_89b3_c306b771165e.slice/crio-ebd6c9811848fe5c80312ddeab3622af24dfb3cb14ef7939c15d9ae5a26297ed WatchSource:0}: Error finding container ebd6c9811848fe5c80312ddeab3622af24dfb3cb14ef7939c15d9ae5a26297ed: Status 404 returned error can't find the container with id ebd6c9811848fe5c80312ddeab3622af24dfb3cb14ef7939c15d9ae5a26297ed Jul 09 22:46:51.767607 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:46:51.767566 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-eb6719bd-722zs" event={"ID":"5e177509-1d82-4fee-89b3-c306b771165e","Type":"ContainerStarted","Data":"68efc49a9ead930f69a9437007d913d1989e434252fcf23e9003688ba8b0777e"} Jul 09 22:46:51.768110 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:46:51.767633 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-eb6719bd-722zs" event={"ID":"5e177509-1d82-4fee-89b3-c306b771165e","Type":"ContainerStarted","Data":"ebd6c9811848fe5c80312ddeab3622af24dfb3cb14ef7939c15d9ae5a26297ed"} Jul 09 22:46:51.786415 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:46:51.786365 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-eb6719bd-722zs" podStartSLOduration=1.786348001 podStartE2EDuration="1.786348001s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 22:46:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 22:46:51.783914139 +0000 UTC m=+2203.695366841" watchObservedRunningTime="2026-07-09 22:46:51.786348001 +0000 UTC m=+2203.697800734" Jul 09 22:46:52.774077 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:46:52.774041 2579 generic.go:356] "Generic (PLEG): container finished" podID="5e177509-1d82-4fee-89b3-c306b771165e" containerID="68efc49a9ead930f69a9437007d913d1989e434252fcf23e9003688ba8b0777e" exitCode=0 Jul 09 22:46:52.774481 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:46:52.774127 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-eb6719bd-722zs" event={"ID":"5e177509-1d82-4fee-89b3-c306b771165e","Type":"ContainerDied","Data":"68efc49a9ead930f69a9437007d913d1989e434252fcf23e9003688ba8b0777e"} Jul 09 22:46:53.919262 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:46:53.919230 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-eb6719bd-722zs" Jul 09 22:46:53.984661 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:46:53.984625 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5e177509-1d82-4fee-89b3-c306b771165e-maas-api-service-ca\") pod \"5e177509-1d82-4fee-89b3-c306b771165e\" (UID: \"5e177509-1d82-4fee-89b3-c306b771165e\") " Jul 09 22:46:53.985017 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:46:53.984992 2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5e177509-1d82-4fee-89b3-c306b771165e-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "5e177509-1d82-4fee-89b3-c306b771165e" (UID: "5e177509-1d82-4fee-89b3-c306b771165e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 09 22:46:54.086120 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:46:54.086026 2579 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5e177509-1d82-4fee-89b3-c306b771165e-maas-api-service-ca\") on node \"ip-10-0-136-251.ec2.internal\" DevicePath \"\"" Jul 09 22:46:54.786626 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:46:54.786597 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-eb6719bd-722zs" Jul 09 22:46:54.786803 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:46:54.786644 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-eb6719bd-722zs" event={"ID":"5e177509-1d82-4fee-89b3-c306b771165e","Type":"ContainerDied","Data":"ebd6c9811848fe5c80312ddeab3622af24dfb3cb14ef7939c15d9ae5a26297ed"} Jul 09 22:46:54.786803 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:46:54.786690 2579 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ebd6c9811848fe5c80312ddeab3622af24dfb3cb14ef7939c15d9ae5a26297ed" Jul 09 22:47:06.411521 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:47:06.411483 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-xdfrr"] Jul 09 22:47:08.755258 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:47:08.755224 2579 scope.go:117] "RemoveContainer" containerID="52697fca0d3488631f381867c2a8f5dc799058fbae9948a6f9648acf3a01a9c3" Jul 09 22:48:08.776051 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:48:08.776018 2579 scope.go:117] "RemoveContainer" containerID="c75c9c19a5df4b5ba10b010bd2d9622dd3f193e8df38e80d0f19bf10a6c9efaa" Jul 09 22:51:02.004128 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:51:02.004093 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-eb6719bd-openshift-default-5cf45475f-ksshj"] Jul 09 22:51:02.006657 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:51:02.004317 2579 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-eb6719bd-openshift-default-5cf45475f-ksshj" podUID="9190ba20-80d6-463a-98fc-3ce89ba9179d" containerName="istio-proxy" containerID="cri-o://2da7553edf1d8ff4b15b2248687bbca3a66fcc81ef4b1960a45a1ed87b336142" gracePeriod=30 Jul 09 22:51:05.817280 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:51:05.817234 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-xdfrr"] Jul 09 22:51:07.137440 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:51:07.137397 2579 generic.go:356] "Generic (PLEG): container finished" podID="9190ba20-80d6-463a-98fc-3ce89ba9179d" containerID="2da7553edf1d8ff4b15b2248687bbca3a66fcc81ef4b1960a45a1ed87b336142" exitCode=0 Jul 09 22:51:07.137798 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:51:07.137467 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-eb6719bd-openshift-default-5cf45475f-ksshj" event={"ID":"9190ba20-80d6-463a-98fc-3ce89ba9179d","Type":"ContainerDied","Data":"2da7553edf1d8ff4b15b2248687bbca3a66fcc81ef4b1960a45a1ed87b336142"} Jul 09 22:51:07.137798 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:51:07.137537 2579 scope.go:117] "RemoveContainer" containerID="87b1e798c3c21bf90b58d4ff26afa713a03fc91fb27a2317515b898b63b234c6" Jul 09 22:51:07.266129 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:51:07.266064 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-eb6719bd-openshift-default-5cf45475f-ksshj" Jul 09 22:51:07.394011 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:51:07.393975 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9190ba20-80d6-463a-98fc-3ce89ba9179d-workload-socket\") pod \"9190ba20-80d6-463a-98fc-3ce89ba9179d\" (UID: \"9190ba20-80d6-463a-98fc-3ce89ba9179d\") " Jul 09 22:51:07.394011 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:51:07.394017 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4fkgp\" (UniqueName: \"kubernetes.io/projected/9190ba20-80d6-463a-98fc-3ce89ba9179d-kube-api-access-4fkgp\") pod \"9190ba20-80d6-463a-98fc-3ce89ba9179d\" (UID: \"9190ba20-80d6-463a-98fc-3ce89ba9179d\") " Jul 09 22:51:07.394294 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:51:07.394049 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9190ba20-80d6-463a-98fc-3ce89ba9179d-istio-podinfo\") pod \"9190ba20-80d6-463a-98fc-3ce89ba9179d\" (UID: \"9190ba20-80d6-463a-98fc-3ce89ba9179d\") " Jul 09 22:51:07.394294 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:51:07.394077 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9190ba20-80d6-463a-98fc-3ce89ba9179d-istio-token\") pod \"9190ba20-80d6-463a-98fc-3ce89ba9179d\" (UID: \"9190ba20-80d6-463a-98fc-3ce89ba9179d\") " Jul 09 22:51:07.394294 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:51:07.394107 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9190ba20-80d6-463a-98fc-3ce89ba9179d-istio-envoy\") pod \"9190ba20-80d6-463a-98fc-3ce89ba9179d\" (UID: \"9190ba20-80d6-463a-98fc-3ce89ba9179d\") " Jul 09 22:51:07.394294 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:51:07.394125 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9190ba20-80d6-463a-98fc-3ce89ba9179d-workload-certs\") pod \"9190ba20-80d6-463a-98fc-3ce89ba9179d\" (UID: \"9190ba20-80d6-463a-98fc-3ce89ba9179d\") " Jul 09 22:51:07.394294 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:51:07.394189 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9190ba20-80d6-463a-98fc-3ce89ba9179d-credential-socket\") pod \"9190ba20-80d6-463a-98fc-3ce89ba9179d\" (UID: \"9190ba20-80d6-463a-98fc-3ce89ba9179d\") " Jul 09 22:51:07.394294 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:51:07.394234 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9190ba20-80d6-463a-98fc-3ce89ba9179d-istiod-ca-cert\") pod \"9190ba20-80d6-463a-98fc-3ce89ba9179d\" (UID: \"9190ba20-80d6-463a-98fc-3ce89ba9179d\") " Jul 09 22:51:07.394294 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:51:07.394270 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9190ba20-80d6-463a-98fc-3ce89ba9179d-istio-data\") pod \"9190ba20-80d6-463a-98fc-3ce89ba9179d\" (UID: \"9190ba20-80d6-463a-98fc-3ce89ba9179d\") " Jul 09 22:51:07.394660 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:51:07.394361 2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9190ba20-80d6-463a-98fc-3ce89ba9179d-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "9190ba20-80d6-463a-98fc-3ce89ba9179d" (UID: "9190ba20-80d6-463a-98fc-3ce89ba9179d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 22:51:07.394660 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:51:07.394614 2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9190ba20-80d6-463a-98fc-3ce89ba9179d-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "9190ba20-80d6-463a-98fc-3ce89ba9179d" (UID: "9190ba20-80d6-463a-98fc-3ce89ba9179d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 22:51:07.394773 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:51:07.394674 2579 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9190ba20-80d6-463a-98fc-3ce89ba9179d-workload-socket\") on node \"ip-10-0-136-251.ec2.internal\" DevicePath \"\"" Jul 09 22:51:07.394828 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:51:07.394803 2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9190ba20-80d6-463a-98fc-3ce89ba9179d-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "9190ba20-80d6-463a-98fc-3ce89ba9179d" (UID: "9190ba20-80d6-463a-98fc-3ce89ba9179d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 22:51:07.394912 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:51:07.394893 2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9190ba20-80d6-463a-98fc-3ce89ba9179d-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "9190ba20-80d6-463a-98fc-3ce89ba9179d" (UID: "9190ba20-80d6-463a-98fc-3ce89ba9179d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 09 22:51:07.395211 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:51:07.395187 2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9190ba20-80d6-463a-98fc-3ce89ba9179d-istio-data" (OuterVolumeSpecName: "istio-data") pod "9190ba20-80d6-463a-98fc-3ce89ba9179d" (UID: "9190ba20-80d6-463a-98fc-3ce89ba9179d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 22:51:07.396523 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:51:07.396500 2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9190ba20-80d6-463a-98fc-3ce89ba9179d-kube-api-access-4fkgp" (OuterVolumeSpecName: "kube-api-access-4fkgp") pod "9190ba20-80d6-463a-98fc-3ce89ba9179d" (UID: "9190ba20-80d6-463a-98fc-3ce89ba9179d"). InnerVolumeSpecName "kube-api-access-4fkgp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 22:51:07.396523 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:51:07.396517 2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9190ba20-80d6-463a-98fc-3ce89ba9179d-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "9190ba20-80d6-463a-98fc-3ce89ba9179d" (UID: "9190ba20-80d6-463a-98fc-3ce89ba9179d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 22:51:07.396777 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:51:07.396758 2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9190ba20-80d6-463a-98fc-3ce89ba9179d-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "9190ba20-80d6-463a-98fc-3ce89ba9179d" (UID: "9190ba20-80d6-463a-98fc-3ce89ba9179d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 09 22:51:07.396828 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:51:07.396780 2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9190ba20-80d6-463a-98fc-3ce89ba9179d-istio-token" (OuterVolumeSpecName: "istio-token") pod "9190ba20-80d6-463a-98fc-3ce89ba9179d" (UID: "9190ba20-80d6-463a-98fc-3ce89ba9179d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 22:51:07.496137 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:51:07.496088 2579 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9190ba20-80d6-463a-98fc-3ce89ba9179d-workload-certs\") on node \"ip-10-0-136-251.ec2.internal\" DevicePath \"\"" Jul 09 22:51:07.496137 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:51:07.496127 2579 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9190ba20-80d6-463a-98fc-3ce89ba9179d-credential-socket\") on node \"ip-10-0-136-251.ec2.internal\" DevicePath \"\"" Jul 09 22:51:07.496137 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:51:07.496140 2579 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9190ba20-80d6-463a-98fc-3ce89ba9179d-istiod-ca-cert\") on node \"ip-10-0-136-251.ec2.internal\" DevicePath \"\"" Jul 09 22:51:07.496391 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:51:07.496151 2579 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9190ba20-80d6-463a-98fc-3ce89ba9179d-istio-data\") on node \"ip-10-0-136-251.ec2.internal\" DevicePath \"\"" Jul 09 22:51:07.496391 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:51:07.496163 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4fkgp\" (UniqueName: \"kubernetes.io/projected/9190ba20-80d6-463a-98fc-3ce89ba9179d-kube-api-access-4fkgp\") on node \"ip-10-0-136-251.ec2.internal\" DevicePath \"\"" Jul 09 22:51:07.496391 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:51:07.496174 2579 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9190ba20-80d6-463a-98fc-3ce89ba9179d-istio-podinfo\") on node \"ip-10-0-136-251.ec2.internal\" DevicePath \"\"" Jul 09 22:51:07.496391 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:51:07.496185 2579 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9190ba20-80d6-463a-98fc-3ce89ba9179d-istio-token\") on node \"ip-10-0-136-251.ec2.internal\" DevicePath \"\"" Jul 09 22:51:07.496391 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:51:07.496195 2579 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9190ba20-80d6-463a-98fc-3ce89ba9179d-istio-envoy\") on node \"ip-10-0-136-251.ec2.internal\" DevicePath \"\"" Jul 09 22:51:08.143755 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:51:08.143716 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-eb6719bd-openshift-default-5cf45475f-ksshj" Jul 09 22:51:08.144245 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:51:08.143763 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-eb6719bd-openshift-default-5cf45475f-ksshj" event={"ID":"9190ba20-80d6-463a-98fc-3ce89ba9179d","Type":"ContainerDied","Data":"bd6479a90ee22bb170181e478abd8882a22bab12b8948ef9a46e050b874ed3e5"} Jul 09 22:51:08.144245 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:51:08.143820 2579 scope.go:117] "RemoveContainer" containerID="2da7553edf1d8ff4b15b2248687bbca3a66fcc81ef4b1960a45a1ed87b336142" Jul 09 22:51:08.172430 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:51:08.172394 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-eb6719bd-openshift-default-5cf45475f-ksshj"] Jul 09 22:51:08.176167 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:51:08.176137 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-eb6719bd-openshift-default-5cf45475f-ksshj"] Jul 09 22:51:08.564488 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:51:08.564453 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9190ba20-80d6-463a-98fc-3ce89ba9179d" path="/var/lib/kubelet/pods/9190ba20-80d6-463a-98fc-3ce89ba9179d/volumes" Jul 09 22:51:08.813415 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:51:08.813386 2579 scope.go:117] "RemoveContainer" containerID="f8147c174f041ea94faa77544648ea9772412b3a2c230683d2837eca76976a8a" Jul 09 22:51:20.007439 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:51:20.007397 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-xdfrr"] Jul 09 22:51:56.976370 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:51:56.976335 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-288815-openshift-default-7cc9fd9c89-2vrqk"] Jul 09 22:51:56.976867 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:51:56.976709 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9190ba20-80d6-463a-98fc-3ce89ba9179d" containerName="istio-proxy" Jul 09 22:51:56.976867 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:51:56.976721 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="9190ba20-80d6-463a-98fc-3ce89ba9179d" containerName="istio-proxy" Jul 09 22:51:56.976867 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:51:56.976735 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9190ba20-80d6-463a-98fc-3ce89ba9179d" containerName="istio-proxy" Jul 09 22:51:56.976867 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:51:56.976741 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="9190ba20-80d6-463a-98fc-3ce89ba9179d" containerName="istio-proxy" Jul 09 22:51:56.976867 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:51:56.976756 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5e177509-1d82-4fee-89b3-c306b771165e" containerName="revoke-keys" Jul 09 22:51:56.976867 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:51:56.976761 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="5e177509-1d82-4fee-89b3-c306b771165e" containerName="revoke-keys" Jul 09 22:51:56.976867 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:51:56.976815 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="9190ba20-80d6-463a-98fc-3ce89ba9179d" containerName="istio-proxy" Jul 09 22:51:56.976867 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:51:56.976822 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="9190ba20-80d6-463a-98fc-3ce89ba9179d" containerName="istio-proxy" Jul 09 22:51:56.976867 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:51:56.976829 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="5e177509-1d82-4fee-89b3-c306b771165e" containerName="revoke-keys" Jul 09 22:51:56.979957 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:51:56.979936 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-288815-openshift-default-7cc9fd9c89-2vrqk" Jul 09 22:51:56.983438 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:51:56.983420 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-288815-openshift-default-dockercfg-cx8wm\"" Jul 09 22:51:56.991502 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:51:56.991474 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-288815-openshift-default-7cc9fd9c89-2vrqk"] Jul 09 22:51:57.020697 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:51:57.020646 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1895db98-827c-4878-9434-be32c716a416-istio-token\") pod \"e2e-inf-a-288815-openshift-default-7cc9fd9c89-2vrqk\" (UID: \"1895db98-827c-4878-9434-be32c716a416\") " pod="openshift-ingress/e2e-inf-a-288815-openshift-default-7cc9fd9c89-2vrqk" Jul 09 22:51:57.020697 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:51:57.020708 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1895db98-827c-4878-9434-be32c716a416-workload-socket\") pod \"e2e-inf-a-288815-openshift-default-7cc9fd9c89-2vrqk\" (UID: \"1895db98-827c-4878-9434-be32c716a416\") " pod="openshift-ingress/e2e-inf-a-288815-openshift-default-7cc9fd9c89-2vrqk" Jul 09 22:51:57.020984 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:51:57.020730 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1895db98-827c-4878-9434-be32c716a416-credential-socket\") pod \"e2e-inf-a-288815-openshift-default-7cc9fd9c89-2vrqk\" (UID: \"1895db98-827c-4878-9434-be32c716a416\") " pod="openshift-ingress/e2e-inf-a-288815-openshift-default-7cc9fd9c89-2vrqk" Jul 09 22:51:57.020984 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:51:57.020759 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1895db98-827c-4878-9434-be32c716a416-istio-envoy\") pod \"e2e-inf-a-288815-openshift-default-7cc9fd9c89-2vrqk\" (UID: \"1895db98-827c-4878-9434-be32c716a416\") " pod="openshift-ingress/e2e-inf-a-288815-openshift-default-7cc9fd9c89-2vrqk" Jul 09 22:51:57.020984 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:51:57.020793 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1895db98-827c-4878-9434-be32c716a416-istiod-ca-cert\") pod \"e2e-inf-a-288815-openshift-default-7cc9fd9c89-2vrqk\" (UID: \"1895db98-827c-4878-9434-be32c716a416\") " pod="openshift-ingress/e2e-inf-a-288815-openshift-default-7cc9fd9c89-2vrqk" Jul 09 22:51:57.020984 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:51:57.020866 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1895db98-827c-4878-9434-be32c716a416-workload-certs\") pod \"e2e-inf-a-288815-openshift-default-7cc9fd9c89-2vrqk\" (UID: \"1895db98-827c-4878-9434-be32c716a416\") " pod="openshift-ingress/e2e-inf-a-288815-openshift-default-7cc9fd9c89-2vrqk" Jul 09 22:51:57.020984 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:51:57.020938 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1895db98-827c-4878-9434-be32c716a416-istio-data\") pod \"e2e-inf-a-288815-openshift-default-7cc9fd9c89-2vrqk\" (UID: \"1895db98-827c-4878-9434-be32c716a416\") " pod="openshift-ingress/e2e-inf-a-288815-openshift-default-7cc9fd9c89-2vrqk" Jul 09 22:51:57.020984 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:51:57.020958 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1895db98-827c-4878-9434-be32c716a416-istio-podinfo\") pod \"e2e-inf-a-288815-openshift-default-7cc9fd9c89-2vrqk\" (UID: \"1895db98-827c-4878-9434-be32c716a416\") " pod="openshift-ingress/e2e-inf-a-288815-openshift-default-7cc9fd9c89-2vrqk" Jul 09 22:51:57.021182 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:51:57.020986 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-trvb6\" (UniqueName: \"kubernetes.io/projected/1895db98-827c-4878-9434-be32c716a416-kube-api-access-trvb6\") pod \"e2e-inf-a-288815-openshift-default-7cc9fd9c89-2vrqk\" (UID: \"1895db98-827c-4878-9434-be32c716a416\") " pod="openshift-ingress/e2e-inf-a-288815-openshift-default-7cc9fd9c89-2vrqk" Jul 09 22:51:57.121746 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:51:57.121696 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1895db98-827c-4878-9434-be32c716a416-workload-socket\") pod \"e2e-inf-a-288815-openshift-default-7cc9fd9c89-2vrqk\" (UID: \"1895db98-827c-4878-9434-be32c716a416\") " pod="openshift-ingress/e2e-inf-a-288815-openshift-default-7cc9fd9c89-2vrqk" Jul 09 22:51:57.121746 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:51:57.121753 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1895db98-827c-4878-9434-be32c716a416-credential-socket\") pod \"e2e-inf-a-288815-openshift-default-7cc9fd9c89-2vrqk\" (UID: \"1895db98-827c-4878-9434-be32c716a416\") " pod="openshift-ingress/e2e-inf-a-288815-openshift-default-7cc9fd9c89-2vrqk" Jul 09 22:51:57.122067 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:51:57.121797 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1895db98-827c-4878-9434-be32c716a416-istio-envoy\") pod \"e2e-inf-a-288815-openshift-default-7cc9fd9c89-2vrqk\" (UID: \"1895db98-827c-4878-9434-be32c716a416\") " pod="openshift-ingress/e2e-inf-a-288815-openshift-default-7cc9fd9c89-2vrqk" Jul 09 22:51:57.122067 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:51:57.121830 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1895db98-827c-4878-9434-be32c716a416-istiod-ca-cert\") pod \"e2e-inf-a-288815-openshift-default-7cc9fd9c89-2vrqk\" (UID: \"1895db98-827c-4878-9434-be32c716a416\") " pod="openshift-ingress/e2e-inf-a-288815-openshift-default-7cc9fd9c89-2vrqk" Jul 09 22:51:57.122067 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:51:57.121880 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1895db98-827c-4878-9434-be32c716a416-workload-certs\") pod \"e2e-inf-a-288815-openshift-default-7cc9fd9c89-2vrqk\" (UID: \"1895db98-827c-4878-9434-be32c716a416\") " pod="openshift-ingress/e2e-inf-a-288815-openshift-default-7cc9fd9c89-2vrqk" Jul 09 22:51:57.122067 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:51:57.121969 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1895db98-827c-4878-9434-be32c716a416-istio-data\") pod \"e2e-inf-a-288815-openshift-default-7cc9fd9c89-2vrqk\" (UID: \"1895db98-827c-4878-9434-be32c716a416\") " pod="openshift-ingress/e2e-inf-a-288815-openshift-default-7cc9fd9c89-2vrqk" Jul 09 22:51:57.122067 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:51:57.121999 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1895db98-827c-4878-9434-be32c716a416-istio-podinfo\") pod \"e2e-inf-a-288815-openshift-default-7cc9fd9c89-2vrqk\" (UID: \"1895db98-827c-4878-9434-be32c716a416\") " pod="openshift-ingress/e2e-inf-a-288815-openshift-default-7cc9fd9c89-2vrqk" Jul 09 22:51:57.122067 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:51:57.122036 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-trvb6\" (UniqueName: \"kubernetes.io/projected/1895db98-827c-4878-9434-be32c716a416-kube-api-access-trvb6\") pod \"e2e-inf-a-288815-openshift-default-7cc9fd9c89-2vrqk\" (UID: \"1895db98-827c-4878-9434-be32c716a416\") " pod="openshift-ingress/e2e-inf-a-288815-openshift-default-7cc9fd9c89-2vrqk" Jul 09 22:51:57.122067 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:51:57.122063 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1895db98-827c-4878-9434-be32c716a416-istio-token\") pod \"e2e-inf-a-288815-openshift-default-7cc9fd9c89-2vrqk\" (UID: \"1895db98-827c-4878-9434-be32c716a416\") " pod="openshift-ingress/e2e-inf-a-288815-openshift-default-7cc9fd9c89-2vrqk" Jul 09 22:51:57.122445 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:51:57.122224 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1895db98-827c-4878-9434-be32c716a416-workload-socket\") pod \"e2e-inf-a-288815-openshift-default-7cc9fd9c89-2vrqk\" (UID: \"1895db98-827c-4878-9434-be32c716a416\") " pod="openshift-ingress/e2e-inf-a-288815-openshift-default-7cc9fd9c89-2vrqk" Jul 09 22:51:57.122445 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:51:57.122264 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1895db98-827c-4878-9434-be32c716a416-credential-socket\") pod \"e2e-inf-a-288815-openshift-default-7cc9fd9c89-2vrqk\" (UID: \"1895db98-827c-4878-9434-be32c716a416\") " pod="openshift-ingress/e2e-inf-a-288815-openshift-default-7cc9fd9c89-2vrqk" Jul 09 22:51:57.122547 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:51:57.122440 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1895db98-827c-4878-9434-be32c716a416-workload-certs\") pod \"e2e-inf-a-288815-openshift-default-7cc9fd9c89-2vrqk\" (UID: \"1895db98-827c-4878-9434-be32c716a416\") " pod="openshift-ingress/e2e-inf-a-288815-openshift-default-7cc9fd9c89-2vrqk" Jul 09 22:51:57.122547 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:51:57.122455 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1895db98-827c-4878-9434-be32c716a416-istio-data\") pod \"e2e-inf-a-288815-openshift-default-7cc9fd9c89-2vrqk\" (UID: \"1895db98-827c-4878-9434-be32c716a416\") " pod="openshift-ingress/e2e-inf-a-288815-openshift-default-7cc9fd9c89-2vrqk" Jul 09 22:51:57.122655 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:51:57.122623 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1895db98-827c-4878-9434-be32c716a416-istiod-ca-cert\") pod \"e2e-inf-a-288815-openshift-default-7cc9fd9c89-2vrqk\" (UID: \"1895db98-827c-4878-9434-be32c716a416\") " pod="openshift-ingress/e2e-inf-a-288815-openshift-default-7cc9fd9c89-2vrqk" Jul 09 22:51:57.124309 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:51:57.124289 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1895db98-827c-4878-9434-be32c716a416-istio-envoy\") pod \"e2e-inf-a-288815-openshift-default-7cc9fd9c89-2vrqk\" (UID: \"1895db98-827c-4878-9434-be32c716a416\") " pod="openshift-ingress/e2e-inf-a-288815-openshift-default-7cc9fd9c89-2vrqk" Jul 09 22:51:57.124582 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:51:57.124565 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1895db98-827c-4878-9434-be32c716a416-istio-podinfo\") pod \"e2e-inf-a-288815-openshift-default-7cc9fd9c89-2vrqk\" (UID: \"1895db98-827c-4878-9434-be32c716a416\") " pod="openshift-ingress/e2e-inf-a-288815-openshift-default-7cc9fd9c89-2vrqk" Jul 09 22:51:57.131200 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:51:57.131173 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1895db98-827c-4878-9434-be32c716a416-istio-token\") pod \"e2e-inf-a-288815-openshift-default-7cc9fd9c89-2vrqk\" (UID: \"1895db98-827c-4878-9434-be32c716a416\") " pod="openshift-ingress/e2e-inf-a-288815-openshift-default-7cc9fd9c89-2vrqk" Jul 09 22:51:57.131296 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:51:57.131243 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-trvb6\" (UniqueName: \"kubernetes.io/projected/1895db98-827c-4878-9434-be32c716a416-kube-api-access-trvb6\") pod \"e2e-inf-a-288815-openshift-default-7cc9fd9c89-2vrqk\" (UID: \"1895db98-827c-4878-9434-be32c716a416\") " pod="openshift-ingress/e2e-inf-a-288815-openshift-default-7cc9fd9c89-2vrqk" Jul 09 22:51:57.290320 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:51:57.290276 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-288815-openshift-default-7cc9fd9c89-2vrqk" Jul 09 22:51:57.437116 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:51:57.437084 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-288815-openshift-default-7cc9fd9c89-2vrqk"] Jul 09 22:51:57.439395 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:51:57.439361 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1895db98_827c_4878_9434_be32c716a416.slice/crio-f0bc77053741fafda75d64977afb2681c2143d369608f6c900ddd8697bb06e81 WatchSource:0}: Error finding container f0bc77053741fafda75d64977afb2681c2143d369608f6c900ddd8697bb06e81: Status 404 returned error can't find the container with id f0bc77053741fafda75d64977afb2681c2143d369608f6c900ddd8697bb06e81 Jul 09 22:51:57.444939 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:51:57.444888 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 09 22:51:57.445040 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:51:57.444990 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 09 22:51:57.445040 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:51:57.445030 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 09 22:51:58.402653 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:51:58.402608 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-288815-openshift-default-7cc9fd9c89-2vrqk" event={"ID":"1895db98-827c-4878-9434-be32c716a416","Type":"ContainerStarted","Data":"7b991832e5b264b79480cd6dc820b306c59dfc3317d2e8ccf1711dad46f25666"} Jul 09 22:51:58.403102 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:51:58.402707 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-288815-openshift-default-7cc9fd9c89-2vrqk" event={"ID":"1895db98-827c-4878-9434-be32c716a416","Type":"ContainerStarted","Data":"f0bc77053741fafda75d64977afb2681c2143d369608f6c900ddd8697bb06e81"} Jul 09 22:51:58.424206 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:51:58.424156 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-288815-openshift-default-7cc9fd9c89-2vrqk" podStartSLOduration=2.42414223 podStartE2EDuration="2.42414223s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 22:51:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 22:51:58.42301911 +0000 UTC m=+2510.334471828" watchObservedRunningTime="2026-07-09 22:51:58.42414223 +0000 UTC m=+2510.335594930" Jul 09 22:51:59.290857 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:51:59.290816 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-288815-openshift-default-7cc9fd9c89-2vrqk" Jul 09 22:51:59.295962 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:51:59.295936 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-288815-openshift-default-7cc9fd9c89-2vrqk" Jul 09 22:51:59.407800 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:51:59.407771 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-288815-openshift-default-7cc9fd9c89-2vrqk" Jul 09 22:51:59.408690 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:51:59.408671 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-288815-openshift-default-7cc9fd9c89-2vrqk" Jul 09 22:52:05.560700 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:52:05.560624 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-d2543e-openshift-default-55686d679c-nbqjg"] Jul 09 22:52:05.564498 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:52:05.564481 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-d2543e-openshift-default-55686d679c-nbqjg" Jul 09 22:52:05.567811 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:52:05.567790 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-d2543e-openshift-default-dockercfg-p28t6\"" Jul 09 22:52:05.578648 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:52:05.578622 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-d2543e-openshift-default-55686d679c-nbqjg"] Jul 09 22:52:05.594969 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:52:05.594936 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/04e62c1d-d3ec-4760-b5e1-d8d31a1e653d-credential-socket\") pod \"e2e-inf-b-d2543e-openshift-default-55686d679c-nbqjg\" (UID: \"04e62c1d-d3ec-4760-b5e1-d8d31a1e653d\") " pod="openshift-ingress/e2e-inf-b-d2543e-openshift-default-55686d679c-nbqjg" Jul 09 22:52:05.594969 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:52:05.594971 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/04e62c1d-d3ec-4760-b5e1-d8d31a1e653d-istio-envoy\") pod \"e2e-inf-b-d2543e-openshift-default-55686d679c-nbqjg\" (UID: \"04e62c1d-d3ec-4760-b5e1-d8d31a1e653d\") " pod="openshift-ingress/e2e-inf-b-d2543e-openshift-default-55686d679c-nbqjg" Jul 09 22:52:05.595171 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:52:05.594998 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/04e62c1d-d3ec-4760-b5e1-d8d31a1e653d-istio-token\") pod \"e2e-inf-b-d2543e-openshift-default-55686d679c-nbqjg\" (UID: \"04e62c1d-d3ec-4760-b5e1-d8d31a1e653d\") " pod="openshift-ingress/e2e-inf-b-d2543e-openshift-default-55686d679c-nbqjg" Jul 09 22:52:05.595171 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:52:05.595120 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/04e62c1d-d3ec-4760-b5e1-d8d31a1e653d-istiod-ca-cert\") pod \"e2e-inf-b-d2543e-openshift-default-55686d679c-nbqjg\" (UID: \"04e62c1d-d3ec-4760-b5e1-d8d31a1e653d\") " pod="openshift-ingress/e2e-inf-b-d2543e-openshift-default-55686d679c-nbqjg" Jul 09 22:52:05.595243 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:52:05.595198 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/04e62c1d-d3ec-4760-b5e1-d8d31a1e653d-istio-data\") pod \"e2e-inf-b-d2543e-openshift-default-55686d679c-nbqjg\" (UID: \"04e62c1d-d3ec-4760-b5e1-d8d31a1e653d\") " pod="openshift-ingress/e2e-inf-b-d2543e-openshift-default-55686d679c-nbqjg" Jul 09 22:52:05.595243 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:52:05.595232 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5pnf5\" (UniqueName: \"kubernetes.io/projected/04e62c1d-d3ec-4760-b5e1-d8d31a1e653d-kube-api-access-5pnf5\") pod \"e2e-inf-b-d2543e-openshift-default-55686d679c-nbqjg\" (UID: \"04e62c1d-d3ec-4760-b5e1-d8d31a1e653d\") " pod="openshift-ingress/e2e-inf-b-d2543e-openshift-default-55686d679c-nbqjg" Jul 09 22:52:05.595312 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:52:05.595262 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/04e62c1d-d3ec-4760-b5e1-d8d31a1e653d-workload-certs\") pod \"e2e-inf-b-d2543e-openshift-default-55686d679c-nbqjg\" (UID: \"04e62c1d-d3ec-4760-b5e1-d8d31a1e653d\") " pod="openshift-ingress/e2e-inf-b-d2543e-openshift-default-55686d679c-nbqjg" Jul 09 22:52:05.595312 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:52:05.595295 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/04e62c1d-d3ec-4760-b5e1-d8d31a1e653d-istio-podinfo\") pod \"e2e-inf-b-d2543e-openshift-default-55686d679c-nbqjg\" (UID: \"04e62c1d-d3ec-4760-b5e1-d8d31a1e653d\") " pod="openshift-ingress/e2e-inf-b-d2543e-openshift-default-55686d679c-nbqjg" Jul 09 22:52:05.595387 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:52:05.595371 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/04e62c1d-d3ec-4760-b5e1-d8d31a1e653d-workload-socket\") pod \"e2e-inf-b-d2543e-openshift-default-55686d679c-nbqjg\" (UID: \"04e62c1d-d3ec-4760-b5e1-d8d31a1e653d\") " pod="openshift-ingress/e2e-inf-b-d2543e-openshift-default-55686d679c-nbqjg" Jul 09 22:52:05.695810 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:52:05.695777 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/04e62c1d-d3ec-4760-b5e1-d8d31a1e653d-credential-socket\") pod \"e2e-inf-b-d2543e-openshift-default-55686d679c-nbqjg\" (UID: \"04e62c1d-d3ec-4760-b5e1-d8d31a1e653d\") " pod="openshift-ingress/e2e-inf-b-d2543e-openshift-default-55686d679c-nbqjg" Jul 09 22:52:05.695810 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:52:05.695810 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/04e62c1d-d3ec-4760-b5e1-d8d31a1e653d-istio-envoy\") pod \"e2e-inf-b-d2543e-openshift-default-55686d679c-nbqjg\" (UID: \"04e62c1d-d3ec-4760-b5e1-d8d31a1e653d\") " pod="openshift-ingress/e2e-inf-b-d2543e-openshift-default-55686d679c-nbqjg" Jul 09 22:52:05.696111 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:52:05.695833 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/04e62c1d-d3ec-4760-b5e1-d8d31a1e653d-istio-token\") pod \"e2e-inf-b-d2543e-openshift-default-55686d679c-nbqjg\" (UID: \"04e62c1d-d3ec-4760-b5e1-d8d31a1e653d\") " pod="openshift-ingress/e2e-inf-b-d2543e-openshift-default-55686d679c-nbqjg" Jul 09 22:52:05.696111 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:52:05.695871 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/04e62c1d-d3ec-4760-b5e1-d8d31a1e653d-istiod-ca-cert\") pod \"e2e-inf-b-d2543e-openshift-default-55686d679c-nbqjg\" (UID: \"04e62c1d-d3ec-4760-b5e1-d8d31a1e653d\") " pod="openshift-ingress/e2e-inf-b-d2543e-openshift-default-55686d679c-nbqjg" Jul 09 22:52:05.696111 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:52:05.695902 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/04e62c1d-d3ec-4760-b5e1-d8d31a1e653d-istio-data\") pod \"e2e-inf-b-d2543e-openshift-default-55686d679c-nbqjg\" (UID: \"04e62c1d-d3ec-4760-b5e1-d8d31a1e653d\") " pod="openshift-ingress/e2e-inf-b-d2543e-openshift-default-55686d679c-nbqjg" Jul 09 22:52:05.696111 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:52:05.695939 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5pnf5\" (UniqueName: \"kubernetes.io/projected/04e62c1d-d3ec-4760-b5e1-d8d31a1e653d-kube-api-access-5pnf5\") pod \"e2e-inf-b-d2543e-openshift-default-55686d679c-nbqjg\" (UID: \"04e62c1d-d3ec-4760-b5e1-d8d31a1e653d\") " pod="openshift-ingress/e2e-inf-b-d2543e-openshift-default-55686d679c-nbqjg" Jul 09 22:52:05.696333 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:52:05.696245 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/04e62c1d-d3ec-4760-b5e1-d8d31a1e653d-credential-socket\") pod \"e2e-inf-b-d2543e-openshift-default-55686d679c-nbqjg\" (UID: \"04e62c1d-d3ec-4760-b5e1-d8d31a1e653d\") " pod="openshift-ingress/e2e-inf-b-d2543e-openshift-default-55686d679c-nbqjg" Jul 09 22:52:05.696333 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:52:05.696303 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/04e62c1d-d3ec-4760-b5e1-d8d31a1e653d-workload-certs\") pod \"e2e-inf-b-d2543e-openshift-default-55686d679c-nbqjg\" (UID: \"04e62c1d-d3ec-4760-b5e1-d8d31a1e653d\") " pod="openshift-ingress/e2e-inf-b-d2543e-openshift-default-55686d679c-nbqjg" Jul 09 22:52:05.696443 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:52:05.696306 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/04e62c1d-d3ec-4760-b5e1-d8d31a1e653d-istio-data\") pod \"e2e-inf-b-d2543e-openshift-default-55686d679c-nbqjg\" (UID: \"04e62c1d-d3ec-4760-b5e1-d8d31a1e653d\") " pod="openshift-ingress/e2e-inf-b-d2543e-openshift-default-55686d679c-nbqjg" Jul 09 22:52:05.696443 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:52:05.696334 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/04e62c1d-d3ec-4760-b5e1-d8d31a1e653d-istio-podinfo\") pod \"e2e-inf-b-d2543e-openshift-default-55686d679c-nbqjg\" (UID: \"04e62c1d-d3ec-4760-b5e1-d8d31a1e653d\") " pod="openshift-ingress/e2e-inf-b-d2543e-openshift-default-55686d679c-nbqjg" Jul 09 22:52:05.696554 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:52:05.696439 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/04e62c1d-d3ec-4760-b5e1-d8d31a1e653d-workload-socket\") pod \"e2e-inf-b-d2543e-openshift-default-55686d679c-nbqjg\" (UID: \"04e62c1d-d3ec-4760-b5e1-d8d31a1e653d\") " pod="openshift-ingress/e2e-inf-b-d2543e-openshift-default-55686d679c-nbqjg" Jul 09 22:52:05.696554 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:52:05.696529 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/04e62c1d-d3ec-4760-b5e1-d8d31a1e653d-workload-certs\") pod \"e2e-inf-b-d2543e-openshift-default-55686d679c-nbqjg\" (UID: \"04e62c1d-d3ec-4760-b5e1-d8d31a1e653d\") " pod="openshift-ingress/e2e-inf-b-d2543e-openshift-default-55686d679c-nbqjg" Jul 09 22:52:05.696661 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:52:05.696644 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/04e62c1d-d3ec-4760-b5e1-d8d31a1e653d-istiod-ca-cert\") pod \"e2e-inf-b-d2543e-openshift-default-55686d679c-nbqjg\" (UID: \"04e62c1d-d3ec-4760-b5e1-d8d31a1e653d\") " pod="openshift-ingress/e2e-inf-b-d2543e-openshift-default-55686d679c-nbqjg" Jul 09 22:52:05.696840 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:52:05.696812 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/04e62c1d-d3ec-4760-b5e1-d8d31a1e653d-workload-socket\") pod \"e2e-inf-b-d2543e-openshift-default-55686d679c-nbqjg\" (UID: \"04e62c1d-d3ec-4760-b5e1-d8d31a1e653d\") " pod="openshift-ingress/e2e-inf-b-d2543e-openshift-default-55686d679c-nbqjg" Jul 09 22:52:05.698811 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:52:05.698792 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/04e62c1d-d3ec-4760-b5e1-d8d31a1e653d-istio-podinfo\") pod \"e2e-inf-b-d2543e-openshift-default-55686d679c-nbqjg\" (UID: \"04e62c1d-d3ec-4760-b5e1-d8d31a1e653d\") " pod="openshift-ingress/e2e-inf-b-d2543e-openshift-default-55686d679c-nbqjg" Jul 09 22:52:05.698939 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:52:05.698796 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/04e62c1d-d3ec-4760-b5e1-d8d31a1e653d-istio-envoy\") pod \"e2e-inf-b-d2543e-openshift-default-55686d679c-nbqjg\" (UID: \"04e62c1d-d3ec-4760-b5e1-d8d31a1e653d\") " pod="openshift-ingress/e2e-inf-b-d2543e-openshift-default-55686d679c-nbqjg" Jul 09 22:52:05.704549 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:52:05.704516 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/04e62c1d-d3ec-4760-b5e1-d8d31a1e653d-istio-token\") pod \"e2e-inf-b-d2543e-openshift-default-55686d679c-nbqjg\" (UID: \"04e62c1d-d3ec-4760-b5e1-d8d31a1e653d\") " pod="openshift-ingress/e2e-inf-b-d2543e-openshift-default-55686d679c-nbqjg" Jul 09 22:52:05.704709 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:52:05.704686 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5pnf5\" (UniqueName: \"kubernetes.io/projected/04e62c1d-d3ec-4760-b5e1-d8d31a1e653d-kube-api-access-5pnf5\") pod \"e2e-inf-b-d2543e-openshift-default-55686d679c-nbqjg\" (UID: \"04e62c1d-d3ec-4760-b5e1-d8d31a1e653d\") " pod="openshift-ingress/e2e-inf-b-d2543e-openshift-default-55686d679c-nbqjg" Jul 09 22:52:05.876407 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:52:05.876319 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-d2543e-openshift-default-55686d679c-nbqjg" Jul 09 22:52:06.019495 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:52:06.019464 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-d2543e-openshift-default-55686d679c-nbqjg"] Jul 09 22:52:06.022320 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:52:06.022291 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod04e62c1d_d3ec_4760_b5e1_d8d31a1e653d.slice/crio-00e996c8b434dfd435ffb0ba20f38d3557bfe4c7345dabd448e320af36b7a223 WatchSource:0}: Error finding container 00e996c8b434dfd435ffb0ba20f38d3557bfe4c7345dabd448e320af36b7a223: Status 404 returned error can't find the container with id 00e996c8b434dfd435ffb0ba20f38d3557bfe4c7345dabd448e320af36b7a223 Jul 09 22:52:06.024549 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:52:06.024519 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 09 22:52:06.024648 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:52:06.024585 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 09 22:52:06.024648 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:52:06.024615 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 09 22:52:06.457329 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:52:06.450358 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-d2543e-openshift-default-55686d679c-nbqjg" event={"ID":"04e62c1d-d3ec-4760-b5e1-d8d31a1e653d","Type":"ContainerStarted","Data":"2395f2a58268a546fe2e61fd0413065d2bdddfc0e7414ca2d8be8e654db64e96"} Jul 09 22:52:06.457329 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:52:06.450483 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-d2543e-openshift-default-55686d679c-nbqjg" event={"ID":"04e62c1d-d3ec-4760-b5e1-d8d31a1e653d","Type":"ContainerStarted","Data":"00e996c8b434dfd435ffb0ba20f38d3557bfe4c7345dabd448e320af36b7a223"} Jul 09 22:52:06.474339 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:52:06.474284 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-d2543e-openshift-default-55686d679c-nbqjg" podStartSLOduration=1.474266651 podStartE2EDuration="1.474266651s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 22:52:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 22:52:06.472240947 +0000 UTC m=+2518.383693648" watchObservedRunningTime="2026-07-09 22:52:06.474266651 +0000 UTC m=+2518.385719351" Jul 09 22:52:06.877053 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:52:06.877014 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-d2543e-openshift-default-55686d679c-nbqjg" Jul 09 22:52:07.717738 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:52:07.717700 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-xdfrr"] Jul 09 22:52:07.882316 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:52:07.882283 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-d2543e-openshift-default-55686d679c-nbqjg" Jul 09 22:52:08.464394 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:52:08.464358 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-d2543e-openshift-default-55686d679c-nbqjg" Jul 09 22:52:08.465547 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:52:08.465521 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-d2543e-openshift-default-55686d679c-nbqjg" Jul 09 22:52:15.451208 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:52:15.451159 2579 patch_prober.go:28] interesting pod/e2e-inf-a-288815-openshift-default-7cc9fd9c89-2vrqk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.48:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Jul 09 22:52:15.452106 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:52:15.451853 2579 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-288815-openshift-default-7cc9fd9c89-2vrqk" podUID="1895db98-827c-4878-9434-be32c716a416" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.48:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Jul 09 22:52:16.510319 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:52:16.510224 2579 generic.go:356] "Generic (PLEG): container finished" podID="1895db98-827c-4878-9434-be32c716a416" containerID="7b991832e5b264b79480cd6dc820b306c59dfc3317d2e8ccf1711dad46f25666" exitCode=0 Jul 09 22:52:16.510851 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:52:16.510318 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-288815-openshift-default-7cc9fd9c89-2vrqk" event={"ID":"1895db98-827c-4878-9434-be32c716a416","Type":"ContainerDied","Data":"7b991832e5b264b79480cd6dc820b306c59dfc3317d2e8ccf1711dad46f25666"} Jul 09 22:52:16.510851 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:52:16.510780 2579 scope.go:117] "RemoveContainer" containerID="7b991832e5b264b79480cd6dc820b306c59dfc3317d2e8ccf1711dad46f25666" Jul 09 22:52:16.511991 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:52:16.511961 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 09 22:52:16.512065 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:52:16.512050 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 09 22:52:16.512155 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:52:16.512099 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 09 22:52:17.517206 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:52:17.517169 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-288815-openshift-default-7cc9fd9c89-2vrqk" event={"ID":"1895db98-827c-4878-9434-be32c716a416","Type":"ContainerStarted","Data":"de553b26035602e3b5b872d7a6f98d090201f3f7171fbe80f064f664730f0a32"} Jul 09 22:52:18.291370 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:52:18.291325 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-288815-openshift-default-7cc9fd9c89-2vrqk" Jul 09 22:52:18.291567 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:52:18.291464 2579 patch_prober.go:28] interesting pod/e2e-inf-a-288815-openshift-default-7cc9fd9c89-2vrqk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused" start-of-body= Jul 09 22:52:18.291567 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:52:18.291510 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-288815-openshift-default-7cc9fd9c89-2vrqk" podUID="1895db98-827c-4878-9434-be32c716a416" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused" Jul 09 22:52:18.915710 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:52:18.915675 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-xdfrr"] Jul 09 22:52:19.291828 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:52:19.291776 2579 patch_prober.go:28] interesting pod/e2e-inf-a-288815-openshift-default-7cc9fd9c89-2vrqk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused" start-of-body= Jul 09 22:52:19.292072 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:52:19.291873 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-288815-openshift-default-7cc9fd9c89-2vrqk" podUID="1895db98-827c-4878-9434-be32c716a416" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused" Jul 09 22:52:20.291181 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:52:20.291138 2579 patch_prober.go:28] interesting pod/e2e-inf-a-288815-openshift-default-7cc9fd9c89-2vrqk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused" start-of-body= Jul 09 22:52:20.291604 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:52:20.291207 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-288815-openshift-default-7cc9fd9c89-2vrqk" podUID="1895db98-827c-4878-9434-be32c716a416" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused" Jul 09 22:52:21.291751 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:52:21.291696 2579 patch_prober.go:28] interesting pod/e2e-inf-a-288815-openshift-default-7cc9fd9c89-2vrqk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused" start-of-body= Jul 09 22:52:21.292296 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:52:21.291760 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-288815-openshift-default-7cc9fd9c89-2vrqk" podUID="1895db98-827c-4878-9434-be32c716a416" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused" Jul 09 22:52:22.294602 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:52:22.294565 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-288815-openshift-default-7cc9fd9c89-2vrqk" Jul 09 22:52:22.295152 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:52:22.294836 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-288815-openshift-default-7cc9fd9c89-2vrqk" Jul 09 22:52:22.295672 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:52:22.295652 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-288815-openshift-default-7cc9fd9c89-2vrqk" Jul 09 22:52:23.466686 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:52:23.466647 2579 patch_prober.go:28] interesting pod/e2e-inf-b-d2543e-openshift-default-55686d679c-nbqjg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused" start-of-body= Jul 09 22:52:23.467155 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:52:23.466709 2579 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-d2543e-openshift-default-55686d679c-nbqjg" podUID="04e62c1d-d3ec-4760-b5e1-d8d31a1e653d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused" Jul 09 22:52:23.553059 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:52:23.553018 2579 generic.go:356] "Generic (PLEG): container finished" podID="04e62c1d-d3ec-4760-b5e1-d8d31a1e653d" containerID="2395f2a58268a546fe2e61fd0413065d2bdddfc0e7414ca2d8be8e654db64e96" exitCode=0 Jul 09 22:52:23.553256 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:52:23.553098 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-d2543e-openshift-default-55686d679c-nbqjg" event={"ID":"04e62c1d-d3ec-4760-b5e1-d8d31a1e653d","Type":"ContainerDied","Data":"2395f2a58268a546fe2e61fd0413065d2bdddfc0e7414ca2d8be8e654db64e96"} Jul 09 22:52:23.553621 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:52:23.553604 2579 scope.go:117] "RemoveContainer" containerID="2395f2a58268a546fe2e61fd0413065d2bdddfc0e7414ca2d8be8e654db64e96" Jul 09 22:52:23.554590 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:52:23.554562 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 09 22:52:23.554645 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:52:23.554631 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 09 22:52:23.554684 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:52:23.554663 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 09 22:52:24.563543 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:52:24.563498 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-d2543e-openshift-default-55686d679c-nbqjg" event={"ID":"04e62c1d-d3ec-4760-b5e1-d8d31a1e653d","Type":"ContainerStarted","Data":"f24ea49287d0dd2d4a2ec6a96a172cd41a8989771da7e7d4499a78210ef17d35"} Jul 09 22:52:24.876857 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:52:24.876742 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-d2543e-openshift-default-55686d679c-nbqjg" Jul 09 22:52:24.877182 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:52:24.877157 2579 patch_prober.go:28] interesting pod/e2e-inf-b-d2543e-openshift-default-55686d679c-nbqjg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused" start-of-body= Jul 09 22:52:24.877288 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:52:24.877213 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-d2543e-openshift-default-55686d679c-nbqjg" podUID="04e62c1d-d3ec-4760-b5e1-d8d31a1e653d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused" Jul 09 22:52:25.877061 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:52:25.877030 2579 patch_prober.go:28] interesting pod/e2e-inf-b-d2543e-openshift-default-55686d679c-nbqjg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused" start-of-body= Jul 09 22:52:25.877444 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:52:25.877093 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-d2543e-openshift-default-55686d679c-nbqjg" podUID="04e62c1d-d3ec-4760-b5e1-d8d31a1e653d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused" Jul 09 22:52:26.877280 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:52:26.877243 2579 patch_prober.go:28] interesting pod/e2e-inf-b-d2543e-openshift-default-55686d679c-nbqjg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused" start-of-body= Jul 09 22:52:26.877781 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:52:26.877307 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-d2543e-openshift-default-55686d679c-nbqjg" podUID="04e62c1d-d3ec-4760-b5e1-d8d31a1e653d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused" Jul 09 22:52:27.877574 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:52:27.877536 2579 patch_prober.go:28] interesting pod/e2e-inf-b-d2543e-openshift-default-55686d679c-nbqjg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused" start-of-body= Jul 09 22:52:27.878043 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:52:27.877593 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-d2543e-openshift-default-55686d679c-nbqjg" podUID="04e62c1d-d3ec-4760-b5e1-d8d31a1e653d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused" Jul 09 22:52:28.880948 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:52:28.880884 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-d2543e-openshift-default-55686d679c-nbqjg" Jul 09 22:52:28.881426 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:52:28.881316 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-d2543e-openshift-default-55686d679c-nbqjg" Jul 09 22:52:28.881907 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:52:28.881887 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-d2543e-openshift-default-55686d679c-nbqjg" Jul 09 22:52:36.817371 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:52:36.817329 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-xdfrr"] Jul 09 22:53:15.215760 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:53:15.215718 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-xdfrr"] Jul 09 22:53:29.511308 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:53:29.511267 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-xdfrr"] Jul 09 22:53:32.946140 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:53:32.946042 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-288815-n4kpm"] Jul 09 22:53:32.949869 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:53:32.949849 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-288815-n4kpm" Jul 09 22:53:32.952274 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:53:32.952251 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fdzhc\"" Jul 09 22:53:32.952370 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:53:32.952272 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 09 22:53:32.956320 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:53:32.956293 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-288815-n4kpm"] Jul 09 22:53:33.026028 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:53:33.025988 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9bc7fad2-3dad-4e26-a232-cfb0a2739fc4-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-288815-n4kpm\" (UID: \"9bc7fad2-3dad-4e26-a232-cfb0a2739fc4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-288815-n4kpm" Jul 09 22:53:33.126599 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:53:33.126552 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9bc7fad2-3dad-4e26-a232-cfb0a2739fc4-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-288815-n4kpm\" (UID: \"9bc7fad2-3dad-4e26-a232-cfb0a2739fc4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-288815-n4kpm" Jul 09 22:53:33.127281 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:53:33.127260 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9bc7fad2-3dad-4e26-a232-cfb0a2739fc4-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-288815-n4kpm\" (UID: \"9bc7fad2-3dad-4e26-a232-cfb0a2739fc4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-288815-n4kpm" Jul 09 22:53:33.262068 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:53:33.262029 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-288815-n4kpm" Jul 09 22:53:33.404001 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:53:33.403974 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-288815-n4kpm"] Jul 09 22:53:33.405829 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:53:33.405804 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9bc7fad2_3dad_4e26_a232_cfb0a2739fc4.slice/crio-27b20e94946c7249b36dbfb866659b5fc49059a1d6e042aee04b6699c94e9651 WatchSource:0}: Error finding container 27b20e94946c7249b36dbfb866659b5fc49059a1d6e042aee04b6699c94e9651: Status 404 returned error can't find the container with id 27b20e94946c7249b36dbfb866659b5fc49059a1d6e042aee04b6699c94e9651 Jul 09 22:53:33.941534 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:53:33.941442 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-288815-n4kpm" event={"ID":"9bc7fad2-3dad-4e26-a232-cfb0a2739fc4","Type":"ContainerStarted","Data":"084e927baac3222821ce3ea321b606596927462ef89784dc7ccf2e258fec45b2"} Jul 09 22:53:33.941534 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:53:33.941505 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-288815-n4kpm" event={"ID":"9bc7fad2-3dad-4e26-a232-cfb0a2739fc4","Type":"ContainerStarted","Data":"27b20e94946c7249b36dbfb866659b5fc49059a1d6e042aee04b6699c94e9651"} Jul 09 22:53:33.958311 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:53:33.958255 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-288815-n4kpm" podStartSLOduration=1.958235975 podStartE2EDuration="1.958235975s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 22:53:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 22:53:33.955813362 +0000 UTC m=+2605.867266062" watchObservedRunningTime="2026-07-09 22:53:33.958235975 +0000 UTC m=+2605.869688676" Jul 09 22:53:34.948376 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:53:34.948340 2579 generic.go:356] "Generic (PLEG): container finished" podID="9bc7fad2-3dad-4e26-a232-cfb0a2739fc4" containerID="084e927baac3222821ce3ea321b606596927462ef89784dc7ccf2e258fec45b2" exitCode=0 Jul 09 22:53:34.948538 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:53:34.948424 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-288815-n4kpm" event={"ID":"9bc7fad2-3dad-4e26-a232-cfb0a2739fc4","Type":"ContainerDied","Data":"084e927baac3222821ce3ea321b606596927462ef89784dc7ccf2e258fec45b2"} Jul 09 22:53:36.092304 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:53:36.092278 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-288815-n4kpm" Jul 09 22:53:36.153945 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:53:36.153893 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9bc7fad2-3dad-4e26-a232-cfb0a2739fc4-maas-api-service-ca\") pod \"9bc7fad2-3dad-4e26-a232-cfb0a2739fc4\" (UID: \"9bc7fad2-3dad-4e26-a232-cfb0a2739fc4\") " Jul 09 22:53:36.154284 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:53:36.154260 2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9bc7fad2-3dad-4e26-a232-cfb0a2739fc4-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "9bc7fad2-3dad-4e26-a232-cfb0a2739fc4" (UID: "9bc7fad2-3dad-4e26-a232-cfb0a2739fc4"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 09 22:53:36.255073 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:53:36.255036 2579 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9bc7fad2-3dad-4e26-a232-cfb0a2739fc4-maas-api-service-ca\") on node \"ip-10-0-136-251.ec2.internal\" DevicePath \"\"" Jul 09 22:53:36.961389 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:53:36.961293 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-288815-n4kpm" Jul 09 22:53:36.961389 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:53:36.961331 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-288815-n4kpm" event={"ID":"9bc7fad2-3dad-4e26-a232-cfb0a2739fc4","Type":"ContainerDied","Data":"27b20e94946c7249b36dbfb866659b5fc49059a1d6e042aee04b6699c94e9651"} Jul 09 22:53:36.961389 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:53:36.961375 2579 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="27b20e94946c7249b36dbfb866659b5fc49059a1d6e042aee04b6699c94e9651" Jul 09 22:53:46.315542 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:53:46.315504 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-xdfrr"] Jul 09 22:53:58.697933 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:53:58.697881 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-288815-openshift-default-7cc9fd9c89-2vrqk"] Jul 09 22:53:58.698434 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:53:58.698101 2579 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-288815-openshift-default-7cc9fd9c89-2vrqk" podUID="1895db98-827c-4878-9434-be32c716a416" containerName="istio-proxy" containerID="cri-o://de553b26035602e3b5b872d7a6f98d090201f3f7171fbe80f064f664730f0a32" gracePeriod=30 Jul 09 22:54:03.951545 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:54:03.951514 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-288815-openshift-default-7cc9fd9c89-2vrqk" Jul 09 22:54:04.106744 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:54:04.106711 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1895db98-827c-4878-9434-be32c716a416-istiod-ca-cert\") pod \"1895db98-827c-4878-9434-be32c716a416\" (UID: \"1895db98-827c-4878-9434-be32c716a416\") " Jul 09 22:54:04.106972 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:54:04.106761 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-trvb6\" (UniqueName: \"kubernetes.io/projected/1895db98-827c-4878-9434-be32c716a416-kube-api-access-trvb6\") pod \"1895db98-827c-4878-9434-be32c716a416\" (UID: \"1895db98-827c-4878-9434-be32c716a416\") " Jul 09 22:54:04.106972 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:54:04.106786 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1895db98-827c-4878-9434-be32c716a416-credential-socket\") pod \"1895db98-827c-4878-9434-be32c716a416\" (UID: \"1895db98-827c-4878-9434-be32c716a416\") " Jul 09 22:54:04.106972 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:54:04.106808 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1895db98-827c-4878-9434-be32c716a416-istio-data\") pod \"1895db98-827c-4878-9434-be32c716a416\" (UID: \"1895db98-827c-4878-9434-be32c716a416\") " Jul 09 22:54:04.106972 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:54:04.106870 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1895db98-827c-4878-9434-be32c716a416-istio-token\") pod \"1895db98-827c-4878-9434-be32c716a416\" (UID: \"1895db98-827c-4878-9434-be32c716a416\") " Jul 09 22:54:04.106972 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:54:04.106914 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1895db98-827c-4878-9434-be32c716a416-workload-socket\") pod \"1895db98-827c-4878-9434-be32c716a416\" (UID: \"1895db98-827c-4878-9434-be32c716a416\") " Jul 09 22:54:04.106972 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:54:04.106953 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1895db98-827c-4878-9434-be32c716a416-istio-podinfo\") pod \"1895db98-827c-4878-9434-be32c716a416\" (UID: \"1895db98-827c-4878-9434-be32c716a416\") " Jul 09 22:54:04.107356 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:54:04.106997 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1895db98-827c-4878-9434-be32c716a416-istio-envoy\") pod \"1895db98-827c-4878-9434-be32c716a416\" (UID: \"1895db98-827c-4878-9434-be32c716a416\") " Jul 09 22:54:04.107356 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:54:04.107023 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1895db98-827c-4878-9434-be32c716a416-workload-certs\") pod \"1895db98-827c-4878-9434-be32c716a416\" (UID: \"1895db98-827c-4878-9434-be32c716a416\") " Jul 09 22:54:04.107356 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:54:04.107238 2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1895db98-827c-4878-9434-be32c716a416-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "1895db98-827c-4878-9434-be32c716a416" (UID: "1895db98-827c-4878-9434-be32c716a416"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 22:54:04.107356 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:54:04.107158 2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1895db98-827c-4878-9434-be32c716a416-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "1895db98-827c-4878-9434-be32c716a416" (UID: "1895db98-827c-4878-9434-be32c716a416"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 09 22:54:04.107563 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:54:04.107466 2579 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1895db98-827c-4878-9434-be32c716a416-workload-socket\") on node \"ip-10-0-136-251.ec2.internal\" DevicePath \"\"" Jul 09 22:54:04.107563 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:54:04.107487 2579 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1895db98-827c-4878-9434-be32c716a416-istiod-ca-cert\") on node \"ip-10-0-136-251.ec2.internal\" DevicePath \"\"" Jul 09 22:54:04.107708 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:54:04.107679 2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1895db98-827c-4878-9434-be32c716a416-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "1895db98-827c-4878-9434-be32c716a416" (UID: "1895db98-827c-4878-9434-be32c716a416"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 22:54:04.107857 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:54:04.107725 2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1895db98-827c-4878-9434-be32c716a416-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "1895db98-827c-4878-9434-be32c716a416" (UID: "1895db98-827c-4878-9434-be32c716a416"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 22:54:04.108089 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:54:04.108059 2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1895db98-827c-4878-9434-be32c716a416-istio-data" (OuterVolumeSpecName: "istio-data") pod "1895db98-827c-4878-9434-be32c716a416" (UID: "1895db98-827c-4878-9434-be32c716a416"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 22:54:04.108194 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:54:04.108137 2579 generic.go:356] "Generic (PLEG): container finished" podID="1895db98-827c-4878-9434-be32c716a416" containerID="de553b26035602e3b5b872d7a6f98d090201f3f7171fbe80f064f664730f0a32" exitCode=0 Jul 09 22:54:04.108259 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:54:04.108220 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-288815-openshift-default-7cc9fd9c89-2vrqk" event={"ID":"1895db98-827c-4878-9434-be32c716a416","Type":"ContainerDied","Data":"de553b26035602e3b5b872d7a6f98d090201f3f7171fbe80f064f664730f0a32"} Jul 09 22:54:04.108318 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:54:04.108277 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-288815-openshift-default-7cc9fd9c89-2vrqk" event={"ID":"1895db98-827c-4878-9434-be32c716a416","Type":"ContainerDied","Data":"f0bc77053741fafda75d64977afb2681c2143d369608f6c900ddd8697bb06e81"} Jul 09 22:54:04.108318 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:54:04.108308 2579 scope.go:117] "RemoveContainer" containerID="de553b26035602e3b5b872d7a6f98d090201f3f7171fbe80f064f664730f0a32" Jul 09 22:54:04.108505 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:54:04.108485 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-288815-openshift-default-7cc9fd9c89-2vrqk" Jul 09 22:54:04.109512 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:54:04.109451 2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1895db98-827c-4878-9434-be32c716a416-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "1895db98-827c-4878-9434-be32c716a416" (UID: "1895db98-827c-4878-9434-be32c716a416"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 09 22:54:04.109512 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:54:04.109492 2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1895db98-827c-4878-9434-be32c716a416-kube-api-access-trvb6" (OuterVolumeSpecName: "kube-api-access-trvb6") pod "1895db98-827c-4878-9434-be32c716a416" (UID: "1895db98-827c-4878-9434-be32c716a416"). InnerVolumeSpecName "kube-api-access-trvb6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 22:54:04.109681 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:54:04.109556 2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1895db98-827c-4878-9434-be32c716a416-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "1895db98-827c-4878-9434-be32c716a416" (UID: "1895db98-827c-4878-9434-be32c716a416"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 22:54:04.109812 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:54:04.109727 2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1895db98-827c-4878-9434-be32c716a416-istio-token" (OuterVolumeSpecName: "istio-token") pod "1895db98-827c-4878-9434-be32c716a416" (UID: "1895db98-827c-4878-9434-be32c716a416"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 22:54:04.129251 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:54:04.129224 2579 scope.go:117] "RemoveContainer" containerID="7b991832e5b264b79480cd6dc820b306c59dfc3317d2e8ccf1711dad46f25666" Jul 09 22:54:04.137910 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:54:04.137888 2579 scope.go:117] "RemoveContainer" containerID="de553b26035602e3b5b872d7a6f98d090201f3f7171fbe80f064f664730f0a32" Jul 09 22:54:04.138323 ip-10-0-136-251 kubenswrapper[2579]: E0709 22:54:04.138298 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"de553b26035602e3b5b872d7a6f98d090201f3f7171fbe80f064f664730f0a32\": container with ID starting with de553b26035602e3b5b872d7a6f98d090201f3f7171fbe80f064f664730f0a32 not found: ID does not exist" containerID="de553b26035602e3b5b872d7a6f98d090201f3f7171fbe80f064f664730f0a32" Jul 09 22:54:04.138388 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:54:04.138333 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"de553b26035602e3b5b872d7a6f98d090201f3f7171fbe80f064f664730f0a32"} err="failed to get container status \"de553b26035602e3b5b872d7a6f98d090201f3f7171fbe80f064f664730f0a32\": rpc error: code = NotFound desc = could not find container \"de553b26035602e3b5b872d7a6f98d090201f3f7171fbe80f064f664730f0a32\": container with ID starting with de553b26035602e3b5b872d7a6f98d090201f3f7171fbe80f064f664730f0a32 not found: ID does not exist" Jul 09 22:54:04.138388 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:54:04.138354 2579 scope.go:117] "RemoveContainer" containerID="7b991832e5b264b79480cd6dc820b306c59dfc3317d2e8ccf1711dad46f25666" Jul 09 22:54:04.138630 ip-10-0-136-251 kubenswrapper[2579]: E0709 22:54:04.138609 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7b991832e5b264b79480cd6dc820b306c59dfc3317d2e8ccf1711dad46f25666\": container with ID starting with 7b991832e5b264b79480cd6dc820b306c59dfc3317d2e8ccf1711dad46f25666 not found: ID does not exist" containerID="7b991832e5b264b79480cd6dc820b306c59dfc3317d2e8ccf1711dad46f25666" Jul 09 22:54:04.138670 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:54:04.138635 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7b991832e5b264b79480cd6dc820b306c59dfc3317d2e8ccf1711dad46f25666"} err="failed to get container status \"7b991832e5b264b79480cd6dc820b306c59dfc3317d2e8ccf1711dad46f25666\": rpc error: code = NotFound desc = could not find container \"7b991832e5b264b79480cd6dc820b306c59dfc3317d2e8ccf1711dad46f25666\": container with ID starting with 7b991832e5b264b79480cd6dc820b306c59dfc3317d2e8ccf1711dad46f25666 not found: ID does not exist" Jul 09 22:54:04.208812 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:54:04.208775 2579 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1895db98-827c-4878-9434-be32c716a416-istio-token\") on node \"ip-10-0-136-251.ec2.internal\" DevicePath \"\"" Jul 09 22:54:04.208812 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:54:04.208806 2579 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1895db98-827c-4878-9434-be32c716a416-istio-podinfo\") on node \"ip-10-0-136-251.ec2.internal\" DevicePath \"\"" Jul 09 22:54:04.208812 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:54:04.208815 2579 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1895db98-827c-4878-9434-be32c716a416-istio-envoy\") on node \"ip-10-0-136-251.ec2.internal\" DevicePath \"\"" Jul 09 22:54:04.209084 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:54:04.208826 2579 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1895db98-827c-4878-9434-be32c716a416-workload-certs\") on node \"ip-10-0-136-251.ec2.internal\" DevicePath \"\"" Jul 09 22:54:04.209084 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:54:04.208834 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-trvb6\" (UniqueName: \"kubernetes.io/projected/1895db98-827c-4878-9434-be32c716a416-kube-api-access-trvb6\") on node \"ip-10-0-136-251.ec2.internal\" DevicePath \"\"" Jul 09 22:54:04.209084 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:54:04.208843 2579 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1895db98-827c-4878-9434-be32c716a416-credential-socket\") on node \"ip-10-0-136-251.ec2.internal\" DevicePath \"\"" Jul 09 22:54:04.209084 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:54:04.208851 2579 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1895db98-827c-4878-9434-be32c716a416-istio-data\") on node \"ip-10-0-136-251.ec2.internal\" DevicePath \"\"" Jul 09 22:54:04.431471 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:54:04.431432 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-288815-openshift-default-7cc9fd9c89-2vrqk"] Jul 09 22:54:04.436617 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:54:04.436589 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-288815-openshift-default-7cc9fd9c89-2vrqk"] Jul 09 22:54:04.564405 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:54:04.564369 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1895db98-827c-4878-9434-be32c716a416" path="/var/lib/kubelet/pods/1895db98-827c-4878-9434-be32c716a416/volumes" Jul 09 22:54:13.015554 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:54:13.015514 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-xdfrr"] Jul 09 22:54:24.827569 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:54:24.827526 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-d2543e-openshift-default-55686d679c-nbqjg"] Jul 09 22:54:24.828234 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:54:24.827792 2579 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-d2543e-openshift-default-55686d679c-nbqjg" podUID="04e62c1d-d3ec-4760-b5e1-d8d31a1e653d" containerName="istio-proxy" containerID="cri-o://f24ea49287d0dd2d4a2ec6a96a172cd41a8989771da7e7d4499a78210ef17d35" gracePeriod=30 Jul 09 22:54:25.853813 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:54:25.853776 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-e98c97-openshift-default-58df5648d5-ljkpq"] Jul 09 22:54:25.854240 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:54:25.854224 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1895db98-827c-4878-9434-be32c716a416" containerName="istio-proxy" Jul 09 22:54:25.854240 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:54:25.854241 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="1895db98-827c-4878-9434-be32c716a416" containerName="istio-proxy" Jul 09 22:54:25.854329 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:54:25.854249 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1895db98-827c-4878-9434-be32c716a416" containerName="istio-proxy" Jul 09 22:54:25.854329 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:54:25.854255 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="1895db98-827c-4878-9434-be32c716a416" containerName="istio-proxy" Jul 09 22:54:25.854329 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:54:25.854262 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9bc7fad2-3dad-4e26-a232-cfb0a2739fc4" containerName="revoke-keys" Jul 09 22:54:25.854329 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:54:25.854268 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="9bc7fad2-3dad-4e26-a232-cfb0a2739fc4" containerName="revoke-keys" Jul 09 22:54:25.854329 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:54:25.854328 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="1895db98-827c-4878-9434-be32c716a416" containerName="istio-proxy" Jul 09 22:54:25.854492 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:54:25.854337 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="1895db98-827c-4878-9434-be32c716a416" containerName="istio-proxy" Jul 09 22:54:25.854492 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:54:25.854370 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="9bc7fad2-3dad-4e26-a232-cfb0a2739fc4" containerName="revoke-keys" Jul 09 22:54:25.857520 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:54:25.857504 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-e98c97-openshift-default-58df5648d5-ljkpq" Jul 09 22:54:25.860503 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:54:25.860480 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-e98c97-openshift-default-dockercfg-j2fcv\"" Jul 09 22:54:25.880256 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:54:25.880220 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-e98c97-openshift-default-58df5648d5-ljkpq"] Jul 09 22:54:26.008978 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:54:26.008900 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5d550c17-9bf5-4730-bbe0-6f2305a0a90d-istio-data\") pod \"e2e-api-a-e98c97-openshift-default-58df5648d5-ljkpq\" (UID: \"5d550c17-9bf5-4730-bbe0-6f2305a0a90d\") " pod="openshift-ingress/e2e-api-a-e98c97-openshift-default-58df5648d5-ljkpq" Jul 09 22:54:26.009173 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:54:26.009015 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d550c17-9bf5-4730-bbe0-6f2305a0a90d-credential-socket\") pod \"e2e-api-a-e98c97-openshift-default-58df5648d5-ljkpq\" (UID: \"5d550c17-9bf5-4730-bbe0-6f2305a0a90d\") " pod="openshift-ingress/e2e-api-a-e98c97-openshift-default-58df5648d5-ljkpq" Jul 09 22:54:26.009173 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:54:26.009037 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5d550c17-9bf5-4730-bbe0-6f2305a0a90d-istio-envoy\") pod \"e2e-api-a-e98c97-openshift-default-58df5648d5-ljkpq\" (UID: \"5d550c17-9bf5-4730-bbe0-6f2305a0a90d\") " pod="openshift-ingress/e2e-api-a-e98c97-openshift-default-58df5648d5-ljkpq" Jul 09 22:54:26.009173 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:54:26.009087 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5d550c17-9bf5-4730-bbe0-6f2305a0a90d-istiod-ca-cert\") pod \"e2e-api-a-e98c97-openshift-default-58df5648d5-ljkpq\" (UID: \"5d550c17-9bf5-4730-bbe0-6f2305a0a90d\") " pod="openshift-ingress/e2e-api-a-e98c97-openshift-default-58df5648d5-ljkpq" Jul 09 22:54:26.009173 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:54:26.009107 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d550c17-9bf5-4730-bbe0-6f2305a0a90d-workload-socket\") pod \"e2e-api-a-e98c97-openshift-default-58df5648d5-ljkpq\" (UID: \"5d550c17-9bf5-4730-bbe0-6f2305a0a90d\") " pod="openshift-ingress/e2e-api-a-e98c97-openshift-default-58df5648d5-ljkpq" Jul 09 22:54:26.009173 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:54:26.009127 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5d550c17-9bf5-4730-bbe0-6f2305a0a90d-istio-token\") pod \"e2e-api-a-e98c97-openshift-default-58df5648d5-ljkpq\" (UID: \"5d550c17-9bf5-4730-bbe0-6f2305a0a90d\") " pod="openshift-ingress/e2e-api-a-e98c97-openshift-default-58df5648d5-ljkpq" Jul 09 22:54:26.009173 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:54:26.009154 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5d550c17-9bf5-4730-bbe0-6f2305a0a90d-workload-certs\") pod \"e2e-api-a-e98c97-openshift-default-58df5648d5-ljkpq\" (UID: \"5d550c17-9bf5-4730-bbe0-6f2305a0a90d\") " pod="openshift-ingress/e2e-api-a-e98c97-openshift-default-58df5648d5-ljkpq" Jul 09 22:54:26.009173 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:54:26.009175 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5d550c17-9bf5-4730-bbe0-6f2305a0a90d-istio-podinfo\") pod \"e2e-api-a-e98c97-openshift-default-58df5648d5-ljkpq\" (UID: \"5d550c17-9bf5-4730-bbe0-6f2305a0a90d\") " pod="openshift-ingress/e2e-api-a-e98c97-openshift-default-58df5648d5-ljkpq" Jul 09 22:54:26.009408 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:54:26.009193 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tdgqc\" (UniqueName: \"kubernetes.io/projected/5d550c17-9bf5-4730-bbe0-6f2305a0a90d-kube-api-access-tdgqc\") pod \"e2e-api-a-e98c97-openshift-default-58df5648d5-ljkpq\" (UID: \"5d550c17-9bf5-4730-bbe0-6f2305a0a90d\") " pod="openshift-ingress/e2e-api-a-e98c97-openshift-default-58df5648d5-ljkpq" Jul 09 22:54:26.110708 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:54:26.110620 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5d550c17-9bf5-4730-bbe0-6f2305a0a90d-istiod-ca-cert\") pod \"e2e-api-a-e98c97-openshift-default-58df5648d5-ljkpq\" (UID: \"5d550c17-9bf5-4730-bbe0-6f2305a0a90d\") " pod="openshift-ingress/e2e-api-a-e98c97-openshift-default-58df5648d5-ljkpq" Jul 09 22:54:26.110708 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:54:26.110658 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d550c17-9bf5-4730-bbe0-6f2305a0a90d-workload-socket\") pod \"e2e-api-a-e98c97-openshift-default-58df5648d5-ljkpq\" (UID: \"5d550c17-9bf5-4730-bbe0-6f2305a0a90d\") " pod="openshift-ingress/e2e-api-a-e98c97-openshift-default-58df5648d5-ljkpq" Jul 09 22:54:26.110708 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:54:26.110685 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5d550c17-9bf5-4730-bbe0-6f2305a0a90d-istio-token\") pod \"e2e-api-a-e98c97-openshift-default-58df5648d5-ljkpq\" (UID: \"5d550c17-9bf5-4730-bbe0-6f2305a0a90d\") " pod="openshift-ingress/e2e-api-a-e98c97-openshift-default-58df5648d5-ljkpq" Jul 09 22:54:26.111045 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:54:26.110711 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5d550c17-9bf5-4730-bbe0-6f2305a0a90d-workload-certs\") pod \"e2e-api-a-e98c97-openshift-default-58df5648d5-ljkpq\" (UID: \"5d550c17-9bf5-4730-bbe0-6f2305a0a90d\") " pod="openshift-ingress/e2e-api-a-e98c97-openshift-default-58df5648d5-ljkpq" Jul 09 22:54:26.111045 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:54:26.110732 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5d550c17-9bf5-4730-bbe0-6f2305a0a90d-istio-podinfo\") pod \"e2e-api-a-e98c97-openshift-default-58df5648d5-ljkpq\" (UID: \"5d550c17-9bf5-4730-bbe0-6f2305a0a90d\") " pod="openshift-ingress/e2e-api-a-e98c97-openshift-default-58df5648d5-ljkpq" Jul 09 22:54:26.111045 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:54:26.110756 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tdgqc\" (UniqueName: \"kubernetes.io/projected/5d550c17-9bf5-4730-bbe0-6f2305a0a90d-kube-api-access-tdgqc\") pod \"e2e-api-a-e98c97-openshift-default-58df5648d5-ljkpq\" (UID: \"5d550c17-9bf5-4730-bbe0-6f2305a0a90d\") " pod="openshift-ingress/e2e-api-a-e98c97-openshift-default-58df5648d5-ljkpq" Jul 09 22:54:26.111045 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:54:26.110818 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5d550c17-9bf5-4730-bbe0-6f2305a0a90d-istio-data\") pod \"e2e-api-a-e98c97-openshift-default-58df5648d5-ljkpq\" (UID: \"5d550c17-9bf5-4730-bbe0-6f2305a0a90d\") " pod="openshift-ingress/e2e-api-a-e98c97-openshift-default-58df5648d5-ljkpq" Jul 09 22:54:26.111045 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:54:26.110865 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d550c17-9bf5-4730-bbe0-6f2305a0a90d-credential-socket\") pod \"e2e-api-a-e98c97-openshift-default-58df5648d5-ljkpq\" (UID: \"5d550c17-9bf5-4730-bbe0-6f2305a0a90d\") " pod="openshift-ingress/e2e-api-a-e98c97-openshift-default-58df5648d5-ljkpq" Jul 09 22:54:26.111045 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:54:26.110894 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5d550c17-9bf5-4730-bbe0-6f2305a0a90d-istio-envoy\") pod \"e2e-api-a-e98c97-openshift-default-58df5648d5-ljkpq\" (UID: \"5d550c17-9bf5-4730-bbe0-6f2305a0a90d\") " pod="openshift-ingress/e2e-api-a-e98c97-openshift-default-58df5648d5-ljkpq" Jul 09 22:54:26.111653 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:54:26.111163 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d550c17-9bf5-4730-bbe0-6f2305a0a90d-workload-socket\") pod \"e2e-api-a-e98c97-openshift-default-58df5648d5-ljkpq\" (UID: \"5d550c17-9bf5-4730-bbe0-6f2305a0a90d\") " pod="openshift-ingress/e2e-api-a-e98c97-openshift-default-58df5648d5-ljkpq" Jul 09 22:54:26.111653 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:54:26.111239 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5d550c17-9bf5-4730-bbe0-6f2305a0a90d-workload-certs\") pod \"e2e-api-a-e98c97-openshift-default-58df5648d5-ljkpq\" (UID: \"5d550c17-9bf5-4730-bbe0-6f2305a0a90d\") " pod="openshift-ingress/e2e-api-a-e98c97-openshift-default-58df5648d5-ljkpq" Jul 09 22:54:26.111863 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:54:26.111843 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5d550c17-9bf5-4730-bbe0-6f2305a0a90d-istiod-ca-cert\") pod \"e2e-api-a-e98c97-openshift-default-58df5648d5-ljkpq\" (UID: \"5d550c17-9bf5-4730-bbe0-6f2305a0a90d\") " pod="openshift-ingress/e2e-api-a-e98c97-openshift-default-58df5648d5-ljkpq" Jul 09 22:54:26.111957 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:54:26.111909 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d550c17-9bf5-4730-bbe0-6f2305a0a90d-credential-socket\") pod \"e2e-api-a-e98c97-openshift-default-58df5648d5-ljkpq\" (UID: \"5d550c17-9bf5-4730-bbe0-6f2305a0a90d\") " pod="openshift-ingress/e2e-api-a-e98c97-openshift-default-58df5648d5-ljkpq" Jul 09 22:54:26.112013 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:54:26.111952 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5d550c17-9bf5-4730-bbe0-6f2305a0a90d-istio-data\") pod \"e2e-api-a-e98c97-openshift-default-58df5648d5-ljkpq\" (UID: \"5d550c17-9bf5-4730-bbe0-6f2305a0a90d\") " pod="openshift-ingress/e2e-api-a-e98c97-openshift-default-58df5648d5-ljkpq" Jul 09 22:54:26.113235 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:54:26.113214 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5d550c17-9bf5-4730-bbe0-6f2305a0a90d-istio-envoy\") pod \"e2e-api-a-e98c97-openshift-default-58df5648d5-ljkpq\" (UID: \"5d550c17-9bf5-4730-bbe0-6f2305a0a90d\") " pod="openshift-ingress/e2e-api-a-e98c97-openshift-default-58df5648d5-ljkpq" Jul 09 22:54:26.113522 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:54:26.113505 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5d550c17-9bf5-4730-bbe0-6f2305a0a90d-istio-podinfo\") pod \"e2e-api-a-e98c97-openshift-default-58df5648d5-ljkpq\" (UID: \"5d550c17-9bf5-4730-bbe0-6f2305a0a90d\") " pod="openshift-ingress/e2e-api-a-e98c97-openshift-default-58df5648d5-ljkpq" Jul 09 22:54:26.121352 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:54:26.121320 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5d550c17-9bf5-4730-bbe0-6f2305a0a90d-istio-token\") pod \"e2e-api-a-e98c97-openshift-default-58df5648d5-ljkpq\" (UID: \"5d550c17-9bf5-4730-bbe0-6f2305a0a90d\") " pod="openshift-ingress/e2e-api-a-e98c97-openshift-default-58df5648d5-ljkpq" Jul 09 22:54:26.121494 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:54:26.121473 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tdgqc\" (UniqueName: \"kubernetes.io/projected/5d550c17-9bf5-4730-bbe0-6f2305a0a90d-kube-api-access-tdgqc\") pod \"e2e-api-a-e98c97-openshift-default-58df5648d5-ljkpq\" (UID: \"5d550c17-9bf5-4730-bbe0-6f2305a0a90d\") " pod="openshift-ingress/e2e-api-a-e98c97-openshift-default-58df5648d5-ljkpq" Jul 09 22:54:26.168436 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:54:26.168395 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-e98c97-openshift-default-58df5648d5-ljkpq" Jul 09 22:54:26.317567 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:54:26.317541 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-e98c97-openshift-default-58df5648d5-ljkpq"] Jul 09 22:54:26.319372 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:54:26.319345 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5d550c17_9bf5_4730_bbe0_6f2305a0a90d.slice/crio-7fca895f1883779e6f5d09fa7f470ae895780e45664bfb533685787f290ebbe7 WatchSource:0}: Error finding container 7fca895f1883779e6f5d09fa7f470ae895780e45664bfb533685787f290ebbe7: Status 404 returned error can't find the container with id 7fca895f1883779e6f5d09fa7f470ae895780e45664bfb533685787f290ebbe7 Jul 09 22:54:26.321451 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:54:26.321422 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 09 22:54:26.321545 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:54:26.321485 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 09 22:54:26.321545 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:54:26.321514 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 09 22:54:27.235560 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:54:27.235522 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-e98c97-openshift-default-58df5648d5-ljkpq" event={"ID":"5d550c17-9bf5-4730-bbe0-6f2305a0a90d","Type":"ContainerStarted","Data":"07d4ddf7447507f8e42c8db4370968f4f07ba16083b85688168b1262713f1228"} Jul 09 22:54:27.236048 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:54:27.235578 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-e98c97-openshift-default-58df5648d5-ljkpq" event={"ID":"5d550c17-9bf5-4730-bbe0-6f2305a0a90d","Type":"ContainerStarted","Data":"7fca895f1883779e6f5d09fa7f470ae895780e45664bfb533685787f290ebbe7"} Jul 09 22:54:27.256761 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:54:27.256708 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-e98c97-openshift-default-58df5648d5-ljkpq" podStartSLOduration=2.256689886 podStartE2EDuration="2.256689886s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 22:54:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 22:54:27.255322949 +0000 UTC m=+2659.166775641" watchObservedRunningTime="2026-07-09 22:54:27.256689886 +0000 UTC m=+2659.168142577" Jul 09 22:54:28.169061 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:54:28.169020 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-e98c97-openshift-default-58df5648d5-ljkpq" Jul 09 22:54:28.174314 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:54:28.174286 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-e98c97-openshift-default-58df5648d5-ljkpq" Jul 09 22:54:28.241949 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:54:28.241908 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-e98c97-openshift-default-58df5648d5-ljkpq" Jul 09 22:54:28.242826 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:54:28.242807 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-e98c97-openshift-default-58df5648d5-ljkpq" Jul 09 22:54:28.884031 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:54:28.883990 2579 patch_prober.go:28] interesting pod/e2e-inf-b-d2543e-openshift-default-55686d679c-nbqjg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 09 22:54:28.884207 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:54:28.884069 2579 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-d2543e-openshift-default-55686d679c-nbqjg" podUID="04e62c1d-d3ec-4760-b5e1-d8d31a1e653d" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 09 22:54:30.091380 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:54:30.091311 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-d2543e-openshift-default-55686d679c-nbqjg" Jul 09 22:54:30.139997 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:54:30.139956 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/04e62c1d-d3ec-4760-b5e1-d8d31a1e653d-istio-envoy\") pod \"04e62c1d-d3ec-4760-b5e1-d8d31a1e653d\" (UID: \"04e62c1d-d3ec-4760-b5e1-d8d31a1e653d\") " Jul 09 22:54:30.140174 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:54:30.140038 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/04e62c1d-d3ec-4760-b5e1-d8d31a1e653d-istio-token\") pod \"04e62c1d-d3ec-4760-b5e1-d8d31a1e653d\" (UID: \"04e62c1d-d3ec-4760-b5e1-d8d31a1e653d\") " Jul 09 22:54:30.140174 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:54:30.140072 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/04e62c1d-d3ec-4760-b5e1-d8d31a1e653d-istio-podinfo\") pod \"04e62c1d-d3ec-4760-b5e1-d8d31a1e653d\" (UID: \"04e62c1d-d3ec-4760-b5e1-d8d31a1e653d\") " Jul 09 22:54:30.140323 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:54:30.140202 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/04e62c1d-d3ec-4760-b5e1-d8d31a1e653d-istiod-ca-cert\") pod \"04e62c1d-d3ec-4760-b5e1-d8d31a1e653d\" (UID: \"04e62c1d-d3ec-4760-b5e1-d8d31a1e653d\") " Jul 09 22:54:30.140323 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:54:30.140237 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/04e62c1d-d3ec-4760-b5e1-d8d31a1e653d-istio-data\") pod \"04e62c1d-d3ec-4760-b5e1-d8d31a1e653d\" (UID: \"04e62c1d-d3ec-4760-b5e1-d8d31a1e653d\") " Jul 09 22:54:30.140323 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:54:30.140290 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/04e62c1d-d3ec-4760-b5e1-d8d31a1e653d-credential-socket\") pod \"04e62c1d-d3ec-4760-b5e1-d8d31a1e653d\" (UID: \"04e62c1d-d3ec-4760-b5e1-d8d31a1e653d\") " Jul 09 22:54:30.140559 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:54:30.140322 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5pnf5\" (UniqueName: \"kubernetes.io/projected/04e62c1d-d3ec-4760-b5e1-d8d31a1e653d-kube-api-access-5pnf5\") pod \"04e62c1d-d3ec-4760-b5e1-d8d31a1e653d\" (UID: \"04e62c1d-d3ec-4760-b5e1-d8d31a1e653d\") " Jul 09 22:54:30.140559 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:54:30.140349 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/04e62c1d-d3ec-4760-b5e1-d8d31a1e653d-workload-socket\") pod \"04e62c1d-d3ec-4760-b5e1-d8d31a1e653d\" (UID: \"04e62c1d-d3ec-4760-b5e1-d8d31a1e653d\") " Jul 09 22:54:30.140559 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:54:30.140385 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/04e62c1d-d3ec-4760-b5e1-d8d31a1e653d-workload-certs\") pod \"04e62c1d-d3ec-4760-b5e1-d8d31a1e653d\" (UID: \"04e62c1d-d3ec-4760-b5e1-d8d31a1e653d\") " Jul 09 22:54:30.140719 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:54:30.140583 2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/04e62c1d-d3ec-4760-b5e1-d8d31a1e653d-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "04e62c1d-d3ec-4760-b5e1-d8d31a1e653d" (UID: "04e62c1d-d3ec-4760-b5e1-d8d31a1e653d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 09 22:54:30.140868 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:54:30.140789 2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/04e62c1d-d3ec-4760-b5e1-d8d31a1e653d-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "04e62c1d-d3ec-4760-b5e1-d8d31a1e653d" (UID: "04e62c1d-d3ec-4760-b5e1-d8d31a1e653d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 22:54:30.141057 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:54:30.140873 2579 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/04e62c1d-d3ec-4760-b5e1-d8d31a1e653d-workload-certs\") on node \"ip-10-0-136-251.ec2.internal\" DevicePath \"\"" Jul 09 22:54:30.141057 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:54:30.140895 2579 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/04e62c1d-d3ec-4760-b5e1-d8d31a1e653d-istiod-ca-cert\") on node \"ip-10-0-136-251.ec2.internal\" DevicePath \"\"" Jul 09 22:54:30.141057 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:54:30.140897 2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/04e62c1d-d3ec-4760-b5e1-d8d31a1e653d-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "04e62c1d-d3ec-4760-b5e1-d8d31a1e653d" (UID: "04e62c1d-d3ec-4760-b5e1-d8d31a1e653d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 22:54:30.141439 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:54:30.141412 2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/04e62c1d-d3ec-4760-b5e1-d8d31a1e653d-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "04e62c1d-d3ec-4760-b5e1-d8d31a1e653d" (UID: "04e62c1d-d3ec-4760-b5e1-d8d31a1e653d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 22:54:30.141652 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:54:30.141625 2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/04e62c1d-d3ec-4760-b5e1-d8d31a1e653d-istio-data" (OuterVolumeSpecName: "istio-data") pod "04e62c1d-d3ec-4760-b5e1-d8d31a1e653d" (UID: "04e62c1d-d3ec-4760-b5e1-d8d31a1e653d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 22:54:30.142769 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:54:30.142610 2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/04e62c1d-d3ec-4760-b5e1-d8d31a1e653d-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "04e62c1d-d3ec-4760-b5e1-d8d31a1e653d" (UID: "04e62c1d-d3ec-4760-b5e1-d8d31a1e653d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 09 22:54:30.142769 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:54:30.142726 2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/04e62c1d-d3ec-4760-b5e1-d8d31a1e653d-istio-token" (OuterVolumeSpecName: "istio-token") pod "04e62c1d-d3ec-4760-b5e1-d8d31a1e653d" (UID: "04e62c1d-d3ec-4760-b5e1-d8d31a1e653d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 22:54:30.142981 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:54:30.142953 2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/04e62c1d-d3ec-4760-b5e1-d8d31a1e653d-kube-api-access-5pnf5" (OuterVolumeSpecName: "kube-api-access-5pnf5") pod "04e62c1d-d3ec-4760-b5e1-d8d31a1e653d" (UID: "04e62c1d-d3ec-4760-b5e1-d8d31a1e653d"). InnerVolumeSpecName "kube-api-access-5pnf5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 22:54:30.143082 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:54:30.143027 2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/04e62c1d-d3ec-4760-b5e1-d8d31a1e653d-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "04e62c1d-d3ec-4760-b5e1-d8d31a1e653d" (UID: "04e62c1d-d3ec-4760-b5e1-d8d31a1e653d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 22:54:30.241704 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:54:30.241655 2579 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/04e62c1d-d3ec-4760-b5e1-d8d31a1e653d-istio-data\") on node \"ip-10-0-136-251.ec2.internal\" DevicePath \"\"" Jul 09 22:54:30.241704 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:54:30.241703 2579 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/04e62c1d-d3ec-4760-b5e1-d8d31a1e653d-credential-socket\") on node \"ip-10-0-136-251.ec2.internal\" DevicePath \"\"" Jul 09 22:54:30.242044 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:54:30.241718 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5pnf5\" (UniqueName: \"kubernetes.io/projected/04e62c1d-d3ec-4760-b5e1-d8d31a1e653d-kube-api-access-5pnf5\") on node \"ip-10-0-136-251.ec2.internal\" DevicePath \"\"" Jul 09 22:54:30.242044 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:54:30.241733 2579 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/04e62c1d-d3ec-4760-b5e1-d8d31a1e653d-workload-socket\") on node \"ip-10-0-136-251.ec2.internal\" DevicePath \"\"" Jul 09 22:54:30.242044 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:54:30.241747 2579 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/04e62c1d-d3ec-4760-b5e1-d8d31a1e653d-istio-envoy\") on node \"ip-10-0-136-251.ec2.internal\" DevicePath \"\"" Jul 09 22:54:30.242044 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:54:30.241760 2579 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/04e62c1d-d3ec-4760-b5e1-d8d31a1e653d-istio-token\") on node \"ip-10-0-136-251.ec2.internal\" DevicePath \"\"" Jul 09 22:54:30.242044 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:54:30.241772 2579 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/04e62c1d-d3ec-4760-b5e1-d8d31a1e653d-istio-podinfo\") on node \"ip-10-0-136-251.ec2.internal\" DevicePath \"\"" Jul 09 22:54:30.254996 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:54:30.254957 2579 generic.go:356] "Generic (PLEG): container finished" podID="04e62c1d-d3ec-4760-b5e1-d8d31a1e653d" containerID="f24ea49287d0dd2d4a2ec6a96a172cd41a8989771da7e7d4499a78210ef17d35" exitCode=0 Jul 09 22:54:30.255193 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:54:30.255018 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-d2543e-openshift-default-55686d679c-nbqjg" Jul 09 22:54:30.255193 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:54:30.255038 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-d2543e-openshift-default-55686d679c-nbqjg" event={"ID":"04e62c1d-d3ec-4760-b5e1-d8d31a1e653d","Type":"ContainerDied","Data":"f24ea49287d0dd2d4a2ec6a96a172cd41a8989771da7e7d4499a78210ef17d35"} Jul 09 22:54:30.255193 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:54:30.255098 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-d2543e-openshift-default-55686d679c-nbqjg" event={"ID":"04e62c1d-d3ec-4760-b5e1-d8d31a1e653d","Type":"ContainerDied","Data":"00e996c8b434dfd435ffb0ba20f38d3557bfe4c7345dabd448e320af36b7a223"} Jul 09 22:54:30.255193 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:54:30.255121 2579 scope.go:117] "RemoveContainer" containerID="f24ea49287d0dd2d4a2ec6a96a172cd41a8989771da7e7d4499a78210ef17d35" Jul 09 22:54:30.264557 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:54:30.264540 2579 scope.go:117] "RemoveContainer" containerID="2395f2a58268a546fe2e61fd0413065d2bdddfc0e7414ca2d8be8e654db64e96" Jul 09 22:54:30.272850 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:54:30.272828 2579 scope.go:117] "RemoveContainer" containerID="f24ea49287d0dd2d4a2ec6a96a172cd41a8989771da7e7d4499a78210ef17d35" Jul 09 22:54:30.273155 ip-10-0-136-251 kubenswrapper[2579]: E0709 22:54:30.273132 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f24ea49287d0dd2d4a2ec6a96a172cd41a8989771da7e7d4499a78210ef17d35\": container with ID starting with f24ea49287d0dd2d4a2ec6a96a172cd41a8989771da7e7d4499a78210ef17d35 not found: ID does not exist" containerID="f24ea49287d0dd2d4a2ec6a96a172cd41a8989771da7e7d4499a78210ef17d35" Jul 09 22:54:30.273244 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:54:30.273163 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f24ea49287d0dd2d4a2ec6a96a172cd41a8989771da7e7d4499a78210ef17d35"} err="failed to get container status \"f24ea49287d0dd2d4a2ec6a96a172cd41a8989771da7e7d4499a78210ef17d35\": rpc error: code = NotFound desc = could not find container \"f24ea49287d0dd2d4a2ec6a96a172cd41a8989771da7e7d4499a78210ef17d35\": container with ID starting with f24ea49287d0dd2d4a2ec6a96a172cd41a8989771da7e7d4499a78210ef17d35 not found: ID does not exist" Jul 09 22:54:30.273244 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:54:30.273183 2579 scope.go:117] "RemoveContainer" containerID="2395f2a58268a546fe2e61fd0413065d2bdddfc0e7414ca2d8be8e654db64e96" Jul 09 22:54:30.273422 ip-10-0-136-251 kubenswrapper[2579]: E0709 22:54:30.273405 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2395f2a58268a546fe2e61fd0413065d2bdddfc0e7414ca2d8be8e654db64e96\": container with ID starting with 2395f2a58268a546fe2e61fd0413065d2bdddfc0e7414ca2d8be8e654db64e96 not found: ID does not exist" containerID="2395f2a58268a546fe2e61fd0413065d2bdddfc0e7414ca2d8be8e654db64e96" Jul 09 22:54:30.273476 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:54:30.273433 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2395f2a58268a546fe2e61fd0413065d2bdddfc0e7414ca2d8be8e654db64e96"} err="failed to get container status \"2395f2a58268a546fe2e61fd0413065d2bdddfc0e7414ca2d8be8e654db64e96\": rpc error: code = NotFound desc = could not find container \"2395f2a58268a546fe2e61fd0413065d2bdddfc0e7414ca2d8be8e654db64e96\": container with ID starting with 2395f2a58268a546fe2e61fd0413065d2bdddfc0e7414ca2d8be8e654db64e96 not found: ID does not exist" Jul 09 22:54:30.278253 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:54:30.278223 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-d2543e-openshift-default-55686d679c-nbqjg"] Jul 09 22:54:30.283835 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:54:30.283809 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-d2543e-openshift-default-55686d679c-nbqjg"] Jul 09 22:54:30.566445 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:54:30.566411 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="04e62c1d-d3ec-4760-b5e1-d8d31a1e653d" path="/var/lib/kubelet/pods/04e62c1d-d3ec-4760-b5e1-d8d31a1e653d/volumes" Jul 09 22:54:34.442261 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:54:34.442225 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-6ddc6a-openshift-default-6c99cb4b89-2cmhb"] Jul 09 22:54:34.442830 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:54:34.442806 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="04e62c1d-d3ec-4760-b5e1-d8d31a1e653d" containerName="istio-proxy" Jul 09 22:54:34.442830 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:54:34.442832 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="04e62c1d-d3ec-4760-b5e1-d8d31a1e653d" containerName="istio-proxy" Jul 09 22:54:34.443012 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:54:34.443001 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="04e62c1d-d3ec-4760-b5e1-d8d31a1e653d" containerName="istio-proxy" Jul 09 22:54:34.443071 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:54:34.443019 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="04e62c1d-d3ec-4760-b5e1-d8d31a1e653d" containerName="istio-proxy" Jul 09 22:54:34.449994 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:54:34.449382 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-6ddc6a-openshift-default-6c99cb4b89-2cmhb" Jul 09 22:54:34.455320 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:54:34.455297 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-6ddc6a-openshift-default-dockercfg-fs8vd\"" Jul 09 22:54:34.459967 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:54:34.459936 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-6ddc6a-openshift-default-6c99cb4b89-2cmhb"] Jul 09 22:54:34.479500 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:54:34.479468 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4b5a3562-88ca-4ccc-b6bd-6367789bf658-istio-podinfo\") pod \"e2e-api-b-6ddc6a-openshift-default-6c99cb4b89-2cmhb\" (UID: \"4b5a3562-88ca-4ccc-b6bd-6367789bf658\") " pod="openshift-ingress/e2e-api-b-6ddc6a-openshift-default-6c99cb4b89-2cmhb" Jul 09 22:54:34.479649 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:54:34.479507 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b5a3562-88ca-4ccc-b6bd-6367789bf658-credential-socket\") pod \"e2e-api-b-6ddc6a-openshift-default-6c99cb4b89-2cmhb\" (UID: \"4b5a3562-88ca-4ccc-b6bd-6367789bf658\") " pod="openshift-ingress/e2e-api-b-6ddc6a-openshift-default-6c99cb4b89-2cmhb" Jul 09 22:54:34.479649 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:54:34.479543 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4b5a3562-88ca-4ccc-b6bd-6367789bf658-istio-envoy\") pod \"e2e-api-b-6ddc6a-openshift-default-6c99cb4b89-2cmhb\" (UID: \"4b5a3562-88ca-4ccc-b6bd-6367789bf658\") " pod="openshift-ingress/e2e-api-b-6ddc6a-openshift-default-6c99cb4b89-2cmhb" Jul 09 22:54:34.479649 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:54:34.479579 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4b5a3562-88ca-4ccc-b6bd-6367789bf658-istio-token\") pod \"e2e-api-b-6ddc6a-openshift-default-6c99cb4b89-2cmhb\" (UID: \"4b5a3562-88ca-4ccc-b6bd-6367789bf658\") " pod="openshift-ingress/e2e-api-b-6ddc6a-openshift-default-6c99cb4b89-2cmhb" Jul 09 22:54:34.479649 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:54:34.479614 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b5a3562-88ca-4ccc-b6bd-6367789bf658-workload-socket\") pod \"e2e-api-b-6ddc6a-openshift-default-6c99cb4b89-2cmhb\" (UID: \"4b5a3562-88ca-4ccc-b6bd-6367789bf658\") " pod="openshift-ingress/e2e-api-b-6ddc6a-openshift-default-6c99cb4b89-2cmhb" Jul 09 22:54:34.479785 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:54:34.479670 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4b5a3562-88ca-4ccc-b6bd-6367789bf658-workload-certs\") pod \"e2e-api-b-6ddc6a-openshift-default-6c99cb4b89-2cmhb\" (UID: \"4b5a3562-88ca-4ccc-b6bd-6367789bf658\") " pod="openshift-ingress/e2e-api-b-6ddc6a-openshift-default-6c99cb4b89-2cmhb" Jul 09 22:54:34.479785 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:54:34.479720 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4b5a3562-88ca-4ccc-b6bd-6367789bf658-istio-data\") pod \"e2e-api-b-6ddc6a-openshift-default-6c99cb4b89-2cmhb\" (UID: \"4b5a3562-88ca-4ccc-b6bd-6367789bf658\") " pod="openshift-ingress/e2e-api-b-6ddc6a-openshift-default-6c99cb4b89-2cmhb" Jul 09 22:54:34.479785 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:54:34.479736 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4b5a3562-88ca-4ccc-b6bd-6367789bf658-istiod-ca-cert\") pod \"e2e-api-b-6ddc6a-openshift-default-6c99cb4b89-2cmhb\" (UID: \"4b5a3562-88ca-4ccc-b6bd-6367789bf658\") " pod="openshift-ingress/e2e-api-b-6ddc6a-openshift-default-6c99cb4b89-2cmhb" Jul 09 22:54:34.479785 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:54:34.479751 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2hwns\" (UniqueName: \"kubernetes.io/projected/4b5a3562-88ca-4ccc-b6bd-6367789bf658-kube-api-access-2hwns\") pod \"e2e-api-b-6ddc6a-openshift-default-6c99cb4b89-2cmhb\" (UID: \"4b5a3562-88ca-4ccc-b6bd-6367789bf658\") " pod="openshift-ingress/e2e-api-b-6ddc6a-openshift-default-6c99cb4b89-2cmhb" Jul 09 22:54:34.581102 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:54:34.581061 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4b5a3562-88ca-4ccc-b6bd-6367789bf658-istio-podinfo\") pod \"e2e-api-b-6ddc6a-openshift-default-6c99cb4b89-2cmhb\" (UID: \"4b5a3562-88ca-4ccc-b6bd-6367789bf658\") " pod="openshift-ingress/e2e-api-b-6ddc6a-openshift-default-6c99cb4b89-2cmhb" Jul 09 22:54:34.581102 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:54:34.581104 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b5a3562-88ca-4ccc-b6bd-6367789bf658-credential-socket\") pod \"e2e-api-b-6ddc6a-openshift-default-6c99cb4b89-2cmhb\" (UID: \"4b5a3562-88ca-4ccc-b6bd-6367789bf658\") " pod="openshift-ingress/e2e-api-b-6ddc6a-openshift-default-6c99cb4b89-2cmhb" Jul 09 22:54:34.581350 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:54:34.581134 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4b5a3562-88ca-4ccc-b6bd-6367789bf658-istio-envoy\") pod \"e2e-api-b-6ddc6a-openshift-default-6c99cb4b89-2cmhb\" (UID: \"4b5a3562-88ca-4ccc-b6bd-6367789bf658\") " pod="openshift-ingress/e2e-api-b-6ddc6a-openshift-default-6c99cb4b89-2cmhb" Jul 09 22:54:34.581350 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:54:34.581154 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4b5a3562-88ca-4ccc-b6bd-6367789bf658-istio-token\") pod \"e2e-api-b-6ddc6a-openshift-default-6c99cb4b89-2cmhb\" (UID: \"4b5a3562-88ca-4ccc-b6bd-6367789bf658\") " pod="openshift-ingress/e2e-api-b-6ddc6a-openshift-default-6c99cb4b89-2cmhb" Jul 09 22:54:34.581350 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:54:34.581184 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b5a3562-88ca-4ccc-b6bd-6367789bf658-workload-socket\") pod \"e2e-api-b-6ddc6a-openshift-default-6c99cb4b89-2cmhb\" (UID: \"4b5a3562-88ca-4ccc-b6bd-6367789bf658\") " pod="openshift-ingress/e2e-api-b-6ddc6a-openshift-default-6c99cb4b89-2cmhb" Jul 09 22:54:34.581350 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:54:34.581241 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4b5a3562-88ca-4ccc-b6bd-6367789bf658-workload-certs\") pod \"e2e-api-b-6ddc6a-openshift-default-6c99cb4b89-2cmhb\" (UID: \"4b5a3562-88ca-4ccc-b6bd-6367789bf658\") " pod="openshift-ingress/e2e-api-b-6ddc6a-openshift-default-6c99cb4b89-2cmhb" Jul 09 22:54:34.581350 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:54:34.581288 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4b5a3562-88ca-4ccc-b6bd-6367789bf658-istio-data\") pod \"e2e-api-b-6ddc6a-openshift-default-6c99cb4b89-2cmhb\" (UID: \"4b5a3562-88ca-4ccc-b6bd-6367789bf658\") " pod="openshift-ingress/e2e-api-b-6ddc6a-openshift-default-6c99cb4b89-2cmhb" Jul 09 22:54:34.581350 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:54:34.581317 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4b5a3562-88ca-4ccc-b6bd-6367789bf658-istiod-ca-cert\") pod \"e2e-api-b-6ddc6a-openshift-default-6c99cb4b89-2cmhb\" (UID: \"4b5a3562-88ca-4ccc-b6bd-6367789bf658\") " pod="openshift-ingress/e2e-api-b-6ddc6a-openshift-default-6c99cb4b89-2cmhb" Jul 09 22:54:34.581350 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:54:34.581342 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2hwns\" (UniqueName: \"kubernetes.io/projected/4b5a3562-88ca-4ccc-b6bd-6367789bf658-kube-api-access-2hwns\") pod \"e2e-api-b-6ddc6a-openshift-default-6c99cb4b89-2cmhb\" (UID: \"4b5a3562-88ca-4ccc-b6bd-6367789bf658\") " pod="openshift-ingress/e2e-api-b-6ddc6a-openshift-default-6c99cb4b89-2cmhb" Jul 09 22:54:34.582134 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:54:34.581629 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b5a3562-88ca-4ccc-b6bd-6367789bf658-credential-socket\") pod \"e2e-api-b-6ddc6a-openshift-default-6c99cb4b89-2cmhb\" (UID: \"4b5a3562-88ca-4ccc-b6bd-6367789bf658\") " pod="openshift-ingress/e2e-api-b-6ddc6a-openshift-default-6c99cb4b89-2cmhb" Jul 09 22:54:34.582134 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:54:34.581724 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b5a3562-88ca-4ccc-b6bd-6367789bf658-workload-socket\") pod \"e2e-api-b-6ddc6a-openshift-default-6c99cb4b89-2cmhb\" (UID: \"4b5a3562-88ca-4ccc-b6bd-6367789bf658\") " pod="openshift-ingress/e2e-api-b-6ddc6a-openshift-default-6c99cb4b89-2cmhb" Jul 09 22:54:34.582134 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:54:34.581823 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4b5a3562-88ca-4ccc-b6bd-6367789bf658-workload-certs\") pod \"e2e-api-b-6ddc6a-openshift-default-6c99cb4b89-2cmhb\" (UID: \"4b5a3562-88ca-4ccc-b6bd-6367789bf658\") " pod="openshift-ingress/e2e-api-b-6ddc6a-openshift-default-6c99cb4b89-2cmhb" Jul 09 22:54:34.582134 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:54:34.581878 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4b5a3562-88ca-4ccc-b6bd-6367789bf658-istio-data\") pod \"e2e-api-b-6ddc6a-openshift-default-6c99cb4b89-2cmhb\" (UID: \"4b5a3562-88ca-4ccc-b6bd-6367789bf658\") " pod="openshift-ingress/e2e-api-b-6ddc6a-openshift-default-6c99cb4b89-2cmhb" Jul 09 22:54:34.582337 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:54:34.582315 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4b5a3562-88ca-4ccc-b6bd-6367789bf658-istiod-ca-cert\") pod \"e2e-api-b-6ddc6a-openshift-default-6c99cb4b89-2cmhb\" (UID: \"4b5a3562-88ca-4ccc-b6bd-6367789bf658\") " pod="openshift-ingress/e2e-api-b-6ddc6a-openshift-default-6c99cb4b89-2cmhb" Jul 09 22:54:34.583613 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:54:34.583593 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4b5a3562-88ca-4ccc-b6bd-6367789bf658-istio-envoy\") pod \"e2e-api-b-6ddc6a-openshift-default-6c99cb4b89-2cmhb\" (UID: \"4b5a3562-88ca-4ccc-b6bd-6367789bf658\") " pod="openshift-ingress/e2e-api-b-6ddc6a-openshift-default-6c99cb4b89-2cmhb" Jul 09 22:54:34.583809 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:54:34.583793 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4b5a3562-88ca-4ccc-b6bd-6367789bf658-istio-podinfo\") pod \"e2e-api-b-6ddc6a-openshift-default-6c99cb4b89-2cmhb\" (UID: \"4b5a3562-88ca-4ccc-b6bd-6367789bf658\") " pod="openshift-ingress/e2e-api-b-6ddc6a-openshift-default-6c99cb4b89-2cmhb" Jul 09 22:54:34.598004 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:54:34.597967 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4b5a3562-88ca-4ccc-b6bd-6367789bf658-istio-token\") pod \"e2e-api-b-6ddc6a-openshift-default-6c99cb4b89-2cmhb\" (UID: \"4b5a3562-88ca-4ccc-b6bd-6367789bf658\") " pod="openshift-ingress/e2e-api-b-6ddc6a-openshift-default-6c99cb4b89-2cmhb" Jul 09 22:54:34.598191 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:54:34.598168 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2hwns\" (UniqueName: \"kubernetes.io/projected/4b5a3562-88ca-4ccc-b6bd-6367789bf658-kube-api-access-2hwns\") pod \"e2e-api-b-6ddc6a-openshift-default-6c99cb4b89-2cmhb\" (UID: \"4b5a3562-88ca-4ccc-b6bd-6367789bf658\") " pod="openshift-ingress/e2e-api-b-6ddc6a-openshift-default-6c99cb4b89-2cmhb" Jul 09 22:54:34.768638 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:54:34.768595 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-6ddc6a-openshift-default-6c99cb4b89-2cmhb" Jul 09 22:54:34.917724 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:54:34.917686 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-6ddc6a-openshift-default-6c99cb4b89-2cmhb"] Jul 09 22:54:34.918703 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:54:34.918679 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4b5a3562_88ca_4ccc_b6bd_6367789bf658.slice/crio-2c8fbe252de7cb8fea57c9e6e929f9cb03bd094e4d65a38708b2f2e9f2ad5b65 WatchSource:0}: Error finding container 2c8fbe252de7cb8fea57c9e6e929f9cb03bd094e4d65a38708b2f2e9f2ad5b65: Status 404 returned error can't find the container with id 2c8fbe252de7cb8fea57c9e6e929f9cb03bd094e4d65a38708b2f2e9f2ad5b65 Jul 09 22:54:34.920846 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:54:34.920814 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 09 22:54:34.920903 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:54:34.920883 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 09 22:54:34.920954 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:54:34.920911 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 09 22:54:35.214979 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:54:35.214873 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-xdfrr"] Jul 09 22:54:35.288390 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:54:35.288345 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-6ddc6a-openshift-default-6c99cb4b89-2cmhb" event={"ID":"4b5a3562-88ca-4ccc-b6bd-6367789bf658","Type":"ContainerStarted","Data":"e8c18da342a1db7f7b51cf1c24f491f5441cf1be95da218ff575aabd88857a69"} Jul 09 22:54:35.288593 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:54:35.288418 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-6ddc6a-openshift-default-6c99cb4b89-2cmhb" event={"ID":"4b5a3562-88ca-4ccc-b6bd-6367789bf658","Type":"ContainerStarted","Data":"2c8fbe252de7cb8fea57c9e6e929f9cb03bd094e4d65a38708b2f2e9f2ad5b65"} Jul 09 22:54:35.308270 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:54:35.308204 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-6ddc6a-openshift-default-6c99cb4b89-2cmhb" podStartSLOduration=1.308184928 podStartE2EDuration="1.308184928s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 22:54:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 22:54:35.307044617 +0000 UTC m=+2667.218497344" watchObservedRunningTime="2026-07-09 22:54:35.308184928 +0000 UTC m=+2667.219637606" Jul 09 22:54:35.769086 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:54:35.769039 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-6ddc6a-openshift-default-6c99cb4b89-2cmhb" Jul 09 22:54:35.774117 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:54:35.774092 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-6ddc6a-openshift-default-6c99cb4b89-2cmhb" Jul 09 22:54:36.294304 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:54:36.294270 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-6ddc6a-openshift-default-6c99cb4b89-2cmhb" Jul 09 22:54:36.295405 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:54:36.295382 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-6ddc6a-openshift-default-6c99cb4b89-2cmhb" Jul 09 22:54:43.261384 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:54:43.260985 2579 patch_prober.go:28] interesting pod/e2e-api-a-e98c97-openshift-default-58df5648d5-ljkpq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused" start-of-body= Jul 09 22:54:43.261384 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:54:43.261056 2579 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-e98c97-openshift-default-58df5648d5-ljkpq" podUID="5d550c17-9bf5-4730-bbe0-6f2305a0a90d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused" Jul 09 22:54:43.340002 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:54:43.339968 2579 generic.go:356] "Generic (PLEG): container finished" podID="5d550c17-9bf5-4730-bbe0-6f2305a0a90d" containerID="07d4ddf7447507f8e42c8db4370968f4f07ba16083b85688168b1262713f1228" exitCode=0 Jul 09 22:54:43.340181 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:54:43.340036 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-e98c97-openshift-default-58df5648d5-ljkpq" event={"ID":"5d550c17-9bf5-4730-bbe0-6f2305a0a90d","Type":"ContainerDied","Data":"07d4ddf7447507f8e42c8db4370968f4f07ba16083b85688168b1262713f1228"} Jul 09 22:54:43.340500 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:54:43.340482 2579 scope.go:117] "RemoveContainer" containerID="07d4ddf7447507f8e42c8db4370968f4f07ba16083b85688168b1262713f1228" Jul 09 22:54:43.341708 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:54:43.341676 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 09 22:54:43.341809 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:54:43.341765 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 09 22:54:43.341809 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:54:43.341802 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 09 22:54:44.350827 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:54:44.350784 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-e98c97-openshift-default-58df5648d5-ljkpq" event={"ID":"5d550c17-9bf5-4730-bbe0-6f2305a0a90d","Type":"ContainerStarted","Data":"c62cfdb7dfd6942c507edfa6d6eac3ff292f1cd4240ef92a08567d40175bfb49"} Jul 09 22:54:45.169245 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:54:45.169203 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-e98c97-openshift-default-58df5648d5-ljkpq" Jul 09 22:54:45.169669 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:54:45.169642 2579 patch_prober.go:28] interesting pod/e2e-api-a-e98c97-openshift-default-58df5648d5-ljkpq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused" start-of-body= Jul 09 22:54:45.169868 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:54:45.169839 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-e98c97-openshift-default-58df5648d5-ljkpq" podUID="5d550c17-9bf5-4730-bbe0-6f2305a0a90d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused" Jul 09 22:54:46.169231 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:54:46.169193 2579 patch_prober.go:28] interesting pod/e2e-api-a-e98c97-openshift-default-58df5648d5-ljkpq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused" start-of-body= Jul 09 22:54:46.169654 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:54:46.169259 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-e98c97-openshift-default-58df5648d5-ljkpq" podUID="5d550c17-9bf5-4730-bbe0-6f2305a0a90d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused" Jul 09 22:54:47.169612 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:54:47.169442 2579 patch_prober.go:28] interesting pod/e2e-api-a-e98c97-openshift-default-58df5648d5-ljkpq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused" start-of-body= Jul 09 22:54:47.169612 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:54:47.169526 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-e98c97-openshift-default-58df5648d5-ljkpq" podUID="5d550c17-9bf5-4730-bbe0-6f2305a0a90d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused" Jul 09 22:54:48.169688 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:54:48.169632 2579 patch_prober.go:28] interesting pod/e2e-api-a-e98c97-openshift-default-58df5648d5-ljkpq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused" start-of-body= Jul 09 22:54:48.170261 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:54:48.169727 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-e98c97-openshift-default-58df5648d5-ljkpq" podUID="5d550c17-9bf5-4730-bbe0-6f2305a0a90d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused" Jul 09 22:54:48.224885 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:54:48.224842 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-xdfrr"] Jul 09 22:54:49.173677 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:54:49.173648 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-e98c97-openshift-default-58df5648d5-ljkpq" Jul 09 22:54:49.174120 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:54:49.173946 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-e98c97-openshift-default-58df5648d5-ljkpq" Jul 09 22:54:49.174721 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:54:49.174705 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-e98c97-openshift-default-58df5648d5-ljkpq" Jul 09 22:54:50.396619 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:54:50.396588 2579 generic.go:356] "Generic (PLEG): container finished" podID="4b5a3562-88ca-4ccc-b6bd-6367789bf658" containerID="e8c18da342a1db7f7b51cf1c24f491f5441cf1be95da218ff575aabd88857a69" exitCode=0 Jul 09 22:54:50.397082 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:54:50.396663 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-6ddc6a-openshift-default-6c99cb4b89-2cmhb" event={"ID":"4b5a3562-88ca-4ccc-b6bd-6367789bf658","Type":"ContainerDied","Data":"e8c18da342a1db7f7b51cf1c24f491f5441cf1be95da218ff575aabd88857a69"} Jul 09 22:54:50.397195 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:54:50.397182 2579 scope.go:117] "RemoveContainer" containerID="e8c18da342a1db7f7b51cf1c24f491f5441cf1be95da218ff575aabd88857a69" Jul 09 22:54:50.398190 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:54:50.398166 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 09 22:54:50.398260 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:54:50.398227 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 09 22:54:50.398299 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:54:50.398256 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 09 22:54:51.294268 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:54:51.294227 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-6ddc6a-openshift-default-6c99cb4b89-2cmhb" Jul 09 22:54:51.404123 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:54:51.404082 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-6ddc6a-openshift-default-6c99cb4b89-2cmhb" event={"ID":"4b5a3562-88ca-4ccc-b6bd-6367789bf658","Type":"ContainerStarted","Data":"7d19f5b7ff12e5867cad210842972321ca9981dfc5aec0867a4044dc5ca812eb"} Jul 09 22:54:51.769049 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:54:51.769003 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-6ddc6a-openshift-default-6c99cb4b89-2cmhb" Jul 09 22:54:51.770407 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:54:51.770375 2579 patch_prober.go:28] interesting pod/e2e-api-b-6ddc6a-openshift-default-6c99cb4b89-2cmhb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" start-of-body= Jul 09 22:54:51.770574 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:54:51.770427 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-6ddc6a-openshift-default-6c99cb4b89-2cmhb" podUID="4b5a3562-88ca-4ccc-b6bd-6367789bf658" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" Jul 09 22:54:52.769401 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:54:52.769361 2579 patch_prober.go:28] interesting pod/e2e-api-b-6ddc6a-openshift-default-6c99cb4b89-2cmhb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" start-of-body= Jul 09 22:54:52.769950 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:54:52.769419 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-6ddc6a-openshift-default-6c99cb4b89-2cmhb" podUID="4b5a3562-88ca-4ccc-b6bd-6367789bf658" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" Jul 09 22:54:53.769882 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:54:53.769839 2579 patch_prober.go:28] interesting pod/e2e-api-b-6ddc6a-openshift-default-6c99cb4b89-2cmhb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" start-of-body= Jul 09 22:54:53.770319 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:54:53.769901 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-6ddc6a-openshift-default-6c99cb4b89-2cmhb" podUID="4b5a3562-88ca-4ccc-b6bd-6367789bf658" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" Jul 09 22:54:54.769496 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:54:54.769455 2579 patch_prober.go:28] interesting pod/e2e-api-b-6ddc6a-openshift-default-6c99cb4b89-2cmhb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" start-of-body= Jul 09 22:54:54.769687 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:54:54.769531 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-6ddc6a-openshift-default-6c99cb4b89-2cmhb" podUID="4b5a3562-88ca-4ccc-b6bd-6367789bf658" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" Jul 09 22:54:55.773165 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:54:55.773128 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-6ddc6a-openshift-default-6c99cb4b89-2cmhb" Jul 09 22:54:55.773777 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:54:55.773529 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-6ddc6a-openshift-default-6c99cb4b89-2cmhb" Jul 09 22:54:55.774207 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:54:55.774188 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-6ddc6a-openshift-default-6c99cb4b89-2cmhb" Jul 09 22:55:04.709230 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:55:04.709149 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-xdfrr"] Jul 09 22:55:15.530673 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:55:15.530639 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-e98c97-openshift-default-58df5648d5-ljkpq"] Jul 09 22:55:15.531272 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:55:15.530947 2579 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-e98c97-openshift-default-58df5648d5-ljkpq" podUID="5d550c17-9bf5-4730-bbe0-6f2305a0a90d" containerName="istio-proxy" containerID="cri-o://c62cfdb7dfd6942c507edfa6d6eac3ff292f1cd4240ef92a08567d40175bfb49" gracePeriod=30 Jul 09 22:55:19.176378 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:55:19.176342 2579 patch_prober.go:28] interesting pod/e2e-api-a-e98c97-openshift-default-58df5648d5-ljkpq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 09 22:55:19.176766 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:55:19.176400 2579 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-e98c97-openshift-default-58df5648d5-ljkpq" podUID="5d550c17-9bf5-4730-bbe0-6f2305a0a90d" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 09 22:55:20.784035 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:55:20.784007 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-e98c97-openshift-default-58df5648d5-ljkpq" Jul 09 22:55:20.844539 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:55:20.844432 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5d550c17-9bf5-4730-bbe0-6f2305a0a90d-istio-podinfo\") pod \"5d550c17-9bf5-4730-bbe0-6f2305a0a90d\" (UID: \"5d550c17-9bf5-4730-bbe0-6f2305a0a90d\") " Jul 09 22:55:20.844539 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:55:20.844477 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d550c17-9bf5-4730-bbe0-6f2305a0a90d-credential-socket\") pod \"5d550c17-9bf5-4730-bbe0-6f2305a0a90d\" (UID: \"5d550c17-9bf5-4730-bbe0-6f2305a0a90d\") " Jul 09 22:55:20.844539 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:55:20.844495 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5d550c17-9bf5-4730-bbe0-6f2305a0a90d-istiod-ca-cert\") pod \"5d550c17-9bf5-4730-bbe0-6f2305a0a90d\" (UID: \"5d550c17-9bf5-4730-bbe0-6f2305a0a90d\") " Jul 09 22:55:20.844872 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:55:20.844547 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tdgqc\" (UniqueName: \"kubernetes.io/projected/5d550c17-9bf5-4730-bbe0-6f2305a0a90d-kube-api-access-tdgqc\") pod \"5d550c17-9bf5-4730-bbe0-6f2305a0a90d\" (UID: \"5d550c17-9bf5-4730-bbe0-6f2305a0a90d\") " Jul 09 22:55:20.844872 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:55:20.844567 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5d550c17-9bf5-4730-bbe0-6f2305a0a90d-istio-data\") pod \"5d550c17-9bf5-4730-bbe0-6f2305a0a90d\" (UID: \"5d550c17-9bf5-4730-bbe0-6f2305a0a90d\") " Jul 09 22:55:20.844872 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:55:20.844644 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5d550c17-9bf5-4730-bbe0-6f2305a0a90d-istio-envoy\") pod \"5d550c17-9bf5-4730-bbe0-6f2305a0a90d\" (UID: \"5d550c17-9bf5-4730-bbe0-6f2305a0a90d\") " Jul 09 22:55:20.844872 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:55:20.844663 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d550c17-9bf5-4730-bbe0-6f2305a0a90d-workload-socket\") pod \"5d550c17-9bf5-4730-bbe0-6f2305a0a90d\" (UID: \"5d550c17-9bf5-4730-bbe0-6f2305a0a90d\") " Jul 09 22:55:20.844872 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:55:20.844685 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5d550c17-9bf5-4730-bbe0-6f2305a0a90d-workload-certs\") pod \"5d550c17-9bf5-4730-bbe0-6f2305a0a90d\" (UID: \"5d550c17-9bf5-4730-bbe0-6f2305a0a90d\") " Jul 09 22:55:20.844872 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:55:20.844744 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5d550c17-9bf5-4730-bbe0-6f2305a0a90d-istio-token\") pod \"5d550c17-9bf5-4730-bbe0-6f2305a0a90d\" (UID: \"5d550c17-9bf5-4730-bbe0-6f2305a0a90d\") " Jul 09 22:55:20.844872 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:55:20.844795 2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d550c17-9bf5-4730-bbe0-6f2305a0a90d-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "5d550c17-9bf5-4730-bbe0-6f2305a0a90d" (UID: "5d550c17-9bf5-4730-bbe0-6f2305a0a90d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 22:55:20.845284 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:55:20.844957 2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d550c17-9bf5-4730-bbe0-6f2305a0a90d-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "5d550c17-9bf5-4730-bbe0-6f2305a0a90d" (UID: "5d550c17-9bf5-4730-bbe0-6f2305a0a90d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 22:55:20.845284 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:55:20.845054 2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5d550c17-9bf5-4730-bbe0-6f2305a0a90d-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "5d550c17-9bf5-4730-bbe0-6f2305a0a90d" (UID: "5d550c17-9bf5-4730-bbe0-6f2305a0a90d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 09 22:55:20.845284 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:55:20.845112 2579 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d550c17-9bf5-4730-bbe0-6f2305a0a90d-credential-socket\") on node \"ip-10-0-136-251.ec2.internal\" DevicePath \"\"" Jul 09 22:55:20.845284 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:55:20.845130 2579 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d550c17-9bf5-4730-bbe0-6f2305a0a90d-workload-socket\") on node \"ip-10-0-136-251.ec2.internal\" DevicePath \"\"" Jul 09 22:55:20.845284 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:55:20.845200 2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d550c17-9bf5-4730-bbe0-6f2305a0a90d-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "5d550c17-9bf5-4730-bbe0-6f2305a0a90d" (UID: "5d550c17-9bf5-4730-bbe0-6f2305a0a90d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 22:55:20.845600 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:55:20.845577 2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d550c17-9bf5-4730-bbe0-6f2305a0a90d-istio-data" (OuterVolumeSpecName: "istio-data") pod "5d550c17-9bf5-4730-bbe0-6f2305a0a90d" (UID: "5d550c17-9bf5-4730-bbe0-6f2305a0a90d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 22:55:20.846876 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:55:20.846846 2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5d550c17-9bf5-4730-bbe0-6f2305a0a90d-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "5d550c17-9bf5-4730-bbe0-6f2305a0a90d" (UID: "5d550c17-9bf5-4730-bbe0-6f2305a0a90d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 09 22:55:20.846876 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:55:20.846859 2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d550c17-9bf5-4730-bbe0-6f2305a0a90d-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "5d550c17-9bf5-4730-bbe0-6f2305a0a90d" (UID: "5d550c17-9bf5-4730-bbe0-6f2305a0a90d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 22:55:20.847212 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:55:20.847190 2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5d550c17-9bf5-4730-bbe0-6f2305a0a90d-kube-api-access-tdgqc" (OuterVolumeSpecName: "kube-api-access-tdgqc") pod "5d550c17-9bf5-4730-bbe0-6f2305a0a90d" (UID: "5d550c17-9bf5-4730-bbe0-6f2305a0a90d"). InnerVolumeSpecName "kube-api-access-tdgqc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 22:55:20.847296 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:55:20.847283 2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5d550c17-9bf5-4730-bbe0-6f2305a0a90d-istio-token" (OuterVolumeSpecName: "istio-token") pod "5d550c17-9bf5-4730-bbe0-6f2305a0a90d" (UID: "5d550c17-9bf5-4730-bbe0-6f2305a0a90d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 22:55:20.946005 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:55:20.945963 2579 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5d550c17-9bf5-4730-bbe0-6f2305a0a90d-workload-certs\") on node \"ip-10-0-136-251.ec2.internal\" DevicePath \"\"" Jul 09 22:55:20.946005 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:55:20.945996 2579 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5d550c17-9bf5-4730-bbe0-6f2305a0a90d-istio-token\") on node \"ip-10-0-136-251.ec2.internal\" DevicePath \"\"" Jul 09 22:55:20.946005 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:55:20.946005 2579 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5d550c17-9bf5-4730-bbe0-6f2305a0a90d-istio-podinfo\") on node \"ip-10-0-136-251.ec2.internal\" DevicePath \"\"" Jul 09 22:55:20.946005 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:55:20.946013 2579 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5d550c17-9bf5-4730-bbe0-6f2305a0a90d-istiod-ca-cert\") on node \"ip-10-0-136-251.ec2.internal\" DevicePath \"\"" Jul 09 22:55:20.946344 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:55:20.946023 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tdgqc\" (UniqueName: \"kubernetes.io/projected/5d550c17-9bf5-4730-bbe0-6f2305a0a90d-kube-api-access-tdgqc\") on node \"ip-10-0-136-251.ec2.internal\" DevicePath \"\"" Jul 09 22:55:20.946344 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:55:20.946033 2579 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5d550c17-9bf5-4730-bbe0-6f2305a0a90d-istio-data\") on node \"ip-10-0-136-251.ec2.internal\" DevicePath \"\"" Jul 09 22:55:20.946344 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:55:20.946040 2579 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5d550c17-9bf5-4730-bbe0-6f2305a0a90d-istio-envoy\") on node \"ip-10-0-136-251.ec2.internal\" DevicePath \"\"" Jul 09 22:55:21.564450 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:55:21.564410 2579 generic.go:356] "Generic (PLEG): container finished" podID="5d550c17-9bf5-4730-bbe0-6f2305a0a90d" containerID="c62cfdb7dfd6942c507edfa6d6eac3ff292f1cd4240ef92a08567d40175bfb49" exitCode=0 Jul 09 22:55:21.564633 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:55:21.564484 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-e98c97-openshift-default-58df5648d5-ljkpq" Jul 09 22:55:21.564633 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:55:21.564500 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-e98c97-openshift-default-58df5648d5-ljkpq" event={"ID":"5d550c17-9bf5-4730-bbe0-6f2305a0a90d","Type":"ContainerDied","Data":"c62cfdb7dfd6942c507edfa6d6eac3ff292f1cd4240ef92a08567d40175bfb49"} Jul 09 22:55:21.564633 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:55:21.564558 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-e98c97-openshift-default-58df5648d5-ljkpq" event={"ID":"5d550c17-9bf5-4730-bbe0-6f2305a0a90d","Type":"ContainerDied","Data":"7fca895f1883779e6f5d09fa7f470ae895780e45664bfb533685787f290ebbe7"} Jul 09 22:55:21.564633 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:55:21.564581 2579 scope.go:117] "RemoveContainer" containerID="c62cfdb7dfd6942c507edfa6d6eac3ff292f1cd4240ef92a08567d40175bfb49" Jul 09 22:55:21.574625 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:55:21.574600 2579 scope.go:117] "RemoveContainer" containerID="07d4ddf7447507f8e42c8db4370968f4f07ba16083b85688168b1262713f1228" Jul 09 22:55:21.583293 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:55:21.583272 2579 scope.go:117] "RemoveContainer" containerID="c62cfdb7dfd6942c507edfa6d6eac3ff292f1cd4240ef92a08567d40175bfb49" Jul 09 22:55:21.583575 ip-10-0-136-251 kubenswrapper[2579]: E0709 22:55:21.583559 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c62cfdb7dfd6942c507edfa6d6eac3ff292f1cd4240ef92a08567d40175bfb49\": container with ID starting with c62cfdb7dfd6942c507edfa6d6eac3ff292f1cd4240ef92a08567d40175bfb49 not found: ID does not exist" containerID="c62cfdb7dfd6942c507edfa6d6eac3ff292f1cd4240ef92a08567d40175bfb49" Jul 09 22:55:21.583616 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:55:21.583582 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c62cfdb7dfd6942c507edfa6d6eac3ff292f1cd4240ef92a08567d40175bfb49"} err="failed to get container status \"c62cfdb7dfd6942c507edfa6d6eac3ff292f1cd4240ef92a08567d40175bfb49\": rpc error: code = NotFound desc = could not find container \"c62cfdb7dfd6942c507edfa6d6eac3ff292f1cd4240ef92a08567d40175bfb49\": container with ID starting with c62cfdb7dfd6942c507edfa6d6eac3ff292f1cd4240ef92a08567d40175bfb49 not found: ID does not exist" Jul 09 22:55:21.583616 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:55:21.583599 2579 scope.go:117] "RemoveContainer" containerID="07d4ddf7447507f8e42c8db4370968f4f07ba16083b85688168b1262713f1228" Jul 09 22:55:21.583844 ip-10-0-136-251 kubenswrapper[2579]: E0709 22:55:21.583823 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"07d4ddf7447507f8e42c8db4370968f4f07ba16083b85688168b1262713f1228\": container with ID starting with 07d4ddf7447507f8e42c8db4370968f4f07ba16083b85688168b1262713f1228 not found: ID does not exist" containerID="07d4ddf7447507f8e42c8db4370968f4f07ba16083b85688168b1262713f1228" Jul 09 22:55:21.583882 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:55:21.583856 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"07d4ddf7447507f8e42c8db4370968f4f07ba16083b85688168b1262713f1228"} err="failed to get container status \"07d4ddf7447507f8e42c8db4370968f4f07ba16083b85688168b1262713f1228\": rpc error: code = NotFound desc = could not find container \"07d4ddf7447507f8e42c8db4370968f4f07ba16083b85688168b1262713f1228\": container with ID starting with 07d4ddf7447507f8e42c8db4370968f4f07ba16083b85688168b1262713f1228 not found: ID does not exist" Jul 09 22:55:21.588730 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:55:21.588702 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-e98c97-openshift-default-58df5648d5-ljkpq"] Jul 09 22:55:21.593899 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:55:21.593871 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-e98c97-openshift-default-58df5648d5-ljkpq"] Jul 09 22:55:22.565656 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:55:22.565622 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5d550c17-9bf5-4730-bbe0-6f2305a0a90d" path="/var/lib/kubelet/pods/5d550c17-9bf5-4730-bbe0-6f2305a0a90d/volumes" Jul 09 22:55:29.414519 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:55:29.414479 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-xdfrr"] Jul 09 22:55:41.703549 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:55:41.703509 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-6ddc6a-openshift-default-6c99cb4b89-2cmhb"] Jul 09 22:55:41.704114 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:55:41.703768 2579 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-6ddc6a-openshift-default-6c99cb4b89-2cmhb" podUID="4b5a3562-88ca-4ccc-b6bd-6367789bf658" containerName="istio-proxy" containerID="cri-o://7d19f5b7ff12e5867cad210842972321ca9981dfc5aec0867a4044dc5ca812eb" gracePeriod=30 Jul 09 22:55:42.609066 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:55:42.609019 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-8a6043-openshift-default-9b5ff877f-tnrjf"] Jul 09 22:55:42.609610 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:55:42.609590 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5d550c17-9bf5-4730-bbe0-6f2305a0a90d" containerName="istio-proxy" Jul 09 22:55:42.609610 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:55:42.609611 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="5d550c17-9bf5-4730-bbe0-6f2305a0a90d" containerName="istio-proxy" Jul 09 22:55:42.609849 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:55:42.609621 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="04e62c1d-d3ec-4760-b5e1-d8d31a1e653d" containerName="istio-proxy" Jul 09 22:55:42.609849 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:55:42.609627 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="04e62c1d-d3ec-4760-b5e1-d8d31a1e653d" containerName="istio-proxy" Jul 09 22:55:42.609849 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:55:42.609632 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5d550c17-9bf5-4730-bbe0-6f2305a0a90d" containerName="istio-proxy" Jul 09 22:55:42.609849 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:55:42.609638 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="5d550c17-9bf5-4730-bbe0-6f2305a0a90d" containerName="istio-proxy" Jul 09 22:55:42.609849 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:55:42.609753 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="5d550c17-9bf5-4730-bbe0-6f2305a0a90d" containerName="istio-proxy" Jul 09 22:55:42.609849 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:55:42.609761 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="5d550c17-9bf5-4730-bbe0-6f2305a0a90d" containerName="istio-proxy" Jul 09 22:55:42.614614 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:55:42.614588 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-8a6043-openshift-default-9b5ff877f-tnrjf" Jul 09 22:55:42.618086 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:55:42.618058 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-8a6043-openshift-default-dockercfg-m8rb7\"" Jul 09 22:55:42.630334 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:55:42.630304 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-8a6043-openshift-default-9b5ff877f-tnrjf"] Jul 09 22:55:42.733227 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:55:42.733188 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5d38a047-94c4-4dfb-aef7-8c96c09c4b33-istio-token\") pod \"e2e-shared-a-8a6043-openshift-default-9b5ff877f-tnrjf\" (UID: \"5d38a047-94c4-4dfb-aef7-8c96c09c4b33\") " pod="openshift-ingress/e2e-shared-a-8a6043-openshift-default-9b5ff877f-tnrjf" Jul 09 22:55:42.733227 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:55:42.733232 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d38a047-94c4-4dfb-aef7-8c96c09c4b33-workload-socket\") pod \"e2e-shared-a-8a6043-openshift-default-9b5ff877f-tnrjf\" (UID: \"5d38a047-94c4-4dfb-aef7-8c96c09c4b33\") " pod="openshift-ingress/e2e-shared-a-8a6043-openshift-default-9b5ff877f-tnrjf" Jul 09 22:55:42.733703 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:55:42.733267 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lb647\" (UniqueName: \"kubernetes.io/projected/5d38a047-94c4-4dfb-aef7-8c96c09c4b33-kube-api-access-lb647\") pod \"e2e-shared-a-8a6043-openshift-default-9b5ff877f-tnrjf\" (UID: \"5d38a047-94c4-4dfb-aef7-8c96c09c4b33\") " pod="openshift-ingress/e2e-shared-a-8a6043-openshift-default-9b5ff877f-tnrjf" Jul 09 22:55:42.733703 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:55:42.733333 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5d38a047-94c4-4dfb-aef7-8c96c09c4b33-istio-envoy\") pod \"e2e-shared-a-8a6043-openshift-default-9b5ff877f-tnrjf\" (UID: \"5d38a047-94c4-4dfb-aef7-8c96c09c4b33\") " pod="openshift-ingress/e2e-shared-a-8a6043-openshift-default-9b5ff877f-tnrjf" Jul 09 22:55:42.733703 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:55:42.733379 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5d38a047-94c4-4dfb-aef7-8c96c09c4b33-istio-data\") pod \"e2e-shared-a-8a6043-openshift-default-9b5ff877f-tnrjf\" (UID: \"5d38a047-94c4-4dfb-aef7-8c96c09c4b33\") " pod="openshift-ingress/e2e-shared-a-8a6043-openshift-default-9b5ff877f-tnrjf" Jul 09 22:55:42.733703 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:55:42.733415 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d38a047-94c4-4dfb-aef7-8c96c09c4b33-credential-socket\") pod \"e2e-shared-a-8a6043-openshift-default-9b5ff877f-tnrjf\" (UID: \"5d38a047-94c4-4dfb-aef7-8c96c09c4b33\") " pod="openshift-ingress/e2e-shared-a-8a6043-openshift-default-9b5ff877f-tnrjf" Jul 09 22:55:42.733703 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:55:42.733510 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5d38a047-94c4-4dfb-aef7-8c96c09c4b33-workload-certs\") pod \"e2e-shared-a-8a6043-openshift-default-9b5ff877f-tnrjf\" (UID: \"5d38a047-94c4-4dfb-aef7-8c96c09c4b33\") " pod="openshift-ingress/e2e-shared-a-8a6043-openshift-default-9b5ff877f-tnrjf" Jul 09 22:55:42.733703 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:55:42.733552 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5d38a047-94c4-4dfb-aef7-8c96c09c4b33-istio-podinfo\") pod \"e2e-shared-a-8a6043-openshift-default-9b5ff877f-tnrjf\" (UID: \"5d38a047-94c4-4dfb-aef7-8c96c09c4b33\") " pod="openshift-ingress/e2e-shared-a-8a6043-openshift-default-9b5ff877f-tnrjf" Jul 09 22:55:42.733703 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:55:42.733571 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5d38a047-94c4-4dfb-aef7-8c96c09c4b33-istiod-ca-cert\") pod \"e2e-shared-a-8a6043-openshift-default-9b5ff877f-tnrjf\" (UID: \"5d38a047-94c4-4dfb-aef7-8c96c09c4b33\") " pod="openshift-ingress/e2e-shared-a-8a6043-openshift-default-9b5ff877f-tnrjf" Jul 09 22:55:42.834875 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:55:42.834837 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5d38a047-94c4-4dfb-aef7-8c96c09c4b33-istio-data\") pod \"e2e-shared-a-8a6043-openshift-default-9b5ff877f-tnrjf\" (UID: \"5d38a047-94c4-4dfb-aef7-8c96c09c4b33\") " pod="openshift-ingress/e2e-shared-a-8a6043-openshift-default-9b5ff877f-tnrjf" Jul 09 22:55:42.835091 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:55:42.834898 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d38a047-94c4-4dfb-aef7-8c96c09c4b33-credential-socket\") pod \"e2e-shared-a-8a6043-openshift-default-9b5ff877f-tnrjf\" (UID: \"5d38a047-94c4-4dfb-aef7-8c96c09c4b33\") " pod="openshift-ingress/e2e-shared-a-8a6043-openshift-default-9b5ff877f-tnrjf" Jul 09 22:55:42.835091 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:55:42.834978 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5d38a047-94c4-4dfb-aef7-8c96c09c4b33-workload-certs\") pod \"e2e-shared-a-8a6043-openshift-default-9b5ff877f-tnrjf\" (UID: \"5d38a047-94c4-4dfb-aef7-8c96c09c4b33\") " pod="openshift-ingress/e2e-shared-a-8a6043-openshift-default-9b5ff877f-tnrjf" Jul 09 22:55:42.835091 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:55:42.835021 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5d38a047-94c4-4dfb-aef7-8c96c09c4b33-istio-podinfo\") pod \"e2e-shared-a-8a6043-openshift-default-9b5ff877f-tnrjf\" (UID: \"5d38a047-94c4-4dfb-aef7-8c96c09c4b33\") " pod="openshift-ingress/e2e-shared-a-8a6043-openshift-default-9b5ff877f-tnrjf" Jul 09 22:55:42.835091 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:55:42.835048 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5d38a047-94c4-4dfb-aef7-8c96c09c4b33-istiod-ca-cert\") pod \"e2e-shared-a-8a6043-openshift-default-9b5ff877f-tnrjf\" (UID: \"5d38a047-94c4-4dfb-aef7-8c96c09c4b33\") " pod="openshift-ingress/e2e-shared-a-8a6043-openshift-default-9b5ff877f-tnrjf" Jul 09 22:55:42.835091 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:55:42.835081 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5d38a047-94c4-4dfb-aef7-8c96c09c4b33-istio-token\") pod \"e2e-shared-a-8a6043-openshift-default-9b5ff877f-tnrjf\" (UID: \"5d38a047-94c4-4dfb-aef7-8c96c09c4b33\") " pod="openshift-ingress/e2e-shared-a-8a6043-openshift-default-9b5ff877f-tnrjf" Jul 09 22:55:42.835377 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:55:42.835114 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d38a047-94c4-4dfb-aef7-8c96c09c4b33-workload-socket\") pod \"e2e-shared-a-8a6043-openshift-default-9b5ff877f-tnrjf\" (UID: \"5d38a047-94c4-4dfb-aef7-8c96c09c4b33\") " pod="openshift-ingress/e2e-shared-a-8a6043-openshift-default-9b5ff877f-tnrjf" Jul 09 22:55:42.835377 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:55:42.835150 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lb647\" (UniqueName: \"kubernetes.io/projected/5d38a047-94c4-4dfb-aef7-8c96c09c4b33-kube-api-access-lb647\") pod \"e2e-shared-a-8a6043-openshift-default-9b5ff877f-tnrjf\" (UID: \"5d38a047-94c4-4dfb-aef7-8c96c09c4b33\") " pod="openshift-ingress/e2e-shared-a-8a6043-openshift-default-9b5ff877f-tnrjf" Jul 09 22:55:42.835377 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:55:42.835197 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5d38a047-94c4-4dfb-aef7-8c96c09c4b33-istio-envoy\") pod \"e2e-shared-a-8a6043-openshift-default-9b5ff877f-tnrjf\" (UID: \"5d38a047-94c4-4dfb-aef7-8c96c09c4b33\") " pod="openshift-ingress/e2e-shared-a-8a6043-openshift-default-9b5ff877f-tnrjf" Jul 09 22:55:42.835377 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:55:42.835271 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5d38a047-94c4-4dfb-aef7-8c96c09c4b33-istio-data\") pod \"e2e-shared-a-8a6043-openshift-default-9b5ff877f-tnrjf\" (UID: \"5d38a047-94c4-4dfb-aef7-8c96c09c4b33\") " pod="openshift-ingress/e2e-shared-a-8a6043-openshift-default-9b5ff877f-tnrjf" Jul 09 22:55:42.835608 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:55:42.835374 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5d38a047-94c4-4dfb-aef7-8c96c09c4b33-workload-certs\") pod \"e2e-shared-a-8a6043-openshift-default-9b5ff877f-tnrjf\" (UID: \"5d38a047-94c4-4dfb-aef7-8c96c09c4b33\") " pod="openshift-ingress/e2e-shared-a-8a6043-openshift-default-9b5ff877f-tnrjf" Jul 09 22:55:42.835608 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:55:42.835410 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d38a047-94c4-4dfb-aef7-8c96c09c4b33-credential-socket\") pod \"e2e-shared-a-8a6043-openshift-default-9b5ff877f-tnrjf\" (UID: \"5d38a047-94c4-4dfb-aef7-8c96c09c4b33\") " pod="openshift-ingress/e2e-shared-a-8a6043-openshift-default-9b5ff877f-tnrjf" Jul 09 22:55:42.835681 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:55:42.835623 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d38a047-94c4-4dfb-aef7-8c96c09c4b33-workload-socket\") pod \"e2e-shared-a-8a6043-openshift-default-9b5ff877f-tnrjf\" (UID: \"5d38a047-94c4-4dfb-aef7-8c96c09c4b33\") " pod="openshift-ingress/e2e-shared-a-8a6043-openshift-default-9b5ff877f-tnrjf" Jul 09 22:55:42.835844 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:55:42.835826 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5d38a047-94c4-4dfb-aef7-8c96c09c4b33-istiod-ca-cert\") pod \"e2e-shared-a-8a6043-openshift-default-9b5ff877f-tnrjf\" (UID: \"5d38a047-94c4-4dfb-aef7-8c96c09c4b33\") " pod="openshift-ingress/e2e-shared-a-8a6043-openshift-default-9b5ff877f-tnrjf" Jul 09 22:55:42.837479 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:55:42.837459 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5d38a047-94c4-4dfb-aef7-8c96c09c4b33-istio-envoy\") pod \"e2e-shared-a-8a6043-openshift-default-9b5ff877f-tnrjf\" (UID: \"5d38a047-94c4-4dfb-aef7-8c96c09c4b33\") " pod="openshift-ingress/e2e-shared-a-8a6043-openshift-default-9b5ff877f-tnrjf" Jul 09 22:55:42.837625 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:55:42.837606 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5d38a047-94c4-4dfb-aef7-8c96c09c4b33-istio-podinfo\") pod \"e2e-shared-a-8a6043-openshift-default-9b5ff877f-tnrjf\" (UID: \"5d38a047-94c4-4dfb-aef7-8c96c09c4b33\") " pod="openshift-ingress/e2e-shared-a-8a6043-openshift-default-9b5ff877f-tnrjf" Jul 09 22:55:42.845065 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:55:42.845028 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5d38a047-94c4-4dfb-aef7-8c96c09c4b33-istio-token\") pod \"e2e-shared-a-8a6043-openshift-default-9b5ff877f-tnrjf\" (UID: \"5d38a047-94c4-4dfb-aef7-8c96c09c4b33\") " pod="openshift-ingress/e2e-shared-a-8a6043-openshift-default-9b5ff877f-tnrjf" Jul 09 22:55:42.845218 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:55:42.845109 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lb647\" (UniqueName: \"kubernetes.io/projected/5d38a047-94c4-4dfb-aef7-8c96c09c4b33-kube-api-access-lb647\") pod \"e2e-shared-a-8a6043-openshift-default-9b5ff877f-tnrjf\" (UID: \"5d38a047-94c4-4dfb-aef7-8c96c09c4b33\") " pod="openshift-ingress/e2e-shared-a-8a6043-openshift-default-9b5ff877f-tnrjf" Jul 09 22:55:42.925405 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:55:42.925307 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-8a6043-openshift-default-9b5ff877f-tnrjf" Jul 09 22:55:43.076983 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:55:43.076954 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-8a6043-openshift-default-9b5ff877f-tnrjf"] Jul 09 22:55:43.078016 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:55:43.077979 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5d38a047_94c4_4dfb_aef7_8c96c09c4b33.slice/crio-a9afbfbf91ea365a6290ef58d8c53c96ca5deee82e3d785371be273488f66557 WatchSource:0}: Error finding container a9afbfbf91ea365a6290ef58d8c53c96ca5deee82e3d785371be273488f66557: Status 404 returned error can't find the container with id a9afbfbf91ea365a6290ef58d8c53c96ca5deee82e3d785371be273488f66557 Jul 09 22:55:43.080342 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:55:43.080304 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 09 22:55:43.080419 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:55:43.080397 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 09 22:55:43.080461 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:55:43.080450 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 09 22:55:43.685330 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:55:43.685288 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-8a6043-openshift-default-9b5ff877f-tnrjf" event={"ID":"5d38a047-94c4-4dfb-aef7-8c96c09c4b33","Type":"ContainerStarted","Data":"b267e23e798d2be20660c11893703c1997b518c5ee8ebefcfb3d5d8f97b30929"} Jul 09 22:55:43.685523 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:55:43.685360 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-8a6043-openshift-default-9b5ff877f-tnrjf" event={"ID":"5d38a047-94c4-4dfb-aef7-8c96c09c4b33","Type":"ContainerStarted","Data":"a9afbfbf91ea365a6290ef58d8c53c96ca5deee82e3d785371be273488f66557"} Jul 09 22:55:43.706430 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:55:43.706378 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-8a6043-openshift-default-9b5ff877f-tnrjf" podStartSLOduration=1.7063610850000002 podStartE2EDuration="1.706361085s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 22:55:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 22:55:43.704870268 +0000 UTC m=+2735.616322970" watchObservedRunningTime="2026-07-09 22:55:43.706361085 +0000 UTC m=+2735.617813785" Jul 09 22:55:43.925827 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:55:43.925783 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-8a6043-openshift-default-9b5ff877f-tnrjf" Jul 09 22:55:44.931031 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:55:44.930997 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-8a6043-openshift-default-9b5ff877f-tnrjf" Jul 09 22:55:45.703476 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:55:45.703435 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-8a6043-openshift-default-9b5ff877f-tnrjf" Jul 09 22:55:45.704551 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:55:45.704533 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-8a6043-openshift-default-9b5ff877f-tnrjf" Jul 09 22:55:46.966590 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:55:46.966566 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-6ddc6a-openshift-default-6c99cb4b89-2cmhb" Jul 09 22:55:47.075282 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:55:47.075233 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4b5a3562-88ca-4ccc-b6bd-6367789bf658-istiod-ca-cert\") pod \"4b5a3562-88ca-4ccc-b6bd-6367789bf658\" (UID: \"4b5a3562-88ca-4ccc-b6bd-6367789bf658\") " Jul 09 22:55:47.075513 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:55:47.075302 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b5a3562-88ca-4ccc-b6bd-6367789bf658-credential-socket\") pod \"4b5a3562-88ca-4ccc-b6bd-6367789bf658\" (UID: \"4b5a3562-88ca-4ccc-b6bd-6367789bf658\") " Jul 09 22:55:47.075513 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:55:47.075323 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2hwns\" (UniqueName: \"kubernetes.io/projected/4b5a3562-88ca-4ccc-b6bd-6367789bf658-kube-api-access-2hwns\") pod \"4b5a3562-88ca-4ccc-b6bd-6367789bf658\" (UID: \"4b5a3562-88ca-4ccc-b6bd-6367789bf658\") " Jul 09 22:55:47.075513 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:55:47.075361 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4b5a3562-88ca-4ccc-b6bd-6367789bf658-istio-token\") pod \"4b5a3562-88ca-4ccc-b6bd-6367789bf658\" (UID: \"4b5a3562-88ca-4ccc-b6bd-6367789bf658\") " Jul 09 22:55:47.075513 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:55:47.075380 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4b5a3562-88ca-4ccc-b6bd-6367789bf658-istio-data\") pod \"4b5a3562-88ca-4ccc-b6bd-6367789bf658\" (UID: \"4b5a3562-88ca-4ccc-b6bd-6367789bf658\") " Jul 09 22:55:47.075513 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:55:47.075411 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4b5a3562-88ca-4ccc-b6bd-6367789bf658-istio-envoy\") pod \"4b5a3562-88ca-4ccc-b6bd-6367789bf658\" (UID: \"4b5a3562-88ca-4ccc-b6bd-6367789bf658\") " Jul 09 22:55:47.075513 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:55:47.075438 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4b5a3562-88ca-4ccc-b6bd-6367789bf658-workload-certs\") pod \"4b5a3562-88ca-4ccc-b6bd-6367789bf658\" (UID: \"4b5a3562-88ca-4ccc-b6bd-6367789bf658\") " Jul 09 22:55:47.075513 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:55:47.075496 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4b5a3562-88ca-4ccc-b6bd-6367789bf658-istio-podinfo\") pod \"4b5a3562-88ca-4ccc-b6bd-6367789bf658\" (UID: \"4b5a3562-88ca-4ccc-b6bd-6367789bf658\") " Jul 09 22:55:47.075901 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:55:47.075545 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b5a3562-88ca-4ccc-b6bd-6367789bf658-workload-socket\") pod \"4b5a3562-88ca-4ccc-b6bd-6367789bf658\" (UID: \"4b5a3562-88ca-4ccc-b6bd-6367789bf658\") " Jul 09 22:55:47.075901 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:55:47.075593 2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4b5a3562-88ca-4ccc-b6bd-6367789bf658-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "4b5a3562-88ca-4ccc-b6bd-6367789bf658" (UID: "4b5a3562-88ca-4ccc-b6bd-6367789bf658"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 22:55:47.075901 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:55:47.075725 2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4b5a3562-88ca-4ccc-b6bd-6367789bf658-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "4b5a3562-88ca-4ccc-b6bd-6367789bf658" (UID: "4b5a3562-88ca-4ccc-b6bd-6367789bf658"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 22:55:47.075901 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:55:47.075766 2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4b5a3562-88ca-4ccc-b6bd-6367789bf658-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "4b5a3562-88ca-4ccc-b6bd-6367789bf658" (UID: "4b5a3562-88ca-4ccc-b6bd-6367789bf658"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 09 22:55:47.076172 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:55:47.075970 2579 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4b5a3562-88ca-4ccc-b6bd-6367789bf658-istiod-ca-cert\") on node \"ip-10-0-136-251.ec2.internal\" DevicePath \"\"" Jul 09 22:55:47.076172 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:55:47.075991 2579 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b5a3562-88ca-4ccc-b6bd-6367789bf658-credential-socket\") on node \"ip-10-0-136-251.ec2.internal\" DevicePath \"\"" Jul 09 22:55:47.076172 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:55:47.075992 2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4b5a3562-88ca-4ccc-b6bd-6367789bf658-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "4b5a3562-88ca-4ccc-b6bd-6367789bf658" (UID: "4b5a3562-88ca-4ccc-b6bd-6367789bf658"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 22:55:47.076172 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:55:47.076007 2579 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4b5a3562-88ca-4ccc-b6bd-6367789bf658-workload-certs\") on node \"ip-10-0-136-251.ec2.internal\" DevicePath \"\"" Jul 09 22:55:47.076474 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:55:47.076453 2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4b5a3562-88ca-4ccc-b6bd-6367789bf658-istio-data" (OuterVolumeSpecName: "istio-data") pod "4b5a3562-88ca-4ccc-b6bd-6367789bf658" (UID: "4b5a3562-88ca-4ccc-b6bd-6367789bf658"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 22:55:47.077804 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:55:47.077780 2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4b5a3562-88ca-4ccc-b6bd-6367789bf658-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "4b5a3562-88ca-4ccc-b6bd-6367789bf658" (UID: "4b5a3562-88ca-4ccc-b6bd-6367789bf658"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 09 22:55:47.077907 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:55:47.077803 2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4b5a3562-88ca-4ccc-b6bd-6367789bf658-istio-token" (OuterVolumeSpecName: "istio-token") pod "4b5a3562-88ca-4ccc-b6bd-6367789bf658" (UID: "4b5a3562-88ca-4ccc-b6bd-6367789bf658"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 22:55:47.078080 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:55:47.078062 2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4b5a3562-88ca-4ccc-b6bd-6367789bf658-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "4b5a3562-88ca-4ccc-b6bd-6367789bf658" (UID: "4b5a3562-88ca-4ccc-b6bd-6367789bf658"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 22:55:47.078136 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:55:47.078119 2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4b5a3562-88ca-4ccc-b6bd-6367789bf658-kube-api-access-2hwns" (OuterVolumeSpecName: "kube-api-access-2hwns") pod "4b5a3562-88ca-4ccc-b6bd-6367789bf658" (UID: "4b5a3562-88ca-4ccc-b6bd-6367789bf658"). InnerVolumeSpecName "kube-api-access-2hwns". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 22:55:47.176641 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:55:47.176591 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2hwns\" (UniqueName: \"kubernetes.io/projected/4b5a3562-88ca-4ccc-b6bd-6367789bf658-kube-api-access-2hwns\") on node \"ip-10-0-136-251.ec2.internal\" DevicePath \"\"" Jul 09 22:55:47.176641 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:55:47.176632 2579 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4b5a3562-88ca-4ccc-b6bd-6367789bf658-istio-token\") on node \"ip-10-0-136-251.ec2.internal\" DevicePath \"\"" Jul 09 22:55:47.176641 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:55:47.176643 2579 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4b5a3562-88ca-4ccc-b6bd-6367789bf658-istio-data\") on node \"ip-10-0-136-251.ec2.internal\" DevicePath \"\"" Jul 09 22:55:47.176641 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:55:47.176651 2579 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4b5a3562-88ca-4ccc-b6bd-6367789bf658-istio-envoy\") on node \"ip-10-0-136-251.ec2.internal\" DevicePath \"\"" Jul 09 22:55:47.176641 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:55:47.176659 2579 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4b5a3562-88ca-4ccc-b6bd-6367789bf658-istio-podinfo\") on node \"ip-10-0-136-251.ec2.internal\" DevicePath \"\"" Jul 09 22:55:47.177005 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:55:47.176667 2579 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b5a3562-88ca-4ccc-b6bd-6367789bf658-workload-socket\") on node \"ip-10-0-136-251.ec2.internal\" DevicePath \"\"" Jul 09 22:55:47.716572 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:55:47.716540 2579 generic.go:356] "Generic (PLEG): container finished" podID="4b5a3562-88ca-4ccc-b6bd-6367789bf658" containerID="7d19f5b7ff12e5867cad210842972321ca9981dfc5aec0867a4044dc5ca812eb" exitCode=0 Jul 09 22:55:47.716851 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:55:47.716589 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-6ddc6a-openshift-default-6c99cb4b89-2cmhb" Jul 09 22:55:47.716851 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:55:47.716630 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-6ddc6a-openshift-default-6c99cb4b89-2cmhb" event={"ID":"4b5a3562-88ca-4ccc-b6bd-6367789bf658","Type":"ContainerDied","Data":"7d19f5b7ff12e5867cad210842972321ca9981dfc5aec0867a4044dc5ca812eb"} Jul 09 22:55:47.716851 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:55:47.716685 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-6ddc6a-openshift-default-6c99cb4b89-2cmhb" event={"ID":"4b5a3562-88ca-4ccc-b6bd-6367789bf658","Type":"ContainerDied","Data":"2c8fbe252de7cb8fea57c9e6e929f9cb03bd094e4d65a38708b2f2e9f2ad5b65"} Jul 09 22:55:47.716851 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:55:47.716708 2579 scope.go:117] "RemoveContainer" containerID="7d19f5b7ff12e5867cad210842972321ca9981dfc5aec0867a4044dc5ca812eb" Jul 09 22:55:47.726524 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:55:47.726507 2579 scope.go:117] "RemoveContainer" containerID="e8c18da342a1db7f7b51cf1c24f491f5441cf1be95da218ff575aabd88857a69" Jul 09 22:55:47.735013 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:55:47.734989 2579 scope.go:117] "RemoveContainer" containerID="7d19f5b7ff12e5867cad210842972321ca9981dfc5aec0867a4044dc5ca812eb" Jul 09 22:55:47.735291 ip-10-0-136-251 kubenswrapper[2579]: E0709 22:55:47.735271 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7d19f5b7ff12e5867cad210842972321ca9981dfc5aec0867a4044dc5ca812eb\": container with ID starting with 7d19f5b7ff12e5867cad210842972321ca9981dfc5aec0867a4044dc5ca812eb not found: ID does not exist" containerID="7d19f5b7ff12e5867cad210842972321ca9981dfc5aec0867a4044dc5ca812eb" Jul 09 22:55:47.735341 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:55:47.735304 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7d19f5b7ff12e5867cad210842972321ca9981dfc5aec0867a4044dc5ca812eb"} err="failed to get container status \"7d19f5b7ff12e5867cad210842972321ca9981dfc5aec0867a4044dc5ca812eb\": rpc error: code = NotFound desc = could not find container \"7d19f5b7ff12e5867cad210842972321ca9981dfc5aec0867a4044dc5ca812eb\": container with ID starting with 7d19f5b7ff12e5867cad210842972321ca9981dfc5aec0867a4044dc5ca812eb not found: ID does not exist" Jul 09 22:55:47.735341 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:55:47.735324 2579 scope.go:117] "RemoveContainer" containerID="e8c18da342a1db7f7b51cf1c24f491f5441cf1be95da218ff575aabd88857a69" Jul 09 22:55:47.735546 ip-10-0-136-251 kubenswrapper[2579]: E0709 22:55:47.735527 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e8c18da342a1db7f7b51cf1c24f491f5441cf1be95da218ff575aabd88857a69\": container with ID starting with e8c18da342a1db7f7b51cf1c24f491f5441cf1be95da218ff575aabd88857a69 not found: ID does not exist" containerID="e8c18da342a1db7f7b51cf1c24f491f5441cf1be95da218ff575aabd88857a69" Jul 09 22:55:47.735589 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:55:47.735553 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e8c18da342a1db7f7b51cf1c24f491f5441cf1be95da218ff575aabd88857a69"} err="failed to get container status \"e8c18da342a1db7f7b51cf1c24f491f5441cf1be95da218ff575aabd88857a69\": rpc error: code = NotFound desc = could not find container \"e8c18da342a1db7f7b51cf1c24f491f5441cf1be95da218ff575aabd88857a69\": container with ID starting with e8c18da342a1db7f7b51cf1c24f491f5441cf1be95da218ff575aabd88857a69 not found: ID does not exist" Jul 09 22:55:47.741295 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:55:47.741265 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-6ddc6a-openshift-default-6c99cb4b89-2cmhb"] Jul 09 22:55:47.745241 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:55:47.745219 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-6ddc6a-openshift-default-6c99cb4b89-2cmhb"] Jul 09 22:55:48.564942 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:55:48.564887 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4b5a3562-88ca-4ccc-b6bd-6367789bf658" path="/var/lib/kubelet/pods/4b5a3562-88ca-4ccc-b6bd-6367789bf658/volumes" Jul 09 22:55:50.715950 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:55:50.712818 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-862c5d-openshift-default-f4fdbd455-xf87k"] Jul 09 22:55:50.715950 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:55:50.713767 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4b5a3562-88ca-4ccc-b6bd-6367789bf658" containerName="istio-proxy" Jul 09 22:55:50.715950 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:55:50.713790 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="4b5a3562-88ca-4ccc-b6bd-6367789bf658" containerName="istio-proxy" Jul 09 22:55:50.715950 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:55:50.713844 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4b5a3562-88ca-4ccc-b6bd-6367789bf658" containerName="istio-proxy" Jul 09 22:55:50.715950 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:55:50.713853 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="4b5a3562-88ca-4ccc-b6bd-6367789bf658" containerName="istio-proxy" Jul 09 22:55:50.715950 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:55:50.714044 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="4b5a3562-88ca-4ccc-b6bd-6367789bf658" containerName="istio-proxy" Jul 09 22:55:50.715950 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:55:50.714058 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="4b5a3562-88ca-4ccc-b6bd-6367789bf658" containerName="istio-proxy" Jul 09 22:55:50.720899 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:55:50.720865 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-862c5d-openshift-default-f4fdbd455-xf87k" Jul 09 22:55:50.723644 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:55:50.723622 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-862c5d-openshift-default-dockercfg-bx8gm\"" Jul 09 22:55:50.727985 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:55:50.727952 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-862c5d-openshift-default-f4fdbd455-xf87k"] Jul 09 22:55:50.814555 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:55:50.814517 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a3eee859-2f99-438f-b4ad-ca6575ecff6e-istio-envoy\") pod \"e2e-shared-b-862c5d-openshift-default-f4fdbd455-xf87k\" (UID: \"a3eee859-2f99-438f-b4ad-ca6575ecff6e\") " pod="openshift-ingress/e2e-shared-b-862c5d-openshift-default-f4fdbd455-xf87k" Jul 09 22:55:50.814994 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:55:50.814562 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a3eee859-2f99-438f-b4ad-ca6575ecff6e-istiod-ca-cert\") pod \"e2e-shared-b-862c5d-openshift-default-f4fdbd455-xf87k\" (UID: \"a3eee859-2f99-438f-b4ad-ca6575ecff6e\") " pod="openshift-ingress/e2e-shared-b-862c5d-openshift-default-f4fdbd455-xf87k" Jul 09 22:55:50.814994 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:55:50.814636 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a3eee859-2f99-438f-b4ad-ca6575ecff6e-workload-certs\") pod \"e2e-shared-b-862c5d-openshift-default-f4fdbd455-xf87k\" (UID: \"a3eee859-2f99-438f-b4ad-ca6575ecff6e\") " pod="openshift-ingress/e2e-shared-b-862c5d-openshift-default-f4fdbd455-xf87k" Jul 09 22:55:50.814994 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:55:50.814658 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a3eee859-2f99-438f-b4ad-ca6575ecff6e-istio-data\") pod \"e2e-shared-b-862c5d-openshift-default-f4fdbd455-xf87k\" (UID: \"a3eee859-2f99-438f-b4ad-ca6575ecff6e\") " pod="openshift-ingress/e2e-shared-b-862c5d-openshift-default-f4fdbd455-xf87k" Jul 09 22:55:50.814994 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:55:50.814709 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3eee859-2f99-438f-b4ad-ca6575ecff6e-credential-socket\") pod \"e2e-shared-b-862c5d-openshift-default-f4fdbd455-xf87k\" (UID: \"a3eee859-2f99-438f-b4ad-ca6575ecff6e\") " pod="openshift-ingress/e2e-shared-b-862c5d-openshift-default-f4fdbd455-xf87k" Jul 09 22:55:50.814994 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:55:50.814752 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a3eee859-2f99-438f-b4ad-ca6575ecff6e-istio-podinfo\") pod \"e2e-shared-b-862c5d-openshift-default-f4fdbd455-xf87k\" (UID: \"a3eee859-2f99-438f-b4ad-ca6575ecff6e\") " pod="openshift-ingress/e2e-shared-b-862c5d-openshift-default-f4fdbd455-xf87k" Jul 09 22:55:50.814994 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:55:50.814879 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v5xqf\" (UniqueName: \"kubernetes.io/projected/a3eee859-2f99-438f-b4ad-ca6575ecff6e-kube-api-access-v5xqf\") pod \"e2e-shared-b-862c5d-openshift-default-f4fdbd455-xf87k\" (UID: \"a3eee859-2f99-438f-b4ad-ca6575ecff6e\") " pod="openshift-ingress/e2e-shared-b-862c5d-openshift-default-f4fdbd455-xf87k" Jul 09 22:55:50.814994 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:55:50.814908 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a3eee859-2f99-438f-b4ad-ca6575ecff6e-istio-token\") pod \"e2e-shared-b-862c5d-openshift-default-f4fdbd455-xf87k\" (UID: \"a3eee859-2f99-438f-b4ad-ca6575ecff6e\") " pod="openshift-ingress/e2e-shared-b-862c5d-openshift-default-f4fdbd455-xf87k" Jul 09 22:55:50.815420 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:55:50.815045 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3eee859-2f99-438f-b4ad-ca6575ecff6e-workload-socket\") pod \"e2e-shared-b-862c5d-openshift-default-f4fdbd455-xf87k\" (UID: \"a3eee859-2f99-438f-b4ad-ca6575ecff6e\") " pod="openshift-ingress/e2e-shared-b-862c5d-openshift-default-f4fdbd455-xf87k" Jul 09 22:55:50.916215 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:55:50.916155 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3eee859-2f99-438f-b4ad-ca6575ecff6e-credential-socket\") pod \"e2e-shared-b-862c5d-openshift-default-f4fdbd455-xf87k\" (UID: \"a3eee859-2f99-438f-b4ad-ca6575ecff6e\") " pod="openshift-ingress/e2e-shared-b-862c5d-openshift-default-f4fdbd455-xf87k" Jul 09 22:55:50.916215 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:55:50.916225 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a3eee859-2f99-438f-b4ad-ca6575ecff6e-istio-podinfo\") pod \"e2e-shared-b-862c5d-openshift-default-f4fdbd455-xf87k\" (UID: \"a3eee859-2f99-438f-b4ad-ca6575ecff6e\") " pod="openshift-ingress/e2e-shared-b-862c5d-openshift-default-f4fdbd455-xf87k" Jul 09 22:55:50.916479 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:55:50.916276 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v5xqf\" (UniqueName: \"kubernetes.io/projected/a3eee859-2f99-438f-b4ad-ca6575ecff6e-kube-api-access-v5xqf\") pod \"e2e-shared-b-862c5d-openshift-default-f4fdbd455-xf87k\" (UID: \"a3eee859-2f99-438f-b4ad-ca6575ecff6e\") " pod="openshift-ingress/e2e-shared-b-862c5d-openshift-default-f4fdbd455-xf87k" Jul 09 22:55:50.916479 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:55:50.916298 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a3eee859-2f99-438f-b4ad-ca6575ecff6e-istio-token\") pod \"e2e-shared-b-862c5d-openshift-default-f4fdbd455-xf87k\" (UID: \"a3eee859-2f99-438f-b4ad-ca6575ecff6e\") " pod="openshift-ingress/e2e-shared-b-862c5d-openshift-default-f4fdbd455-xf87k" Jul 09 22:55:50.916479 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:55:50.916352 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3eee859-2f99-438f-b4ad-ca6575ecff6e-workload-socket\") pod \"e2e-shared-b-862c5d-openshift-default-f4fdbd455-xf87k\" (UID: \"a3eee859-2f99-438f-b4ad-ca6575ecff6e\") " pod="openshift-ingress/e2e-shared-b-862c5d-openshift-default-f4fdbd455-xf87k" Jul 09 22:55:50.916479 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:55:50.916374 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a3eee859-2f99-438f-b4ad-ca6575ecff6e-istio-envoy\") pod \"e2e-shared-b-862c5d-openshift-default-f4fdbd455-xf87k\" (UID: \"a3eee859-2f99-438f-b4ad-ca6575ecff6e\") " pod="openshift-ingress/e2e-shared-b-862c5d-openshift-default-f4fdbd455-xf87k" Jul 09 22:55:50.916479 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:55:50.916398 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a3eee859-2f99-438f-b4ad-ca6575ecff6e-istiod-ca-cert\") pod \"e2e-shared-b-862c5d-openshift-default-f4fdbd455-xf87k\" (UID: \"a3eee859-2f99-438f-b4ad-ca6575ecff6e\") " pod="openshift-ingress/e2e-shared-b-862c5d-openshift-default-f4fdbd455-xf87k" Jul 09 22:55:50.916479 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:55:50.916427 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a3eee859-2f99-438f-b4ad-ca6575ecff6e-workload-certs\") pod \"e2e-shared-b-862c5d-openshift-default-f4fdbd455-xf87k\" (UID: \"a3eee859-2f99-438f-b4ad-ca6575ecff6e\") " pod="openshift-ingress/e2e-shared-b-862c5d-openshift-default-f4fdbd455-xf87k" Jul 09 22:55:50.916479 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:55:50.916453 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a3eee859-2f99-438f-b4ad-ca6575ecff6e-istio-data\") pod \"e2e-shared-b-862c5d-openshift-default-f4fdbd455-xf87k\" (UID: \"a3eee859-2f99-438f-b4ad-ca6575ecff6e\") " pod="openshift-ingress/e2e-shared-b-862c5d-openshift-default-f4fdbd455-xf87k" Jul 09 22:55:50.916847 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:55:50.916607 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3eee859-2f99-438f-b4ad-ca6575ecff6e-credential-socket\") pod \"e2e-shared-b-862c5d-openshift-default-f4fdbd455-xf87k\" (UID: \"a3eee859-2f99-438f-b4ad-ca6575ecff6e\") " pod="openshift-ingress/e2e-shared-b-862c5d-openshift-default-f4fdbd455-xf87k" Jul 09 22:55:50.916847 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:55:50.916749 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3eee859-2f99-438f-b4ad-ca6575ecff6e-workload-socket\") pod \"e2e-shared-b-862c5d-openshift-default-f4fdbd455-xf87k\" (UID: \"a3eee859-2f99-438f-b4ad-ca6575ecff6e\") " pod="openshift-ingress/e2e-shared-b-862c5d-openshift-default-f4fdbd455-xf87k" Jul 09 22:55:50.916847 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:55:50.916786 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a3eee859-2f99-438f-b4ad-ca6575ecff6e-istio-data\") pod \"e2e-shared-b-862c5d-openshift-default-f4fdbd455-xf87k\" (UID: \"a3eee859-2f99-438f-b4ad-ca6575ecff6e\") " pod="openshift-ingress/e2e-shared-b-862c5d-openshift-default-f4fdbd455-xf87k" Jul 09 22:55:50.916998 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:55:50.916844 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a3eee859-2f99-438f-b4ad-ca6575ecff6e-workload-certs\") pod \"e2e-shared-b-862c5d-openshift-default-f4fdbd455-xf87k\" (UID: \"a3eee859-2f99-438f-b4ad-ca6575ecff6e\") " pod="openshift-ingress/e2e-shared-b-862c5d-openshift-default-f4fdbd455-xf87k" Jul 09 22:55:50.917166 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:55:50.917148 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a3eee859-2f99-438f-b4ad-ca6575ecff6e-istiod-ca-cert\") pod \"e2e-shared-b-862c5d-openshift-default-f4fdbd455-xf87k\" (UID: \"a3eee859-2f99-438f-b4ad-ca6575ecff6e\") " pod="openshift-ingress/e2e-shared-b-862c5d-openshift-default-f4fdbd455-xf87k" Jul 09 22:55:50.918752 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:55:50.918727 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a3eee859-2f99-438f-b4ad-ca6575ecff6e-istio-envoy\") pod \"e2e-shared-b-862c5d-openshift-default-f4fdbd455-xf87k\" (UID: \"a3eee859-2f99-438f-b4ad-ca6575ecff6e\") " pod="openshift-ingress/e2e-shared-b-862c5d-openshift-default-f4fdbd455-xf87k" Jul 09 22:55:50.918866 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:55:50.918802 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a3eee859-2f99-438f-b4ad-ca6575ecff6e-istio-podinfo\") pod \"e2e-shared-b-862c5d-openshift-default-f4fdbd455-xf87k\" (UID: \"a3eee859-2f99-438f-b4ad-ca6575ecff6e\") " pod="openshift-ingress/e2e-shared-b-862c5d-openshift-default-f4fdbd455-xf87k" Jul 09 22:55:50.933634 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:55:50.933603 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a3eee859-2f99-438f-b4ad-ca6575ecff6e-istio-token\") pod \"e2e-shared-b-862c5d-openshift-default-f4fdbd455-xf87k\" (UID: \"a3eee859-2f99-438f-b4ad-ca6575ecff6e\") " pod="openshift-ingress/e2e-shared-b-862c5d-openshift-default-f4fdbd455-xf87k" Jul 09 22:55:50.933775 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:55:50.933741 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v5xqf\" (UniqueName: \"kubernetes.io/projected/a3eee859-2f99-438f-b4ad-ca6575ecff6e-kube-api-access-v5xqf\") pod \"e2e-shared-b-862c5d-openshift-default-f4fdbd455-xf87k\" (UID: \"a3eee859-2f99-438f-b4ad-ca6575ecff6e\") " pod="openshift-ingress/e2e-shared-b-862c5d-openshift-default-f4fdbd455-xf87k" Jul 09 22:55:51.032941 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:55:51.032874 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-862c5d-openshift-default-f4fdbd455-xf87k" Jul 09 22:55:51.185007 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:55:51.184975 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-862c5d-openshift-default-f4fdbd455-xf87k"] Jul 09 22:55:51.186958 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:55:51.186905 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda3eee859_2f99_438f_b4ad_ca6575ecff6e.slice/crio-5122d7b22e0db70c9f99c7ac46c22d8979cdf2a0e0bd1c3bab3830bf084c0cc0 WatchSource:0}: Error finding container 5122d7b22e0db70c9f99c7ac46c22d8979cdf2a0e0bd1c3bab3830bf084c0cc0: Status 404 returned error can't find the container with id 5122d7b22e0db70c9f99c7ac46c22d8979cdf2a0e0bd1c3bab3830bf084c0cc0 Jul 09 22:55:51.189114 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:55:51.189084 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 09 22:55:51.189175 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:55:51.189152 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 09 22:55:51.189211 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:55:51.189180 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 09 22:55:51.742180 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:55:51.742129 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-862c5d-openshift-default-f4fdbd455-xf87k" event={"ID":"a3eee859-2f99-438f-b4ad-ca6575ecff6e","Type":"ContainerStarted","Data":"a5e11bb3de725cfb61f6103f960121ce6e549ebb4858c9298485ab93f5fb9ae7"} Jul 09 22:55:51.742649 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:55:51.742208 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-862c5d-openshift-default-f4fdbd455-xf87k" event={"ID":"a3eee859-2f99-438f-b4ad-ca6575ecff6e","Type":"ContainerStarted","Data":"5122d7b22e0db70c9f99c7ac46c22d8979cdf2a0e0bd1c3bab3830bf084c0cc0"} Jul 09 22:55:51.763936 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:55:51.763872 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-862c5d-openshift-default-f4fdbd455-xf87k" podStartSLOduration=1.763854549 podStartE2EDuration="1.763854549s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 22:55:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 22:55:51.761565985 +0000 UTC m=+2743.673018709" watchObservedRunningTime="2026-07-09 22:55:51.763854549 +0000 UTC m=+2743.675307248" Jul 09 22:55:52.033588 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:55:52.033496 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-862c5d-openshift-default-f4fdbd455-xf87k" Jul 09 22:55:52.035225 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:55:52.035188 2579 patch_prober.go:28] interesting pod/e2e-shared-b-862c5d-openshift-default-f4fdbd455-xf87k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" start-of-body= Jul 09 22:55:52.035370 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:55:52.035241 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-862c5d-openshift-default-f4fdbd455-xf87k" podUID="a3eee859-2f99-438f-b4ad-ca6575ecff6e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" Jul 09 22:55:53.037577 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:55:53.037547 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-862c5d-openshift-default-f4fdbd455-xf87k" Jul 09 22:55:53.320641 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:55:53.320552 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-xdfrr"] Jul 09 22:55:53.754618 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:55:53.754590 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-862c5d-openshift-default-f4fdbd455-xf87k" Jul 09 22:55:53.755670 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:55:53.755644 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-862c5d-openshift-default-f4fdbd455-xf87k" Jul 09 22:56:01.537245 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:56:01.536079 2579 patch_prober.go:28] interesting pod/e2e-shared-a-8a6043-openshift-default-9b5ff877f-tnrjf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.53:15021/healthz/ready\": read tcp 10.132.0.2:38800->10.132.0.53:15021: read: connection reset by peer" start-of-body= Jul 09 22:56:01.537245 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:56:01.536153 2579 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-8a6043-openshift-default-9b5ff877f-tnrjf" podUID="5d38a047-94c4-4dfb-aef7-8c96c09c4b33" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.53:15021/healthz/ready\": read tcp 10.132.0.2:38800->10.132.0.53:15021: read: connection reset by peer" Jul 09 22:56:01.805478 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:56:01.805379 2579 generic.go:356] "Generic (PLEG): container finished" podID="5d38a047-94c4-4dfb-aef7-8c96c09c4b33" containerID="b267e23e798d2be20660c11893703c1997b518c5ee8ebefcfb3d5d8f97b30929" exitCode=0 Jul 09 22:56:01.805478 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:56:01.805433 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-8a6043-openshift-default-9b5ff877f-tnrjf" event={"ID":"5d38a047-94c4-4dfb-aef7-8c96c09c4b33","Type":"ContainerDied","Data":"b267e23e798d2be20660c11893703c1997b518c5ee8ebefcfb3d5d8f97b30929"} Jul 09 22:56:01.805852 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:56:01.805837 2579 scope.go:117] "RemoveContainer" containerID="b267e23e798d2be20660c11893703c1997b518c5ee8ebefcfb3d5d8f97b30929" Jul 09 22:56:01.806895 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:56:01.806867 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 09 22:56:01.806974 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:56:01.806958 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 09 22:56:01.807012 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:56:01.806996 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 09 22:56:02.813690 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:56:02.813655 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-8a6043-openshift-default-9b5ff877f-tnrjf" event={"ID":"5d38a047-94c4-4dfb-aef7-8c96c09c4b33","Type":"ContainerStarted","Data":"15576633fcec968bf6daa19d2164fc8b8742e456e1be4c04a7be6d5e242aa00b"} Jul 09 22:56:02.925988 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:56:02.925952 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-8a6043-openshift-default-9b5ff877f-tnrjf" Jul 09 22:56:02.926337 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:56:02.926311 2579 patch_prober.go:28] interesting pod/e2e-shared-a-8a6043-openshift-default-9b5ff877f-tnrjf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" start-of-body= Jul 09 22:56:02.926458 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:56:02.926369 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-8a6043-openshift-default-9b5ff877f-tnrjf" podUID="5d38a047-94c4-4dfb-aef7-8c96c09c4b33" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" Jul 09 22:56:03.926294 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:56:03.926253 2579 patch_prober.go:28] interesting pod/e2e-shared-a-8a6043-openshift-default-9b5ff877f-tnrjf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" start-of-body= Jul 09 22:56:03.926687 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:56:03.926311 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-8a6043-openshift-default-9b5ff877f-tnrjf" podUID="5d38a047-94c4-4dfb-aef7-8c96c09c4b33" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" Jul 09 22:56:04.620219 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:56:04.620178 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-xdfrr"] Jul 09 22:56:04.926774 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:56:04.926664 2579 patch_prober.go:28] interesting pod/e2e-shared-a-8a6043-openshift-default-9b5ff877f-tnrjf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" start-of-body= Jul 09 22:56:04.926774 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:56:04.926727 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-8a6043-openshift-default-9b5ff877f-tnrjf" podUID="5d38a047-94c4-4dfb-aef7-8c96c09c4b33" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" Jul 09 22:56:05.926610 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:56:05.926571 2579 patch_prober.go:28] interesting pod/e2e-shared-a-8a6043-openshift-default-9b5ff877f-tnrjf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" start-of-body= Jul 09 22:56:05.926841 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:56:05.926628 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-8a6043-openshift-default-9b5ff877f-tnrjf" podUID="5d38a047-94c4-4dfb-aef7-8c96c09c4b33" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" Jul 09 22:56:06.926151 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:56:06.926108 2579 patch_prober.go:28] interesting pod/e2e-shared-a-8a6043-openshift-default-9b5ff877f-tnrjf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" start-of-body= Jul 09 22:56:06.926340 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:56:06.926180 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-8a6043-openshift-default-9b5ff877f-tnrjf" podUID="5d38a047-94c4-4dfb-aef7-8c96c09c4b33" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" Jul 09 22:56:07.930794 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:56:07.930758 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-8a6043-openshift-default-9b5ff877f-tnrjf" Jul 09 22:56:07.931200 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:56:07.931158 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-8a6043-openshift-default-9b5ff877f-tnrjf" Jul 09 22:56:07.931894 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:56:07.931875 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-8a6043-openshift-default-9b5ff877f-tnrjf" Jul 09 22:56:08.756284 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:56:08.756250 2579 patch_prober.go:28] interesting pod/e2e-shared-b-862c5d-openshift-default-f4fdbd455-xf87k container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" start-of-body= Jul 09 22:56:08.756488 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:56:08.756322 2579 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-862c5d-openshift-default-f4fdbd455-xf87k" podUID="a3eee859-2f99-438f-b4ad-ca6575ecff6e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" Jul 09 22:56:08.849536 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:56:08.849500 2579 generic.go:356] "Generic (PLEG): container finished" podID="a3eee859-2f99-438f-b4ad-ca6575ecff6e" containerID="a5e11bb3de725cfb61f6103f960121ce6e549ebb4858c9298485ab93f5fb9ae7" exitCode=0 Jul 09 22:56:08.849717 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:56:08.849577 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-862c5d-openshift-default-f4fdbd455-xf87k" event={"ID":"a3eee859-2f99-438f-b4ad-ca6575ecff6e","Type":"ContainerDied","Data":"a5e11bb3de725cfb61f6103f960121ce6e549ebb4858c9298485ab93f5fb9ae7"} Jul 09 22:56:08.850121 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:56:08.850105 2579 scope.go:117] "RemoveContainer" containerID="a5e11bb3de725cfb61f6103f960121ce6e549ebb4858c9298485ab93f5fb9ae7" Jul 09 22:56:08.851291 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:56:08.851264 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 09 22:56:08.851365 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:56:08.851344 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 09 22:56:08.851404 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:56:08.851387 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 09 22:56:09.863793 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:56:09.863747 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-862c5d-openshift-default-f4fdbd455-xf87k" event={"ID":"a3eee859-2f99-438f-b4ad-ca6575ecff6e","Type":"ContainerStarted","Data":"c4c6267c192d50f7d3d41c00c94f3526281c2cb44b384a699125396e657508fc"} Jul 09 22:56:10.033265 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:56:10.033226 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-862c5d-openshift-default-f4fdbd455-xf87k" Jul 09 22:56:10.033486 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:56:10.033461 2579 patch_prober.go:28] interesting pod/e2e-shared-b-862c5d-openshift-default-f4fdbd455-xf87k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" start-of-body= Jul 09 22:56:10.033541 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:56:10.033523 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-862c5d-openshift-default-f4fdbd455-xf87k" podUID="a3eee859-2f99-438f-b4ad-ca6575ecff6e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" Jul 09 22:56:11.034094 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:56:11.034054 2579 patch_prober.go:28] interesting pod/e2e-shared-b-862c5d-openshift-default-f4fdbd455-xf87k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" start-of-body= Jul 09 22:56:11.034522 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:56:11.034118 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-862c5d-openshift-default-f4fdbd455-xf87k" podUID="a3eee859-2f99-438f-b4ad-ca6575ecff6e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" Jul 09 22:56:12.033497 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:56:12.033460 2579 patch_prober.go:28] interesting pod/e2e-shared-b-862c5d-openshift-default-f4fdbd455-xf87k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" start-of-body= Jul 09 22:56:12.033692 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:56:12.033519 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-862c5d-openshift-default-f4fdbd455-xf87k" podUID="a3eee859-2f99-438f-b4ad-ca6575ecff6e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" Jul 09 22:56:13.034312 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:56:13.034273 2579 patch_prober.go:28] interesting pod/e2e-shared-b-862c5d-openshift-default-f4fdbd455-xf87k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" start-of-body= Jul 09 22:56:13.034698 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:56:13.034334 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-862c5d-openshift-default-f4fdbd455-xf87k" podUID="a3eee859-2f99-438f-b4ad-ca6575ecff6e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" Jul 09 22:56:14.038445 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:56:14.038409 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-862c5d-openshift-default-f4fdbd455-xf87k" Jul 09 22:56:14.038837 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:56:14.038749 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-862c5d-openshift-default-f4fdbd455-xf87k" Jul 09 22:56:14.039462 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:56:14.039446 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-862c5d-openshift-default-f4fdbd455-xf87k" Jul 09 22:56:17.119294 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:56:17.119259 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-xdfrr"] Jul 09 22:56:37.921771 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:56:37.921678 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-862c5d/auth-test-model-862c5d-kserve-7b46ffb775-4mwxg"] Jul 09 22:56:37.924257 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:56:37.924240 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-862c5d/auth-test-model-862c5d-kserve-7b46ffb775-4mwxg" Jul 09 22:56:37.926839 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:56:37.926818 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-862c5d\"/\"openshift-service-ca.crt\"" Jul 09 22:56:37.927025 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:56:37.926823 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-862c5d\"/\"default-dockercfg-m7996\"" Jul 09 22:56:37.928112 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:56:37.928090 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-862c5d\"/\"auth-test-model-862c5d-kserve-self-signed-certs\"" Jul 09 22:56:37.928217 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:56:37.928141 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-862c5d\"/\"kube-root-ca.crt\"" Jul 09 22:56:37.935400 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:56:37.935375 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-862c5d/auth-test-model-862c5d-kserve-7b46ffb775-4mwxg"] Jul 09 22:56:38.000906 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:56:38.000869 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/69f31793-5423-4a00-8045-bcc52ebe74c4-tls-certs\") pod \"auth-test-model-862c5d-kserve-7b46ffb775-4mwxg\" (UID: \"69f31793-5423-4a00-8045-bcc52ebe74c4\") " pod="ai-tenant-e2e-shared-b-862c5d/auth-test-model-862c5d-kserve-7b46ffb775-4mwxg" Jul 09 22:56:38.000906 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:56:38.000907 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/69f31793-5423-4a00-8045-bcc52ebe74c4-model-cache\") pod \"auth-test-model-862c5d-kserve-7b46ffb775-4mwxg\" (UID: \"69f31793-5423-4a00-8045-bcc52ebe74c4\") " pod="ai-tenant-e2e-shared-b-862c5d/auth-test-model-862c5d-kserve-7b46ffb775-4mwxg" Jul 09 22:56:38.001161 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:56:38.000951 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/69f31793-5423-4a00-8045-bcc52ebe74c4-home\") pod \"auth-test-model-862c5d-kserve-7b46ffb775-4mwxg\" (UID: \"69f31793-5423-4a00-8045-bcc52ebe74c4\") " pod="ai-tenant-e2e-shared-b-862c5d/auth-test-model-862c5d-kserve-7b46ffb775-4mwxg" Jul 09 22:56:38.001161 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:56:38.000973 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/69f31793-5423-4a00-8045-bcc52ebe74c4-tmp-dir\") pod \"auth-test-model-862c5d-kserve-7b46ffb775-4mwxg\" (UID: \"69f31793-5423-4a00-8045-bcc52ebe74c4\") " pod="ai-tenant-e2e-shared-b-862c5d/auth-test-model-862c5d-kserve-7b46ffb775-4mwxg" Jul 09 22:56:38.001161 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:56:38.000988 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/69f31793-5423-4a00-8045-bcc52ebe74c4-dshm\") pod \"auth-test-model-862c5d-kserve-7b46ffb775-4mwxg\" (UID: \"69f31793-5423-4a00-8045-bcc52ebe74c4\") " pod="ai-tenant-e2e-shared-b-862c5d/auth-test-model-862c5d-kserve-7b46ffb775-4mwxg" Jul 09 22:56:38.001161 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:56:38.001058 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xj2nc\" (UniqueName: \"kubernetes.io/projected/69f31793-5423-4a00-8045-bcc52ebe74c4-kube-api-access-xj2nc\") pod \"auth-test-model-862c5d-kserve-7b46ffb775-4mwxg\" (UID: \"69f31793-5423-4a00-8045-bcc52ebe74c4\") " pod="ai-tenant-e2e-shared-b-862c5d/auth-test-model-862c5d-kserve-7b46ffb775-4mwxg" Jul 09 22:56:38.102203 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:56:38.102148 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xj2nc\" (UniqueName: \"kubernetes.io/projected/69f31793-5423-4a00-8045-bcc52ebe74c4-kube-api-access-xj2nc\") pod \"auth-test-model-862c5d-kserve-7b46ffb775-4mwxg\" (UID: \"69f31793-5423-4a00-8045-bcc52ebe74c4\") " pod="ai-tenant-e2e-shared-b-862c5d/auth-test-model-862c5d-kserve-7b46ffb775-4mwxg" Jul 09 22:56:38.102406 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:56:38.102256 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/69f31793-5423-4a00-8045-bcc52ebe74c4-tls-certs\") pod \"auth-test-model-862c5d-kserve-7b46ffb775-4mwxg\" (UID: \"69f31793-5423-4a00-8045-bcc52ebe74c4\") " pod="ai-tenant-e2e-shared-b-862c5d/auth-test-model-862c5d-kserve-7b46ffb775-4mwxg" Jul 09 22:56:38.102406 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:56:38.102277 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/69f31793-5423-4a00-8045-bcc52ebe74c4-model-cache\") pod \"auth-test-model-862c5d-kserve-7b46ffb775-4mwxg\" (UID: \"69f31793-5423-4a00-8045-bcc52ebe74c4\") " pod="ai-tenant-e2e-shared-b-862c5d/auth-test-model-862c5d-kserve-7b46ffb775-4mwxg" Jul 09 22:56:38.102406 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:56:38.102309 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/69f31793-5423-4a00-8045-bcc52ebe74c4-home\") pod \"auth-test-model-862c5d-kserve-7b46ffb775-4mwxg\" (UID: \"69f31793-5423-4a00-8045-bcc52ebe74c4\") " pod="ai-tenant-e2e-shared-b-862c5d/auth-test-model-862c5d-kserve-7b46ffb775-4mwxg" Jul 09 22:56:38.102406 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:56:38.102326 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/69f31793-5423-4a00-8045-bcc52ebe74c4-tmp-dir\") pod \"auth-test-model-862c5d-kserve-7b46ffb775-4mwxg\" (UID: \"69f31793-5423-4a00-8045-bcc52ebe74c4\") " pod="ai-tenant-e2e-shared-b-862c5d/auth-test-model-862c5d-kserve-7b46ffb775-4mwxg" Jul 09 22:56:38.102406 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:56:38.102341 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/69f31793-5423-4a00-8045-bcc52ebe74c4-dshm\") pod \"auth-test-model-862c5d-kserve-7b46ffb775-4mwxg\" (UID: \"69f31793-5423-4a00-8045-bcc52ebe74c4\") " pod="ai-tenant-e2e-shared-b-862c5d/auth-test-model-862c5d-kserve-7b46ffb775-4mwxg" Jul 09 22:56:38.102789 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:56:38.102765 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/69f31793-5423-4a00-8045-bcc52ebe74c4-model-cache\") pod \"auth-test-model-862c5d-kserve-7b46ffb775-4mwxg\" (UID: \"69f31793-5423-4a00-8045-bcc52ebe74c4\") " pod="ai-tenant-e2e-shared-b-862c5d/auth-test-model-862c5d-kserve-7b46ffb775-4mwxg" Jul 09 22:56:38.102860 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:56:38.102807 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/69f31793-5423-4a00-8045-bcc52ebe74c4-home\") pod \"auth-test-model-862c5d-kserve-7b46ffb775-4mwxg\" (UID: \"69f31793-5423-4a00-8045-bcc52ebe74c4\") " pod="ai-tenant-e2e-shared-b-862c5d/auth-test-model-862c5d-kserve-7b46ffb775-4mwxg" Jul 09 22:56:38.102860 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:56:38.102828 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/69f31793-5423-4a00-8045-bcc52ebe74c4-tmp-dir\") pod \"auth-test-model-862c5d-kserve-7b46ffb775-4mwxg\" (UID: \"69f31793-5423-4a00-8045-bcc52ebe74c4\") " pod="ai-tenant-e2e-shared-b-862c5d/auth-test-model-862c5d-kserve-7b46ffb775-4mwxg" Jul 09 22:56:38.104683 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:56:38.104655 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/69f31793-5423-4a00-8045-bcc52ebe74c4-dshm\") pod \"auth-test-model-862c5d-kserve-7b46ffb775-4mwxg\" (UID: \"69f31793-5423-4a00-8045-bcc52ebe74c4\") " pod="ai-tenant-e2e-shared-b-862c5d/auth-test-model-862c5d-kserve-7b46ffb775-4mwxg" Jul 09 22:56:38.104832 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:56:38.104813 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/69f31793-5423-4a00-8045-bcc52ebe74c4-tls-certs\") pod \"auth-test-model-862c5d-kserve-7b46ffb775-4mwxg\" (UID: \"69f31793-5423-4a00-8045-bcc52ebe74c4\") " pod="ai-tenant-e2e-shared-b-862c5d/auth-test-model-862c5d-kserve-7b46ffb775-4mwxg" Jul 09 22:56:38.110588 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:56:38.110556 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xj2nc\" (UniqueName: \"kubernetes.io/projected/69f31793-5423-4a00-8045-bcc52ebe74c4-kube-api-access-xj2nc\") pod \"auth-test-model-862c5d-kserve-7b46ffb775-4mwxg\" (UID: \"69f31793-5423-4a00-8045-bcc52ebe74c4\") " pod="ai-tenant-e2e-shared-b-862c5d/auth-test-model-862c5d-kserve-7b46ffb775-4mwxg" Jul 09 22:56:38.235202 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:56:38.235091 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-862c5d/auth-test-model-862c5d-kserve-7b46ffb775-4mwxg" Jul 09 22:56:38.377343 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:56:38.377314 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-862c5d/auth-test-model-862c5d-kserve-7b46ffb775-4mwxg"] Jul 09 22:56:38.379539 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:56:38.379506 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod69f31793_5423_4a00_8045_bcc52ebe74c4.slice/crio-be2cb17fd17763a0b801236d691caece3852c5e509148c4f9ed2c893c18a6554 WatchSource:0}: Error finding container be2cb17fd17763a0b801236d691caece3852c5e509148c4f9ed2c893c18a6554: Status 404 returned error can't find the container with id be2cb17fd17763a0b801236d691caece3852c5e509148c4f9ed2c893c18a6554 Jul 09 22:56:38.381658 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:56:38.381635 2579 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 09 22:56:39.032595 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:56:39.032549 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-862c5d/auth-test-model-862c5d-kserve-7b46ffb775-4mwxg" event={"ID":"69f31793-5423-4a00-8045-bcc52ebe74c4","Type":"ContainerStarted","Data":"be2cb17fd17763a0b801236d691caece3852c5e509148c4f9ed2c893c18a6554"} Jul 09 22:56:40.040079 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:56:40.040047 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-862c5d/auth-test-model-862c5d-kserve-7b46ffb775-4mwxg" event={"ID":"69f31793-5423-4a00-8045-bcc52ebe74c4","Type":"ContainerStarted","Data":"370ca59bbe48f0a1de7084dad9dce53db11ea8cb2815a95ce1f0acfb56e5c940"} Jul 09 22:56:40.057129 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:56:40.057077 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-862c5d/auth-test-model-862c5d-kserve-7b46ffb775-4mwxg" podStartSLOduration=1.990370301 podStartE2EDuration="3.05706056s" totalImagesPullingTime="1.066690259s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 22:56:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 22:56:40.056247924 +0000 UTC m=+2791.967700626" watchObservedRunningTime="2026-07-09 22:56:40.05706056 +0000 UTC m=+2791.968513251" Jul 09 22:56:41.716719 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:56:41.716682 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-xdfrr"] Jul 09 22:56:48.236110 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:56:48.236073 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-862c5d/auth-test-model-862c5d-kserve-7b46ffb775-4mwxg" Jul 09 22:56:48.236531 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:56:48.236128 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-862c5d/auth-test-model-862c5d-kserve-7b46ffb775-4mwxg" Jul 09 22:56:48.248601 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:56:48.248575 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-862c5d/auth-test-model-862c5d-kserve-7b46ffb775-4mwxg" Jul 09 22:56:49.099018 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:56:49.098989 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-862c5d/auth-test-model-862c5d-kserve-7b46ffb775-4mwxg" Jul 09 22:57:02.434167 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:57:02.434128 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-xdfrr"] Jul 09 22:57:19.715004 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:57:19.714971 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-xdfrr"] Jul 09 22:57:37.219026 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:57:37.218988 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-xdfrr"] Jul 09 22:57:53.731314 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:57:53.731277 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-xdfrr"] Jul 09 22:58:11.319849 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:58:11.319759 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-xdfrr"] Jul 09 22:58:29.519617 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:58:29.519578 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-xdfrr"] Jul 09 22:58:43.150591 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:58:43.150555 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-862c5d/auth-test-model-862c5d-kserve-7b46ffb775-4mwxg"] Jul 09 22:58:45.008933 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:58:45.008892 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-xdfrr"] Jul 09 22:58:58.172250 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:58:58.172178 2579 kuberuntime_container.go:872] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-862c5d/auth-test-model-862c5d-kserve-7b46ffb775-4mwxg" podUID="69f31793-5423-4a00-8045-bcc52ebe74c4" containerName="main" containerID="cri-o://370ca59bbe48f0a1de7084dad9dce53db11ea8cb2815a95ce1f0acfb56e5c940" gracePeriod=45 Jul 09 22:58:58.428506 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:58:58.428436 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-862c5d/auth-test-model-862c5d-kserve-7b46ffb775-4mwxg" Jul 09 22:58:58.582468 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:58:58.582430 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xj2nc\" (UniqueName: \"kubernetes.io/projected/69f31793-5423-4a00-8045-bcc52ebe74c4-kube-api-access-xj2nc\") pod \"69f31793-5423-4a00-8045-bcc52ebe74c4\" (UID: \"69f31793-5423-4a00-8045-bcc52ebe74c4\") " Jul 09 22:58:58.582643 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:58:58.582484 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/69f31793-5423-4a00-8045-bcc52ebe74c4-dshm\") pod \"69f31793-5423-4a00-8045-bcc52ebe74c4\" (UID: \"69f31793-5423-4a00-8045-bcc52ebe74c4\") " Jul 09 22:58:58.582643 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:58:58.582554 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/69f31793-5423-4a00-8045-bcc52ebe74c4-home\") pod \"69f31793-5423-4a00-8045-bcc52ebe74c4\" (UID: \"69f31793-5423-4a00-8045-bcc52ebe74c4\") " Jul 09 22:58:58.582643 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:58:58.582582 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/69f31793-5423-4a00-8045-bcc52ebe74c4-model-cache\") pod \"69f31793-5423-4a00-8045-bcc52ebe74c4\" (UID: \"69f31793-5423-4a00-8045-bcc52ebe74c4\") " Jul 09 22:58:58.582643 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:58:58.582611 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/69f31793-5423-4a00-8045-bcc52ebe74c4-tmp-dir\") pod \"69f31793-5423-4a00-8045-bcc52ebe74c4\" (UID: \"69f31793-5423-4a00-8045-bcc52ebe74c4\") " Jul 09 22:58:58.582868 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:58:58.582673 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/69f31793-5423-4a00-8045-bcc52ebe74c4-tls-certs\") pod \"69f31793-5423-4a00-8045-bcc52ebe74c4\" (UID: \"69f31793-5423-4a00-8045-bcc52ebe74c4\") " Jul 09 22:58:58.582951 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:58:58.582883 2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/69f31793-5423-4a00-8045-bcc52ebe74c4-home" (OuterVolumeSpecName: "home") pod "69f31793-5423-4a00-8045-bcc52ebe74c4" (UID: "69f31793-5423-4a00-8045-bcc52ebe74c4"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 22:58:58.582951 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:58:58.582893 2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/69f31793-5423-4a00-8045-bcc52ebe74c4-model-cache" (OuterVolumeSpecName: "model-cache") pod "69f31793-5423-4a00-8045-bcc52ebe74c4" (UID: "69f31793-5423-4a00-8045-bcc52ebe74c4"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 22:58:58.583070 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:58:58.582910 2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/69f31793-5423-4a00-8045-bcc52ebe74c4-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "69f31793-5423-4a00-8045-bcc52ebe74c4" (UID: "69f31793-5423-4a00-8045-bcc52ebe74c4"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 22:58:58.583070 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:58:58.583047 2579 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/69f31793-5423-4a00-8045-bcc52ebe74c4-home\") on node \"ip-10-0-136-251.ec2.internal\" DevicePath \"\"" Jul 09 22:58:58.583070 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:58:58.583060 2579 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/69f31793-5423-4a00-8045-bcc52ebe74c4-model-cache\") on node \"ip-10-0-136-251.ec2.internal\" DevicePath \"\"" Jul 09 22:58:58.583234 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:58:58.583094 2579 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/69f31793-5423-4a00-8045-bcc52ebe74c4-tmp-dir\") on node \"ip-10-0-136-251.ec2.internal\" DevicePath \"\"" Jul 09 22:58:58.584788 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:58:58.584759 2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/69f31793-5423-4a00-8045-bcc52ebe74c4-dshm" (OuterVolumeSpecName: "dshm") pod "69f31793-5423-4a00-8045-bcc52ebe74c4" (UID: "69f31793-5423-4a00-8045-bcc52ebe74c4"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 22:58:58.584911 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:58:58.584834 2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/69f31793-5423-4a00-8045-bcc52ebe74c4-kube-api-access-xj2nc" (OuterVolumeSpecName: "kube-api-access-xj2nc") pod "69f31793-5423-4a00-8045-bcc52ebe74c4" (UID: "69f31793-5423-4a00-8045-bcc52ebe74c4"). InnerVolumeSpecName "kube-api-access-xj2nc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 22:58:58.584911 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:58:58.584842 2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/69f31793-5423-4a00-8045-bcc52ebe74c4-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "69f31793-5423-4a00-8045-bcc52ebe74c4" (UID: "69f31793-5423-4a00-8045-bcc52ebe74c4"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 09 22:58:58.684278 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:58:58.684184 2579 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/69f31793-5423-4a00-8045-bcc52ebe74c4-tls-certs\") on node \"ip-10-0-136-251.ec2.internal\" DevicePath \"\"" Jul 09 22:58:58.684278 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:58:58.684218 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xj2nc\" (UniqueName: \"kubernetes.io/projected/69f31793-5423-4a00-8045-bcc52ebe74c4-kube-api-access-xj2nc\") on node \"ip-10-0-136-251.ec2.internal\" DevicePath \"\"" Jul 09 22:58:58.684278 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:58:58.684230 2579 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/69f31793-5423-4a00-8045-bcc52ebe74c4-dshm\") on node \"ip-10-0-136-251.ec2.internal\" DevicePath \"\"" Jul 09 22:58:58.782225 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:58:58.782184 2579 generic.go:356] "Generic (PLEG): container finished" podID="69f31793-5423-4a00-8045-bcc52ebe74c4" containerID="370ca59bbe48f0a1de7084dad9dce53db11ea8cb2815a95ce1f0acfb56e5c940" exitCode=0 Jul 09 22:58:58.782225 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:58:58.782234 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-862c5d/auth-test-model-862c5d-kserve-7b46ffb775-4mwxg" Jul 09 22:58:58.782478 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:58:58.782267 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-862c5d/auth-test-model-862c5d-kserve-7b46ffb775-4mwxg" event={"ID":"69f31793-5423-4a00-8045-bcc52ebe74c4","Type":"ContainerDied","Data":"370ca59bbe48f0a1de7084dad9dce53db11ea8cb2815a95ce1f0acfb56e5c940"} Jul 09 22:58:58.782478 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:58:58.782307 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-862c5d/auth-test-model-862c5d-kserve-7b46ffb775-4mwxg" event={"ID":"69f31793-5423-4a00-8045-bcc52ebe74c4","Type":"ContainerDied","Data":"be2cb17fd17763a0b801236d691caece3852c5e509148c4f9ed2c893c18a6554"} Jul 09 22:58:58.782478 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:58:58.782325 2579 scope.go:117] "RemoveContainer" containerID="370ca59bbe48f0a1de7084dad9dce53db11ea8cb2815a95ce1f0acfb56e5c940" Jul 09 22:58:58.792711 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:58:58.792692 2579 scope.go:117] "RemoveContainer" containerID="370ca59bbe48f0a1de7084dad9dce53db11ea8cb2815a95ce1f0acfb56e5c940" Jul 09 22:58:58.793049 ip-10-0-136-251 kubenswrapper[2579]: E0709 22:58:58.793026 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"370ca59bbe48f0a1de7084dad9dce53db11ea8cb2815a95ce1f0acfb56e5c940\": container with ID starting with 370ca59bbe48f0a1de7084dad9dce53db11ea8cb2815a95ce1f0acfb56e5c940 not found: ID does not exist" containerID="370ca59bbe48f0a1de7084dad9dce53db11ea8cb2815a95ce1f0acfb56e5c940" Jul 09 22:58:58.793139 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:58:58.793056 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"370ca59bbe48f0a1de7084dad9dce53db11ea8cb2815a95ce1f0acfb56e5c940"} err="failed to get container status \"370ca59bbe48f0a1de7084dad9dce53db11ea8cb2815a95ce1f0acfb56e5c940\": rpc error: code = NotFound desc = could not find container \"370ca59bbe48f0a1de7084dad9dce53db11ea8cb2815a95ce1f0acfb56e5c940\": container with ID starting with 370ca59bbe48f0a1de7084dad9dce53db11ea8cb2815a95ce1f0acfb56e5c940 not found: ID does not exist" Jul 09 22:58:58.807536 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:58:58.807502 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-862c5d/auth-test-model-862c5d-kserve-7b46ffb775-4mwxg"] Jul 09 22:58:58.809613 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:58:58.809588 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-862c5d/auth-test-model-862c5d-kserve-7b46ffb775-4mwxg"] Jul 09 22:59:00.565757 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:59:00.565720 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="69f31793-5423-4a00-8045-bcc52ebe74c4" path="/var/lib/kubelet/pods/69f31793-5423-4a00-8045-bcc52ebe74c4/volumes" Jul 09 22:59:02.522748 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:59:02.522710 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-xdfrr"] Jul 09 22:59:11.609434 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:59:11.609389 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-862c5d/sub-test-model-862c5d-kserve-766649cbb7-bfmc4"] Jul 09 22:59:11.609879 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:59:11.609854 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="69f31793-5423-4a00-8045-bcc52ebe74c4" containerName="main" Jul 09 22:59:11.609879 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:59:11.609869 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="69f31793-5423-4a00-8045-bcc52ebe74c4" containerName="main" Jul 09 22:59:11.609991 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:59:11.609978 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="69f31793-5423-4a00-8045-bcc52ebe74c4" containerName="main" Jul 09 22:59:11.614439 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:59:11.614416 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-862c5d/sub-test-model-862c5d-kserve-766649cbb7-bfmc4" Jul 09 22:59:11.617993 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:59:11.617972 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-862c5d\"/\"default-dockercfg-m7996\"" Jul 09 22:59:11.617993 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:59:11.617985 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-862c5d\"/\"openshift-service-ca.crt\"" Jul 09 22:59:11.618198 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:59:11.617984 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-862c5d\"/\"sub-test-model-862c5d-kserve-self-signed-certs\"" Jul 09 22:59:11.618198 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:59:11.618032 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-862c5d\"/\"kube-root-ca.crt\"" Jul 09 22:59:11.622964 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:59:11.622943 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-862c5d/sub-test-model-862c5d-kserve-766649cbb7-bfmc4"] Jul 09 22:59:11.709200 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:59:11.709164 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bf3f1939-e8e5-42ec-8677-d4cd75068a67-model-cache\") pod \"sub-test-model-862c5d-kserve-766649cbb7-bfmc4\" (UID: \"bf3f1939-e8e5-42ec-8677-d4cd75068a67\") " pod="ai-tenant-e2e-shared-b-862c5d/sub-test-model-862c5d-kserve-766649cbb7-bfmc4" Jul 09 22:59:11.709395 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:59:11.709226 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bf3f1939-e8e5-42ec-8677-d4cd75068a67-tmp-dir\") pod \"sub-test-model-862c5d-kserve-766649cbb7-bfmc4\" (UID: \"bf3f1939-e8e5-42ec-8677-d4cd75068a67\") " pod="ai-tenant-e2e-shared-b-862c5d/sub-test-model-862c5d-kserve-766649cbb7-bfmc4" Jul 09 22:59:11.709395 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:59:11.709253 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k7kjm\" (UniqueName: \"kubernetes.io/projected/bf3f1939-e8e5-42ec-8677-d4cd75068a67-kube-api-access-k7kjm\") pod \"sub-test-model-862c5d-kserve-766649cbb7-bfmc4\" (UID: \"bf3f1939-e8e5-42ec-8677-d4cd75068a67\") " pod="ai-tenant-e2e-shared-b-862c5d/sub-test-model-862c5d-kserve-766649cbb7-bfmc4" Jul 09 22:59:11.709395 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:59:11.709342 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bf3f1939-e8e5-42ec-8677-d4cd75068a67-home\") pod \"sub-test-model-862c5d-kserve-766649cbb7-bfmc4\" (UID: \"bf3f1939-e8e5-42ec-8677-d4cd75068a67\") " pod="ai-tenant-e2e-shared-b-862c5d/sub-test-model-862c5d-kserve-766649cbb7-bfmc4" Jul 09 22:59:11.709395 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:59:11.709365 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bf3f1939-e8e5-42ec-8677-d4cd75068a67-tls-certs\") pod \"sub-test-model-862c5d-kserve-766649cbb7-bfmc4\" (UID: \"bf3f1939-e8e5-42ec-8677-d4cd75068a67\") " pod="ai-tenant-e2e-shared-b-862c5d/sub-test-model-862c5d-kserve-766649cbb7-bfmc4" Jul 09 22:59:11.709560 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:59:11.709403 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bf3f1939-e8e5-42ec-8677-d4cd75068a67-dshm\") pod \"sub-test-model-862c5d-kserve-766649cbb7-bfmc4\" (UID: \"bf3f1939-e8e5-42ec-8677-d4cd75068a67\") " pod="ai-tenant-e2e-shared-b-862c5d/sub-test-model-862c5d-kserve-766649cbb7-bfmc4" Jul 09 22:59:11.809821 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:59:11.809780 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bf3f1939-e8e5-42ec-8677-d4cd75068a67-home\") pod \"sub-test-model-862c5d-kserve-766649cbb7-bfmc4\" (UID: \"bf3f1939-e8e5-42ec-8677-d4cd75068a67\") " pod="ai-tenant-e2e-shared-b-862c5d/sub-test-model-862c5d-kserve-766649cbb7-bfmc4" Jul 09 22:59:11.809821 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:59:11.809820 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bf3f1939-e8e5-42ec-8677-d4cd75068a67-tls-certs\") pod \"sub-test-model-862c5d-kserve-766649cbb7-bfmc4\" (UID: \"bf3f1939-e8e5-42ec-8677-d4cd75068a67\") " pod="ai-tenant-e2e-shared-b-862c5d/sub-test-model-862c5d-kserve-766649cbb7-bfmc4" Jul 09 22:59:11.810122 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:59:11.809842 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bf3f1939-e8e5-42ec-8677-d4cd75068a67-dshm\") pod \"sub-test-model-862c5d-kserve-766649cbb7-bfmc4\" (UID: \"bf3f1939-e8e5-42ec-8677-d4cd75068a67\") " pod="ai-tenant-e2e-shared-b-862c5d/sub-test-model-862c5d-kserve-766649cbb7-bfmc4" Jul 09 22:59:11.810122 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:59:11.809864 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bf3f1939-e8e5-42ec-8677-d4cd75068a67-model-cache\") pod \"sub-test-model-862c5d-kserve-766649cbb7-bfmc4\" (UID: \"bf3f1939-e8e5-42ec-8677-d4cd75068a67\") " pod="ai-tenant-e2e-shared-b-862c5d/sub-test-model-862c5d-kserve-766649cbb7-bfmc4" Jul 09 22:59:11.810122 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:59:11.809933 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bf3f1939-e8e5-42ec-8677-d4cd75068a67-tmp-dir\") pod \"sub-test-model-862c5d-kserve-766649cbb7-bfmc4\" (UID: \"bf3f1939-e8e5-42ec-8677-d4cd75068a67\") " pod="ai-tenant-e2e-shared-b-862c5d/sub-test-model-862c5d-kserve-766649cbb7-bfmc4" Jul 09 22:59:11.810122 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:59:11.809956 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k7kjm\" (UniqueName: \"kubernetes.io/projected/bf3f1939-e8e5-42ec-8677-d4cd75068a67-kube-api-access-k7kjm\") pod \"sub-test-model-862c5d-kserve-766649cbb7-bfmc4\" (UID: \"bf3f1939-e8e5-42ec-8677-d4cd75068a67\") " pod="ai-tenant-e2e-shared-b-862c5d/sub-test-model-862c5d-kserve-766649cbb7-bfmc4" Jul 09 22:59:11.810344 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:59:11.810289 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bf3f1939-e8e5-42ec-8677-d4cd75068a67-home\") pod \"sub-test-model-862c5d-kserve-766649cbb7-bfmc4\" (UID: \"bf3f1939-e8e5-42ec-8677-d4cd75068a67\") " pod="ai-tenant-e2e-shared-b-862c5d/sub-test-model-862c5d-kserve-766649cbb7-bfmc4" Jul 09 22:59:11.810406 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:59:11.810354 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bf3f1939-e8e5-42ec-8677-d4cd75068a67-model-cache\") pod \"sub-test-model-862c5d-kserve-766649cbb7-bfmc4\" (UID: \"bf3f1939-e8e5-42ec-8677-d4cd75068a67\") " pod="ai-tenant-e2e-shared-b-862c5d/sub-test-model-862c5d-kserve-766649cbb7-bfmc4" Jul 09 22:59:11.810462 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:59:11.810405 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bf3f1939-e8e5-42ec-8677-d4cd75068a67-tmp-dir\") pod \"sub-test-model-862c5d-kserve-766649cbb7-bfmc4\" (UID: \"bf3f1939-e8e5-42ec-8677-d4cd75068a67\") " pod="ai-tenant-e2e-shared-b-862c5d/sub-test-model-862c5d-kserve-766649cbb7-bfmc4" Jul 09 22:59:11.812219 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:59:11.812199 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bf3f1939-e8e5-42ec-8677-d4cd75068a67-dshm\") pod \"sub-test-model-862c5d-kserve-766649cbb7-bfmc4\" (UID: \"bf3f1939-e8e5-42ec-8677-d4cd75068a67\") " pod="ai-tenant-e2e-shared-b-862c5d/sub-test-model-862c5d-kserve-766649cbb7-bfmc4" Jul 09 22:59:11.812539 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:59:11.812519 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bf3f1939-e8e5-42ec-8677-d4cd75068a67-tls-certs\") pod \"sub-test-model-862c5d-kserve-766649cbb7-bfmc4\" (UID: \"bf3f1939-e8e5-42ec-8677-d4cd75068a67\") " pod="ai-tenant-e2e-shared-b-862c5d/sub-test-model-862c5d-kserve-766649cbb7-bfmc4" Jul 09 22:59:11.817794 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:59:11.817767 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k7kjm\" (UniqueName: \"kubernetes.io/projected/bf3f1939-e8e5-42ec-8677-d4cd75068a67-kube-api-access-k7kjm\") pod \"sub-test-model-862c5d-kserve-766649cbb7-bfmc4\" (UID: \"bf3f1939-e8e5-42ec-8677-d4cd75068a67\") " pod="ai-tenant-e2e-shared-b-862c5d/sub-test-model-862c5d-kserve-766649cbb7-bfmc4" Jul 09 22:59:11.924514 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:59:11.924411 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-862c5d/sub-test-model-862c5d-kserve-766649cbb7-bfmc4" Jul 09 22:59:12.067323 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:59:12.067297 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-862c5d/sub-test-model-862c5d-kserve-766649cbb7-bfmc4"] Jul 09 22:59:12.069007 ip-10-0-136-251 kubenswrapper[2579]: W0709 22:59:12.068976 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podbf3f1939_e8e5_42ec_8677_d4cd75068a67.slice/crio-8c6ff70d8ae0eef77edffcba75b35b3af6ab90aee3a1a4245d47345f9e44aacc WatchSource:0}: Error finding container 8c6ff70d8ae0eef77edffcba75b35b3af6ab90aee3a1a4245d47345f9e44aacc: Status 404 returned error can't find the container with id 8c6ff70d8ae0eef77edffcba75b35b3af6ab90aee3a1a4245d47345f9e44aacc Jul 09 22:59:12.860013 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:59:12.859974 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-862c5d/sub-test-model-862c5d-kserve-766649cbb7-bfmc4" event={"ID":"bf3f1939-e8e5-42ec-8677-d4cd75068a67","Type":"ContainerStarted","Data":"db0b8e12dd4d881f0712836201ab4f35446cd4f8d7e85ac9cfed6bf46f662d04"} Jul 09 22:59:12.860013 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:59:12.860027 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-862c5d/sub-test-model-862c5d-kserve-766649cbb7-bfmc4" event={"ID":"bf3f1939-e8e5-42ec-8677-d4cd75068a67","Type":"ContainerStarted","Data":"8c6ff70d8ae0eef77edffcba75b35b3af6ab90aee3a1a4245d47345f9e44aacc"} Jul 09 22:59:12.878076 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:59:12.878022 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-862c5d/sub-test-model-862c5d-kserve-766649cbb7-bfmc4" podStartSLOduration=1.8780067200000001 podStartE2EDuration="1.87800672s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 22:59:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 22:59:12.875979899 +0000 UTC m=+2944.787432598" watchObservedRunningTime="2026-07-09 22:59:12.87800672 +0000 UTC m=+2944.789459397" Jul 09 22:59:18.116436 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:59:18.116398 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-xdfrr"] Jul 09 22:59:21.924774 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:59:21.924731 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-862c5d/sub-test-model-862c5d-kserve-766649cbb7-bfmc4" Jul 09 22:59:21.925232 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:59:21.924788 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-862c5d/sub-test-model-862c5d-kserve-766649cbb7-bfmc4" Jul 09 22:59:21.937680 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:59:21.937649 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-862c5d/sub-test-model-862c5d-kserve-766649cbb7-bfmc4" Jul 09 22:59:22.927703 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:59:22.927673 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-862c5d/sub-test-model-862c5d-kserve-766649cbb7-bfmc4" Jul 09 22:59:32.527882 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:59:32.527796 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-xdfrr"] Jul 09 22:59:49.512984 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:59:49.512940 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-862c5d/sub-test-model-862c5d-kserve-766649cbb7-bfmc4"] Jul 09 22:59:51.822451 ip-10-0-136-251 kubenswrapper[2579]: I0709 22:59:51.822411 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-xdfrr"] Jul 09 23:00:00.161563 ip-10-0-136-251 kubenswrapper[2579]: I0709 23:00:00.161527 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-862c5d-29727300-78j45"] Jul 09 23:00:00.165259 ip-10-0-136-251 kubenswrapper[2579]: I0709 23:00:00.165241 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-862c5d-29727300-78j45" Jul 09 23:00:00.168260 ip-10-0-136-251 kubenswrapper[2579]: I0709 23:00:00.168238 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fdzhc\"" Jul 09 23:00:00.172608 ip-10-0-136-251 kubenswrapper[2579]: I0709 23:00:00.172580 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29727300-mfvpq"] Jul 09 23:00:00.176361 ip-10-0-136-251 kubenswrapper[2579]: I0709 23:00:00.176339 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29727300-mfvpq" Jul 09 23:00:00.176479 ip-10-0-136-251 kubenswrapper[2579]: I0709 23:00:00.176441 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-862c5d-29727300-78j45" Jul 09 23:00:00.182885 ip-10-0-136-251 kubenswrapper[2579]: I0709 23:00:00.182805 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-8a6043-29727300-4rt7k"] Jul 09 23:00:00.186457 ip-10-0-136-251 kubenswrapper[2579]: I0709 23:00:00.186432 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29727300-mfvpq" Jul 09 23:00:00.187455 ip-10-0-136-251 kubenswrapper[2579]: I0709 23:00:00.187435 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-8a6043-29727300-4rt7k" Jul 09 23:00:00.190783 ip-10-0-136-251 kubenswrapper[2579]: I0709 23:00:00.190755 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-862c5d-29727300-78j45"] Jul 09 23:00:00.193440 ip-10-0-136-251 kubenswrapper[2579]: I0709 23:00:00.193416 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29727300-mfvpq"] Jul 09 23:00:00.205679 ip-10-0-136-251 kubenswrapper[2579]: I0709 23:00:00.205626 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-8a6043-29727300-4rt7k"] Jul 09 23:00:00.231227 ip-10-0-136-251 kubenswrapper[2579]: I0709 23:00:00.231204 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-8a6043-29727300-4rt7k" Jul 09 23:00:00.358532 ip-10-0-136-251 kubenswrapper[2579]: I0709 23:00:00.358503 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-862c5d-29727300-78j45"] Jul 09 23:00:00.359502 ip-10-0-136-251 kubenswrapper[2579]: W0709 23:00:00.359462 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf5ddc88c_bc30_4c14_aa1f_dd54f4a8e633.slice/crio-5caf36fbd9df8b96b55320a6e407b588ee158b48936e8e0e30bf22c085d4e8fe WatchSource:0}: Error finding container 5caf36fbd9df8b96b55320a6e407b588ee158b48936e8e0e30bf22c085d4e8fe: Status 404 returned error can't find the container with id 5caf36fbd9df8b96b55320a6e407b588ee158b48936e8e0e30bf22c085d4e8fe Jul 09 23:00:00.389204 ip-10-0-136-251 kubenswrapper[2579]: I0709 23:00:00.389108 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29727300-mfvpq"] Jul 09 23:00:00.392462 ip-10-0-136-251 kubenswrapper[2579]: W0709 23:00:00.392431 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf3972d63_c4f4_4011_bea3_681bc562086d.slice/crio-b7c8fe6b43752f2df709da0f8f7b28c9f782200f878e7a9f9ebf0a6fa30bde2b WatchSource:0}: Error finding container b7c8fe6b43752f2df709da0f8f7b28c9f782200f878e7a9f9ebf0a6fa30bde2b: Status 404 returned error can't find the container with id b7c8fe6b43752f2df709da0f8f7b28c9f782200f878e7a9f9ebf0a6fa30bde2b Jul 09 23:00:00.420108 ip-10-0-136-251 kubenswrapper[2579]: I0709 23:00:00.420085 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-8a6043-29727300-4rt7k"] Jul 09 23:00:00.425118 ip-10-0-136-251 kubenswrapper[2579]: W0709 23:00:00.425091 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod834b2c8b_6388_43b8_ba59_a1ab73ddfb1f.slice/crio-854a75d48a3ad7e6c6b5f88fe6f10cec331eb9d99b44f29d395db7f55b0ae33c WatchSource:0}: Error finding container 854a75d48a3ad7e6c6b5f88fe6f10cec331eb9d99b44f29d395db7f55b0ae33c: Status 404 returned error can't find the container with id 854a75d48a3ad7e6c6b5f88fe6f10cec331eb9d99b44f29d395db7f55b0ae33c Jul 09 23:00:01.129055 ip-10-0-136-251 kubenswrapper[2579]: I0709 23:00:01.128951 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-862c5d-29727300-78j45" event={"ID":"f5ddc88c-bc30-4c14-aa1f-dd54f4a8e633","Type":"ContainerStarted","Data":"81d748b8c2edeeea55e6e4df2cc3c60cb506e18c73c3d8ec595894ac5e5b15b4"} Jul 09 23:00:01.129055 ip-10-0-136-251 kubenswrapper[2579]: I0709 23:00:01.129017 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-862c5d-29727300-78j45" event={"ID":"f5ddc88c-bc30-4c14-aa1f-dd54f4a8e633","Type":"ContainerStarted","Data":"5caf36fbd9df8b96b55320a6e407b588ee158b48936e8e0e30bf22c085d4e8fe"} Jul 09 23:00:01.130603 ip-10-0-136-251 kubenswrapper[2579]: I0709 23:00:01.130578 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-8a6043-29727300-4rt7k" event={"ID":"834b2c8b-6388-43b8-ba59-a1ab73ddfb1f","Type":"ContainerStarted","Data":"ddd87fa44ffcaef349bdf8c3bb7e0c5a57f67c84624dae93367a531e1ef79a51"} Jul 09 23:00:01.130718 ip-10-0-136-251 kubenswrapper[2579]: I0709 23:00:01.130623 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-8a6043-29727300-4rt7k" event={"ID":"834b2c8b-6388-43b8-ba59-a1ab73ddfb1f","Type":"ContainerStarted","Data":"854a75d48a3ad7e6c6b5f88fe6f10cec331eb9d99b44f29d395db7f55b0ae33c"} Jul 09 23:00:01.132031 ip-10-0-136-251 kubenswrapper[2579]: I0709 23:00:01.132010 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29727300-mfvpq" event={"ID":"f3972d63-c4f4-4011-bea3-681bc562086d","Type":"ContainerStarted","Data":"f6081ae11d2fdcbeb4db1523cd30496dd5fa85a1a19c41503ebbf121d1c8a5fc"} Jul 09 23:00:01.132157 ip-10-0-136-251 kubenswrapper[2579]: I0709 23:00:01.132057 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29727300-mfvpq" event={"ID":"f3972d63-c4f4-4011-bea3-681bc562086d","Type":"ContainerStarted","Data":"b7c8fe6b43752f2df709da0f8f7b28c9f782200f878e7a9f9ebf0a6fa30bde2b"} Jul 09 23:00:01.146048 ip-10-0-136-251 kubenswrapper[2579]: I0709 23:00:01.145994 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-862c5d-29727300-78j45" podStartSLOduration=1.145976825 podStartE2EDuration="1.145976825s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 23:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 23:00:01.143645475 +0000 UTC m=+2993.055098172" watchObservedRunningTime="2026-07-09 23:00:01.145976825 +0000 UTC m=+2993.057429525" Jul 09 23:00:01.163276 ip-10-0-136-251 kubenswrapper[2579]: I0709 23:00:01.163214 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29727300-mfvpq" podStartSLOduration=1.163192712 podStartE2EDuration="1.163192712s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 23:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 23:00:01.160837305 +0000 UTC m=+2993.072290031" watchObservedRunningTime="2026-07-09 23:00:01.163192712 +0000 UTC m=+2993.074645411" Jul 09 23:00:02.139183 ip-10-0-136-251 kubenswrapper[2579]: I0709 23:00:02.139143 2579 generic.go:356] "Generic (PLEG): container finished" podID="834b2c8b-6388-43b8-ba59-a1ab73ddfb1f" containerID="ddd87fa44ffcaef349bdf8c3bb7e0c5a57f67c84624dae93367a531e1ef79a51" exitCode=0 Jul 09 23:00:02.139401 ip-10-0-136-251 kubenswrapper[2579]: I0709 23:00:02.139232 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-8a6043-29727300-4rt7k" event={"ID":"834b2c8b-6388-43b8-ba59-a1ab73ddfb1f","Type":"ContainerDied","Data":"ddd87fa44ffcaef349bdf8c3bb7e0c5a57f67c84624dae93367a531e1ef79a51"} Jul 09 23:00:02.140693 ip-10-0-136-251 kubenswrapper[2579]: I0709 23:00:02.140671 2579 generic.go:356] "Generic (PLEG): container finished" podID="f3972d63-c4f4-4011-bea3-681bc562086d" containerID="f6081ae11d2fdcbeb4db1523cd30496dd5fa85a1a19c41503ebbf121d1c8a5fc" exitCode=0 Jul 09 23:00:02.140811 ip-10-0-136-251 kubenswrapper[2579]: I0709 23:00:02.140748 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29727300-mfvpq" event={"ID":"f3972d63-c4f4-4011-bea3-681bc562086d","Type":"ContainerDied","Data":"f6081ae11d2fdcbeb4db1523cd30496dd5fa85a1a19c41503ebbf121d1c8a5fc"} Jul 09 23:00:02.142037 ip-10-0-136-251 kubenswrapper[2579]: I0709 23:00:02.142015 2579 generic.go:356] "Generic (PLEG): container finished" podID="f5ddc88c-bc30-4c14-aa1f-dd54f4a8e633" containerID="81d748b8c2edeeea55e6e4df2cc3c60cb506e18c73c3d8ec595894ac5e5b15b4" exitCode=0 Jul 09 23:00:02.142139 ip-10-0-136-251 kubenswrapper[2579]: I0709 23:00:02.142097 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-862c5d-29727300-78j45" event={"ID":"f5ddc88c-bc30-4c14-aa1f-dd54f4a8e633","Type":"ContainerDied","Data":"81d748b8c2edeeea55e6e4df2cc3c60cb506e18c73c3d8ec595894ac5e5b15b4"} Jul 09 23:00:03.311823 ip-10-0-136-251 kubenswrapper[2579]: I0709 23:00:03.311792 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29727300-mfvpq" Jul 09 23:00:03.343640 ip-10-0-136-251 kubenswrapper[2579]: I0709 23:00:03.343614 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-8a6043-29727300-4rt7k" Jul 09 23:00:03.346891 ip-10-0-136-251 kubenswrapper[2579]: I0709 23:00:03.346867 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-862c5d-29727300-78j45" Jul 09 23:00:04.156205 ip-10-0-136-251 kubenswrapper[2579]: I0709 23:00:04.156174 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-8a6043-29727300-4rt7k" Jul 09 23:00:04.156403 ip-10-0-136-251 kubenswrapper[2579]: I0709 23:00:04.156175 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-8a6043-29727300-4rt7k" event={"ID":"834b2c8b-6388-43b8-ba59-a1ab73ddfb1f","Type":"ContainerDied","Data":"854a75d48a3ad7e6c6b5f88fe6f10cec331eb9d99b44f29d395db7f55b0ae33c"} Jul 09 23:00:04.156403 ip-10-0-136-251 kubenswrapper[2579]: I0709 23:00:04.156306 2579 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="854a75d48a3ad7e6c6b5f88fe6f10cec331eb9d99b44f29d395db7f55b0ae33c" Jul 09 23:00:04.157372 ip-10-0-136-251 kubenswrapper[2579]: I0709 23:00:04.157354 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29727300-mfvpq" Jul 09 23:00:04.157458 ip-10-0-136-251 kubenswrapper[2579]: I0709 23:00:04.157387 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29727300-mfvpq" event={"ID":"f3972d63-c4f4-4011-bea3-681bc562086d","Type":"ContainerDied","Data":"b7c8fe6b43752f2df709da0f8f7b28c9f782200f878e7a9f9ebf0a6fa30bde2b"} Jul 09 23:00:04.157458 ip-10-0-136-251 kubenswrapper[2579]: I0709 23:00:04.157433 2579 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b7c8fe6b43752f2df709da0f8f7b28c9f782200f878e7a9f9ebf0a6fa30bde2b" Jul 09 23:00:04.158740 ip-10-0-136-251 kubenswrapper[2579]: I0709 23:00:04.158720 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-862c5d-29727300-78j45" Jul 09 23:00:04.158858 ip-10-0-136-251 kubenswrapper[2579]: I0709 23:00:04.158725 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-862c5d-29727300-78j45" event={"ID":"f5ddc88c-bc30-4c14-aa1f-dd54f4a8e633","Type":"ContainerDied","Data":"5caf36fbd9df8b96b55320a6e407b588ee158b48936e8e0e30bf22c085d4e8fe"} Jul 09 23:00:04.158858 ip-10-0-136-251 kubenswrapper[2579]: I0709 23:00:04.158843 2579 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5caf36fbd9df8b96b55320a6e407b588ee158b48936e8e0e30bf22c085d4e8fe" Jul 09 23:00:04.534099 ip-10-0-136-251 kubenswrapper[2579]: I0709 23:00:04.534055 2579 kuberuntime_container.go:872] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-862c5d/sub-test-model-862c5d-kserve-766649cbb7-bfmc4" podUID="bf3f1939-e8e5-42ec-8677-d4cd75068a67" containerName="main" containerID="cri-o://db0b8e12dd4d881f0712836201ab4f35446cd4f8d7e85ac9cfed6bf46f662d04" gracePeriod=45 Jul 09 23:00:04.803718 ip-10-0-136-251 kubenswrapper[2579]: I0709 23:00:04.803692 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-862c5d/sub-test-model-862c5d-kserve-766649cbb7-bfmc4" Jul 09 23:00:04.913335 ip-10-0-136-251 kubenswrapper[2579]: I0709 23:00:04.913293 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bf3f1939-e8e5-42ec-8677-d4cd75068a67-dshm\") pod \"bf3f1939-e8e5-42ec-8677-d4cd75068a67\" (UID: \"bf3f1939-e8e5-42ec-8677-d4cd75068a67\") " Jul 09 23:00:04.913548 ip-10-0-136-251 kubenswrapper[2579]: I0709 23:00:04.913353 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bf3f1939-e8e5-42ec-8677-d4cd75068a67-home\") pod \"bf3f1939-e8e5-42ec-8677-d4cd75068a67\" (UID: \"bf3f1939-e8e5-42ec-8677-d4cd75068a67\") " Jul 09 23:00:04.913548 ip-10-0-136-251 kubenswrapper[2579]: I0709 23:00:04.913383 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bf3f1939-e8e5-42ec-8677-d4cd75068a67-model-cache\") pod \"bf3f1939-e8e5-42ec-8677-d4cd75068a67\" (UID: \"bf3f1939-e8e5-42ec-8677-d4cd75068a67\") " Jul 09 23:00:04.913548 ip-10-0-136-251 kubenswrapper[2579]: I0709 23:00:04.913432 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bf3f1939-e8e5-42ec-8677-d4cd75068a67-tmp-dir\") pod \"bf3f1939-e8e5-42ec-8677-d4cd75068a67\" (UID: \"bf3f1939-e8e5-42ec-8677-d4cd75068a67\") " Jul 09 23:00:04.913548 ip-10-0-136-251 kubenswrapper[2579]: I0709 23:00:04.913460 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bf3f1939-e8e5-42ec-8677-d4cd75068a67-tls-certs\") pod \"bf3f1939-e8e5-42ec-8677-d4cd75068a67\" (UID: \"bf3f1939-e8e5-42ec-8677-d4cd75068a67\") " Jul 09 23:00:04.913786 ip-10-0-136-251 kubenswrapper[2579]: I0709 23:00:04.913551 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-k7kjm\" (UniqueName: \"kubernetes.io/projected/bf3f1939-e8e5-42ec-8677-d4cd75068a67-kube-api-access-k7kjm\") pod \"bf3f1939-e8e5-42ec-8677-d4cd75068a67\" (UID: \"bf3f1939-e8e5-42ec-8677-d4cd75068a67\") " Jul 09 23:00:04.913786 ip-10-0-136-251 kubenswrapper[2579]: I0709 23:00:04.913627 2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf3f1939-e8e5-42ec-8677-d4cd75068a67-home" (OuterVolumeSpecName: "home") pod "bf3f1939-e8e5-42ec-8677-d4cd75068a67" (UID: "bf3f1939-e8e5-42ec-8677-d4cd75068a67"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 23:00:04.913904 ip-10-0-136-251 kubenswrapper[2579]: I0709 23:00:04.913827 2579 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bf3f1939-e8e5-42ec-8677-d4cd75068a67-home\") on node \"ip-10-0-136-251.ec2.internal\" DevicePath \"\"" Jul 09 23:00:04.914078 ip-10-0-136-251 kubenswrapper[2579]: I0709 23:00:04.914051 2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf3f1939-e8e5-42ec-8677-d4cd75068a67-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "bf3f1939-e8e5-42ec-8677-d4cd75068a67" (UID: "bf3f1939-e8e5-42ec-8677-d4cd75068a67"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 23:00:04.914363 ip-10-0-136-251 kubenswrapper[2579]: I0709 23:00:04.914330 2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf3f1939-e8e5-42ec-8677-d4cd75068a67-model-cache" (OuterVolumeSpecName: "model-cache") pod "bf3f1939-e8e5-42ec-8677-d4cd75068a67" (UID: "bf3f1939-e8e5-42ec-8677-d4cd75068a67"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 23:00:04.915645 ip-10-0-136-251 kubenswrapper[2579]: I0709 23:00:04.915622 2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf3f1939-e8e5-42ec-8677-d4cd75068a67-dshm" (OuterVolumeSpecName: "dshm") pod "bf3f1939-e8e5-42ec-8677-d4cd75068a67" (UID: "bf3f1939-e8e5-42ec-8677-d4cd75068a67"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 23:00:04.915746 ip-10-0-136-251 kubenswrapper[2579]: I0709 23:00:04.915727 2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bf3f1939-e8e5-42ec-8677-d4cd75068a67-kube-api-access-k7kjm" (OuterVolumeSpecName: "kube-api-access-k7kjm") pod "bf3f1939-e8e5-42ec-8677-d4cd75068a67" (UID: "bf3f1939-e8e5-42ec-8677-d4cd75068a67"). InnerVolumeSpecName "kube-api-access-k7kjm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 23:00:04.915796 ip-10-0-136-251 kubenswrapper[2579]: I0709 23:00:04.915775 2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bf3f1939-e8e5-42ec-8677-d4cd75068a67-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "bf3f1939-e8e5-42ec-8677-d4cd75068a67" (UID: "bf3f1939-e8e5-42ec-8677-d4cd75068a67"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 09 23:00:05.014991 ip-10-0-136-251 kubenswrapper[2579]: I0709 23:00:05.014949 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k7kjm\" (UniqueName: \"kubernetes.io/projected/bf3f1939-e8e5-42ec-8677-d4cd75068a67-kube-api-access-k7kjm\") on node \"ip-10-0-136-251.ec2.internal\" DevicePath \"\"" Jul 09 23:00:05.014991 ip-10-0-136-251 kubenswrapper[2579]: I0709 23:00:05.014983 2579 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bf3f1939-e8e5-42ec-8677-d4cd75068a67-dshm\") on node \"ip-10-0-136-251.ec2.internal\" DevicePath \"\"" Jul 09 23:00:05.014991 ip-10-0-136-251 kubenswrapper[2579]: I0709 23:00:05.014994 2579 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bf3f1939-e8e5-42ec-8677-d4cd75068a67-model-cache\") on node \"ip-10-0-136-251.ec2.internal\" DevicePath \"\"" Jul 09 23:00:05.014991 ip-10-0-136-251 kubenswrapper[2579]: I0709 23:00:05.015003 2579 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bf3f1939-e8e5-42ec-8677-d4cd75068a67-tmp-dir\") on node \"ip-10-0-136-251.ec2.internal\" DevicePath \"\"" Jul 09 23:00:05.015274 ip-10-0-136-251 kubenswrapper[2579]: I0709 23:00:05.015011 2579 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bf3f1939-e8e5-42ec-8677-d4cd75068a67-tls-certs\") on node \"ip-10-0-136-251.ec2.internal\" DevicePath \"\"" Jul 09 23:00:05.165083 ip-10-0-136-251 kubenswrapper[2579]: I0709 23:00:05.164983 2579 generic.go:356] "Generic (PLEG): container finished" podID="bf3f1939-e8e5-42ec-8677-d4cd75068a67" containerID="db0b8e12dd4d881f0712836201ab4f35446cd4f8d7e85ac9cfed6bf46f662d04" exitCode=0 Jul 09 23:00:05.165083 ip-10-0-136-251 kubenswrapper[2579]: I0709 23:00:05.165031 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-862c5d/sub-test-model-862c5d-kserve-766649cbb7-bfmc4" Jul 09 23:00:05.165271 ip-10-0-136-251 kubenswrapper[2579]: I0709 23:00:05.165076 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-862c5d/sub-test-model-862c5d-kserve-766649cbb7-bfmc4" event={"ID":"bf3f1939-e8e5-42ec-8677-d4cd75068a67","Type":"ContainerDied","Data":"db0b8e12dd4d881f0712836201ab4f35446cd4f8d7e85ac9cfed6bf46f662d04"} Jul 09 23:00:05.165271 ip-10-0-136-251 kubenswrapper[2579]: I0709 23:00:05.165117 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-862c5d/sub-test-model-862c5d-kserve-766649cbb7-bfmc4" event={"ID":"bf3f1939-e8e5-42ec-8677-d4cd75068a67","Type":"ContainerDied","Data":"8c6ff70d8ae0eef77edffcba75b35b3af6ab90aee3a1a4245d47345f9e44aacc"} Jul 09 23:00:05.165271 ip-10-0-136-251 kubenswrapper[2579]: I0709 23:00:05.165137 2579 scope.go:117] "RemoveContainer" containerID="db0b8e12dd4d881f0712836201ab4f35446cd4f8d7e85ac9cfed6bf46f662d04" Jul 09 23:00:05.178093 ip-10-0-136-251 kubenswrapper[2579]: I0709 23:00:05.178016 2579 scope.go:117] "RemoveContainer" containerID="db0b8e12dd4d881f0712836201ab4f35446cd4f8d7e85ac9cfed6bf46f662d04" Jul 09 23:00:05.178698 ip-10-0-136-251 kubenswrapper[2579]: E0709 23:00:05.178659 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"db0b8e12dd4d881f0712836201ab4f35446cd4f8d7e85ac9cfed6bf46f662d04\": container with ID starting with db0b8e12dd4d881f0712836201ab4f35446cd4f8d7e85ac9cfed6bf46f662d04 not found: ID does not exist" containerID="db0b8e12dd4d881f0712836201ab4f35446cd4f8d7e85ac9cfed6bf46f662d04" Jul 09 23:00:05.178793 ip-10-0-136-251 kubenswrapper[2579]: I0709 23:00:05.178714 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"db0b8e12dd4d881f0712836201ab4f35446cd4f8d7e85ac9cfed6bf46f662d04"} err="failed to get container status \"db0b8e12dd4d881f0712836201ab4f35446cd4f8d7e85ac9cfed6bf46f662d04\": rpc error: code = NotFound desc = could not find container \"db0b8e12dd4d881f0712836201ab4f35446cd4f8d7e85ac9cfed6bf46f662d04\": container with ID starting with db0b8e12dd4d881f0712836201ab4f35446cd4f8d7e85ac9cfed6bf46f662d04 not found: ID does not exist" Jul 09 23:00:05.190394 ip-10-0-136-251 kubenswrapper[2579]: I0709 23:00:05.190365 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-862c5d/sub-test-model-862c5d-kserve-766649cbb7-bfmc4"] Jul 09 23:00:05.192568 ip-10-0-136-251 kubenswrapper[2579]: I0709 23:00:05.192545 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-862c5d/sub-test-model-862c5d-kserve-766649cbb7-bfmc4"] Jul 09 23:00:06.565091 ip-10-0-136-251 kubenswrapper[2579]: I0709 23:00:06.565056 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bf3f1939-e8e5-42ec-8677-d4cd75068a67" path="/var/lib/kubelet/pods/bf3f1939-e8e5-42ec-8677-d4cd75068a67/volumes" Jul 09 23:00:10.715261 ip-10-0-136-251 kubenswrapper[2579]: I0709 23:00:10.715220 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-xdfrr"] Jul 09 23:00:30.601157 ip-10-0-136-251 kubenswrapper[2579]: I0709 23:00:30.601117 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-862c5d/rate-test-model-862c5d-kserve-84c477b7bc-5585s"] Jul 09 23:00:30.601651 ip-10-0-136-251 kubenswrapper[2579]: I0709 23:00:30.601633 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="834b2c8b-6388-43b8-ba59-a1ab73ddfb1f" containerName="cleanup" Jul 09 23:00:30.601700 ip-10-0-136-251 kubenswrapper[2579]: I0709 23:00:30.601653 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="834b2c8b-6388-43b8-ba59-a1ab73ddfb1f" containerName="cleanup" Jul 09 23:00:30.601700 ip-10-0-136-251 kubenswrapper[2579]: I0709 23:00:30.601664 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f5ddc88c-bc30-4c14-aa1f-dd54f4a8e633" containerName="cleanup" Jul 09 23:00:30.601700 ip-10-0-136-251 kubenswrapper[2579]: I0709 23:00:30.601670 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="f5ddc88c-bc30-4c14-aa1f-dd54f4a8e633" containerName="cleanup" Jul 09 23:00:30.601700 ip-10-0-136-251 kubenswrapper[2579]: I0709 23:00:30.601679 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f3972d63-c4f4-4011-bea3-681bc562086d" containerName="cleanup" Jul 09 23:00:30.601700 ip-10-0-136-251 kubenswrapper[2579]: I0709 23:00:30.601687 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="f3972d63-c4f4-4011-bea3-681bc562086d" containerName="cleanup" Jul 09 23:00:30.601849 ip-10-0-136-251 kubenswrapper[2579]: I0709 23:00:30.601707 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bf3f1939-e8e5-42ec-8677-d4cd75068a67" containerName="main" Jul 09 23:00:30.601849 ip-10-0-136-251 kubenswrapper[2579]: I0709 23:00:30.601712 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="bf3f1939-e8e5-42ec-8677-d4cd75068a67" containerName="main" Jul 09 23:00:30.601849 ip-10-0-136-251 kubenswrapper[2579]: I0709 23:00:30.601771 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="f5ddc88c-bc30-4c14-aa1f-dd54f4a8e633" containerName="cleanup" Jul 09 23:00:30.601849 ip-10-0-136-251 kubenswrapper[2579]: I0709 23:00:30.601781 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="bf3f1939-e8e5-42ec-8677-d4cd75068a67" containerName="main" Jul 09 23:00:30.601849 ip-10-0-136-251 kubenswrapper[2579]: I0709 23:00:30.601789 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="f3972d63-c4f4-4011-bea3-681bc562086d" containerName="cleanup" Jul 09 23:00:30.601849 ip-10-0-136-251 kubenswrapper[2579]: I0709 23:00:30.601795 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="834b2c8b-6388-43b8-ba59-a1ab73ddfb1f" containerName="cleanup" Jul 09 23:00:30.604992 ip-10-0-136-251 kubenswrapper[2579]: I0709 23:00:30.604976 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-862c5d/rate-test-model-862c5d-kserve-84c477b7bc-5585s" Jul 09 23:00:30.607514 ip-10-0-136-251 kubenswrapper[2579]: I0709 23:00:30.607488 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-862c5d\"/\"openshift-service-ca.crt\"" Jul 09 23:00:30.608616 ip-10-0-136-251 kubenswrapper[2579]: I0709 23:00:30.608591 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-862c5d\"/\"default-dockercfg-m7996\"" Jul 09 23:00:30.608751 ip-10-0-136-251 kubenswrapper[2579]: I0709 23:00:30.608644 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-862c5d\"/\"rate-test-model-862c5d-kserve-self-signed-certs\"" Jul 09 23:00:30.608751 ip-10-0-136-251 kubenswrapper[2579]: I0709 23:00:30.608596 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-862c5d\"/\"kube-root-ca.crt\"" Jul 09 23:00:30.614718 ip-10-0-136-251 kubenswrapper[2579]: I0709 23:00:30.614687 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-862c5d/rate-test-model-862c5d-kserve-84c477b7bc-5585s"] Jul 09 23:00:30.756524 ip-10-0-136-251 kubenswrapper[2579]: I0709 23:00:30.756481 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5e4deff1-e164-4ef2-b202-f489af121ba2-home\") pod \"rate-test-model-862c5d-kserve-84c477b7bc-5585s\" (UID: \"5e4deff1-e164-4ef2-b202-f489af121ba2\") " pod="ai-tenant-e2e-shared-b-862c5d/rate-test-model-862c5d-kserve-84c477b7bc-5585s" Jul 09 23:00:30.756732 ip-10-0-136-251 kubenswrapper[2579]: I0709 23:00:30.756547 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5e4deff1-e164-4ef2-b202-f489af121ba2-dshm\") pod \"rate-test-model-862c5d-kserve-84c477b7bc-5585s\" (UID: \"5e4deff1-e164-4ef2-b202-f489af121ba2\") " pod="ai-tenant-e2e-shared-b-862c5d/rate-test-model-862c5d-kserve-84c477b7bc-5585s" Jul 09 23:00:30.756732 ip-10-0-136-251 kubenswrapper[2579]: I0709 23:00:30.756591 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5e4deff1-e164-4ef2-b202-f489af121ba2-tmp-dir\") pod \"rate-test-model-862c5d-kserve-84c477b7bc-5585s\" (UID: \"5e4deff1-e164-4ef2-b202-f489af121ba2\") " pod="ai-tenant-e2e-shared-b-862c5d/rate-test-model-862c5d-kserve-84c477b7bc-5585s" Jul 09 23:00:30.756732 ip-10-0-136-251 kubenswrapper[2579]: I0709 23:00:30.756616 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5e4deff1-e164-4ef2-b202-f489af121ba2-tls-certs\") pod \"rate-test-model-862c5d-kserve-84c477b7bc-5585s\" (UID: \"5e4deff1-e164-4ef2-b202-f489af121ba2\") " pod="ai-tenant-e2e-shared-b-862c5d/rate-test-model-862c5d-kserve-84c477b7bc-5585s" Jul 09 23:00:30.756732 ip-10-0-136-251 kubenswrapper[2579]: I0709 23:00:30.756677 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5e4deff1-e164-4ef2-b202-f489af121ba2-model-cache\") pod \"rate-test-model-862c5d-kserve-84c477b7bc-5585s\" (UID: \"5e4deff1-e164-4ef2-b202-f489af121ba2\") " pod="ai-tenant-e2e-shared-b-862c5d/rate-test-model-862c5d-kserve-84c477b7bc-5585s" Jul 09 23:00:30.756732 ip-10-0-136-251 kubenswrapper[2579]: I0709 23:00:30.756708 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9sjlh\" (UniqueName: \"kubernetes.io/projected/5e4deff1-e164-4ef2-b202-f489af121ba2-kube-api-access-9sjlh\") pod \"rate-test-model-862c5d-kserve-84c477b7bc-5585s\" (UID: \"5e4deff1-e164-4ef2-b202-f489af121ba2\") " pod="ai-tenant-e2e-shared-b-862c5d/rate-test-model-862c5d-kserve-84c477b7bc-5585s" Jul 09 23:00:30.857358 ip-10-0-136-251 kubenswrapper[2579]: I0709 23:00:30.857265 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5e4deff1-e164-4ef2-b202-f489af121ba2-model-cache\") pod \"rate-test-model-862c5d-kserve-84c477b7bc-5585s\" (UID: \"5e4deff1-e164-4ef2-b202-f489af121ba2\") " pod="ai-tenant-e2e-shared-b-862c5d/rate-test-model-862c5d-kserve-84c477b7bc-5585s" Jul 09 23:00:30.857358 ip-10-0-136-251 kubenswrapper[2579]: I0709 23:00:30.857319 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9sjlh\" (UniqueName: \"kubernetes.io/projected/5e4deff1-e164-4ef2-b202-f489af121ba2-kube-api-access-9sjlh\") pod \"rate-test-model-862c5d-kserve-84c477b7bc-5585s\" (UID: \"5e4deff1-e164-4ef2-b202-f489af121ba2\") " pod="ai-tenant-e2e-shared-b-862c5d/rate-test-model-862c5d-kserve-84c477b7bc-5585s" Jul 09 23:00:30.857600 ip-10-0-136-251 kubenswrapper[2579]: I0709 23:00:30.857408 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5e4deff1-e164-4ef2-b202-f489af121ba2-home\") pod \"rate-test-model-862c5d-kserve-84c477b7bc-5585s\" (UID: \"5e4deff1-e164-4ef2-b202-f489af121ba2\") " pod="ai-tenant-e2e-shared-b-862c5d/rate-test-model-862c5d-kserve-84c477b7bc-5585s" Jul 09 23:00:30.857600 ip-10-0-136-251 kubenswrapper[2579]: I0709 23:00:30.857470 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5e4deff1-e164-4ef2-b202-f489af121ba2-dshm\") pod \"rate-test-model-862c5d-kserve-84c477b7bc-5585s\" (UID: \"5e4deff1-e164-4ef2-b202-f489af121ba2\") " pod="ai-tenant-e2e-shared-b-862c5d/rate-test-model-862c5d-kserve-84c477b7bc-5585s" Jul 09 23:00:30.857600 ip-10-0-136-251 kubenswrapper[2579]: I0709 23:00:30.857497 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5e4deff1-e164-4ef2-b202-f489af121ba2-tmp-dir\") pod \"rate-test-model-862c5d-kserve-84c477b7bc-5585s\" (UID: \"5e4deff1-e164-4ef2-b202-f489af121ba2\") " pod="ai-tenant-e2e-shared-b-862c5d/rate-test-model-862c5d-kserve-84c477b7bc-5585s" Jul 09 23:00:30.857600 ip-10-0-136-251 kubenswrapper[2579]: I0709 23:00:30.857522 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5e4deff1-e164-4ef2-b202-f489af121ba2-tls-certs\") pod \"rate-test-model-862c5d-kserve-84c477b7bc-5585s\" (UID: \"5e4deff1-e164-4ef2-b202-f489af121ba2\") " pod="ai-tenant-e2e-shared-b-862c5d/rate-test-model-862c5d-kserve-84c477b7bc-5585s" Jul 09 23:00:30.857790 ip-10-0-136-251 kubenswrapper[2579]: I0709 23:00:30.857760 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5e4deff1-e164-4ef2-b202-f489af121ba2-model-cache\") pod \"rate-test-model-862c5d-kserve-84c477b7bc-5585s\" (UID: \"5e4deff1-e164-4ef2-b202-f489af121ba2\") " pod="ai-tenant-e2e-shared-b-862c5d/rate-test-model-862c5d-kserve-84c477b7bc-5585s" Jul 09 23:00:30.857830 ip-10-0-136-251 kubenswrapper[2579]: I0709 23:00:30.857800 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5e4deff1-e164-4ef2-b202-f489af121ba2-home\") pod \"rate-test-model-862c5d-kserve-84c477b7bc-5585s\" (UID: \"5e4deff1-e164-4ef2-b202-f489af121ba2\") " pod="ai-tenant-e2e-shared-b-862c5d/rate-test-model-862c5d-kserve-84c477b7bc-5585s" Jul 09 23:00:30.857868 ip-10-0-136-251 kubenswrapper[2579]: I0709 23:00:30.857840 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5e4deff1-e164-4ef2-b202-f489af121ba2-tmp-dir\") pod \"rate-test-model-862c5d-kserve-84c477b7bc-5585s\" (UID: \"5e4deff1-e164-4ef2-b202-f489af121ba2\") " pod="ai-tenant-e2e-shared-b-862c5d/rate-test-model-862c5d-kserve-84c477b7bc-5585s" Jul 09 23:00:30.859776 ip-10-0-136-251 kubenswrapper[2579]: I0709 23:00:30.859750 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5e4deff1-e164-4ef2-b202-f489af121ba2-dshm\") pod \"rate-test-model-862c5d-kserve-84c477b7bc-5585s\" (UID: \"5e4deff1-e164-4ef2-b202-f489af121ba2\") " pod="ai-tenant-e2e-shared-b-862c5d/rate-test-model-862c5d-kserve-84c477b7bc-5585s" Jul 09 23:00:30.859964 ip-10-0-136-251 kubenswrapper[2579]: I0709 23:00:30.859910 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5e4deff1-e164-4ef2-b202-f489af121ba2-tls-certs\") pod \"rate-test-model-862c5d-kserve-84c477b7bc-5585s\" (UID: \"5e4deff1-e164-4ef2-b202-f489af121ba2\") " pod="ai-tenant-e2e-shared-b-862c5d/rate-test-model-862c5d-kserve-84c477b7bc-5585s" Jul 09 23:00:30.866092 ip-10-0-136-251 kubenswrapper[2579]: I0709 23:00:30.866065 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9sjlh\" (UniqueName: \"kubernetes.io/projected/5e4deff1-e164-4ef2-b202-f489af121ba2-kube-api-access-9sjlh\") pod \"rate-test-model-862c5d-kserve-84c477b7bc-5585s\" (UID: \"5e4deff1-e164-4ef2-b202-f489af121ba2\") " pod="ai-tenant-e2e-shared-b-862c5d/rate-test-model-862c5d-kserve-84c477b7bc-5585s" Jul 09 23:00:30.915714 ip-10-0-136-251 kubenswrapper[2579]: I0709 23:00:30.915678 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-862c5d/rate-test-model-862c5d-kserve-84c477b7bc-5585s" Jul 09 23:00:31.060236 ip-10-0-136-251 kubenswrapper[2579]: I0709 23:00:31.060201 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-862c5d/rate-test-model-862c5d-kserve-84c477b7bc-5585s"] Jul 09 23:00:31.062247 ip-10-0-136-251 kubenswrapper[2579]: W0709 23:00:31.062221 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5e4deff1_e164_4ef2_b202_f489af121ba2.slice/crio-e867fff7426c3f1a695ea86e8a2a4519c516d77caa021f3d76f431fa0a6c3dbe WatchSource:0}: Error finding container e867fff7426c3f1a695ea86e8a2a4519c516d77caa021f3d76f431fa0a6c3dbe: Status 404 returned error can't find the container with id e867fff7426c3f1a695ea86e8a2a4519c516d77caa021f3d76f431fa0a6c3dbe Jul 09 23:00:31.319302 ip-10-0-136-251 kubenswrapper[2579]: I0709 23:00:31.319260 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-862c5d/rate-test-model-862c5d-kserve-84c477b7bc-5585s" event={"ID":"5e4deff1-e164-4ef2-b202-f489af121ba2","Type":"ContainerStarted","Data":"86260817dc05bfb91759446036d46157c1e60ee43b4d0cefa87c33e808309456"} Jul 09 23:00:31.319302 ip-10-0-136-251 kubenswrapper[2579]: I0709 23:00:31.319306 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-862c5d/rate-test-model-862c5d-kserve-84c477b7bc-5585s" event={"ID":"5e4deff1-e164-4ef2-b202-f489af121ba2","Type":"ContainerStarted","Data":"e867fff7426c3f1a695ea86e8a2a4519c516d77caa021f3d76f431fa0a6c3dbe"} Jul 09 23:00:31.338584 ip-10-0-136-251 kubenswrapper[2579]: I0709 23:00:31.338518 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-862c5d/rate-test-model-862c5d-kserve-84c477b7bc-5585s" podStartSLOduration=1.3385011549999999 podStartE2EDuration="1.338501155s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 23:00:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 23:00:31.335711126 +0000 UTC m=+3023.247163846" watchObservedRunningTime="2026-07-09 23:00:31.338501155 +0000 UTC m=+3023.249953855" Jul 09 23:00:35.625648 ip-10-0-136-251 kubenswrapper[2579]: I0709 23:00:35.625610 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-xdfrr"] Jul 09 23:00:40.916310 ip-10-0-136-251 kubenswrapper[2579]: I0709 23:00:40.916269 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-862c5d/rate-test-model-862c5d-kserve-84c477b7bc-5585s" Jul 09 23:00:40.916748 ip-10-0-136-251 kubenswrapper[2579]: I0709 23:00:40.916334 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-862c5d/rate-test-model-862c5d-kserve-84c477b7bc-5585s" Jul 09 23:00:40.929618 ip-10-0-136-251 kubenswrapper[2579]: I0709 23:00:40.929584 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-862c5d/rate-test-model-862c5d-kserve-84c477b7bc-5585s" Jul 09 23:00:41.385795 ip-10-0-136-251 kubenswrapper[2579]: I0709 23:00:41.385762 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-862c5d/rate-test-model-862c5d-kserve-84c477b7bc-5585s" Jul 09 23:00:50.111577 ip-10-0-136-251 kubenswrapper[2579]: I0709 23:00:50.111539 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-xdfrr"] Jul 09 23:01:05.814552 ip-10-0-136-251 kubenswrapper[2579]: I0709 23:01:05.814515 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-xdfrr"] Jul 09 23:01:12.124073 ip-10-0-136-251 kubenswrapper[2579]: I0709 23:01:12.124036 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-862c5d/rate-test-model-862c5d-kserve-84c477b7bc-5585s"] Jul 09 23:01:13.189287 ip-10-0-136-251 kubenswrapper[2579]: I0709 23:01:13.189252 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/test-curl-7235-139910755327152"] Jul 09 23:01:13.192853 ip-10-0-136-251 kubenswrapper[2579]: I0709 23:01:13.192831 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7235-139910755327152" Jul 09 23:01:13.195325 ip-10-0-136-251 kubenswrapper[2579]: I0709 23:01:13.195299 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-dockercfg-2sl77\"" Jul 09 23:01:13.200854 ip-10-0-136-251 kubenswrapper[2579]: I0709 23:01:13.200825 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/test-curl-7235-139910755327152"] Jul 09 23:01:13.368541 ip-10-0-136-251 kubenswrapper[2579]: I0709 23:01:13.368503 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wwvzl\" (UniqueName: \"kubernetes.io/projected/26a6971a-ac65-4f58-a292-ba232ff7ec70-kube-api-access-wwvzl\") pod \"test-curl-7235-139910755327152\" (UID: \"26a6971a-ac65-4f58-a292-ba232ff7ec70\") " pod="openshift-ingress/test-curl-7235-139910755327152" Jul 09 23:01:13.469392 ip-10-0-136-251 kubenswrapper[2579]: I0709 23:01:13.469290 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wwvzl\" (UniqueName: \"kubernetes.io/projected/26a6971a-ac65-4f58-a292-ba232ff7ec70-kube-api-access-wwvzl\") pod \"test-curl-7235-139910755327152\" (UID: \"26a6971a-ac65-4f58-a292-ba232ff7ec70\") " pod="openshift-ingress/test-curl-7235-139910755327152" Jul 09 23:01:13.481769 ip-10-0-136-251 kubenswrapper[2579]: I0709 23:01:13.481734 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wwvzl\" (UniqueName: \"kubernetes.io/projected/26a6971a-ac65-4f58-a292-ba232ff7ec70-kube-api-access-wwvzl\") pod \"test-curl-7235-139910755327152\" (UID: \"26a6971a-ac65-4f58-a292-ba232ff7ec70\") " pod="openshift-ingress/test-curl-7235-139910755327152" Jul 09 23:01:13.503787 ip-10-0-136-251 kubenswrapper[2579]: I0709 23:01:13.503747 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7235-139910755327152" Jul 09 23:01:13.642181 ip-10-0-136-251 kubenswrapper[2579]: I0709 23:01:13.642155 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/test-curl-7235-139910755327152"] Jul 09 23:01:13.643998 ip-10-0-136-251 kubenswrapper[2579]: W0709 23:01:13.643969 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod26a6971a_ac65_4f58_a292_ba232ff7ec70.slice/crio-835fef996c1e9ec0473f9f10ece39d7dc21bbe6ed004731cf599b22262d87bc1 WatchSource:0}: Error finding container 835fef996c1e9ec0473f9f10ece39d7dc21bbe6ed004731cf599b22262d87bc1: Status 404 returned error can't find the container with id 835fef996c1e9ec0473f9f10ece39d7dc21bbe6ed004731cf599b22262d87bc1 Jul 09 23:01:14.560128 ip-10-0-136-251 kubenswrapper[2579]: I0709 23:01:14.560097 2579 generic.go:356] "Generic (PLEG): container finished" podID="26a6971a-ac65-4f58-a292-ba232ff7ec70" containerID="695c1dbf1ee0fe04de82e4a1f7e3b7389a3dd3790200a9f30fc4de38d96cd030" exitCode=0 Jul 09 23:01:14.564459 ip-10-0-136-251 kubenswrapper[2579]: I0709 23:01:14.564431 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/test-curl-7235-139910755327152" event={"ID":"26a6971a-ac65-4f58-a292-ba232ff7ec70","Type":"ContainerDied","Data":"695c1dbf1ee0fe04de82e4a1f7e3b7389a3dd3790200a9f30fc4de38d96cd030"} Jul 09 23:01:14.564607 ip-10-0-136-251 kubenswrapper[2579]: I0709 23:01:14.564467 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/test-curl-7235-139910755327152" event={"ID":"26a6971a-ac65-4f58-a292-ba232ff7ec70","Type":"ContainerStarted","Data":"835fef996c1e9ec0473f9f10ece39d7dc21bbe6ed004731cf599b22262d87bc1"} Jul 09 23:01:15.724337 ip-10-0-136-251 kubenswrapper[2579]: I0709 23:01:15.724312 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7235-139910755327152" Jul 09 23:01:15.759711 ip-10-0-136-251 kubenswrapper[2579]: I0709 23:01:15.759677 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_test-curl-7235-139910755327152_26a6971a-ac65-4f58-a292-ba232ff7ec70/test-curl-7235-139910755327152/0.log" Jul 09 23:01:15.786337 ip-10-0-136-251 kubenswrapper[2579]: I0709 23:01:15.786297 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/test-curl-7235-139910755327152"] Jul 09 23:01:15.789693 ip-10-0-136-251 kubenswrapper[2579]: I0709 23:01:15.789664 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/test-curl-7235-139910755327152"] Jul 09 23:01:15.893208 ip-10-0-136-251 kubenswrapper[2579]: I0709 23:01:15.893108 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wwvzl\" (UniqueName: \"kubernetes.io/projected/26a6971a-ac65-4f58-a292-ba232ff7ec70-kube-api-access-wwvzl\") pod \"26a6971a-ac65-4f58-a292-ba232ff7ec70\" (UID: \"26a6971a-ac65-4f58-a292-ba232ff7ec70\") " Jul 09 23:01:15.895182 ip-10-0-136-251 kubenswrapper[2579]: I0709 23:01:15.895154 2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/26a6971a-ac65-4f58-a292-ba232ff7ec70-kube-api-access-wwvzl" (OuterVolumeSpecName: "kube-api-access-wwvzl") pod "26a6971a-ac65-4f58-a292-ba232ff7ec70" (UID: "26a6971a-ac65-4f58-a292-ba232ff7ec70"). InnerVolumeSpecName "kube-api-access-wwvzl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 23:01:15.922120 ip-10-0-136-251 kubenswrapper[2579]: I0709 23:01:15.922078 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/test-curl-7235-139910755324208"] Jul 09 23:01:15.922563 ip-10-0-136-251 kubenswrapper[2579]: I0709 23:01:15.922545 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="26a6971a-ac65-4f58-a292-ba232ff7ec70" containerName="test-curl-7235-139910755327152" Jul 09 23:01:15.922674 ip-10-0-136-251 kubenswrapper[2579]: I0709 23:01:15.922566 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="26a6971a-ac65-4f58-a292-ba232ff7ec70" containerName="test-curl-7235-139910755327152" Jul 09 23:01:15.922731 ip-10-0-136-251 kubenswrapper[2579]: I0709 23:01:15.922672 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="26a6971a-ac65-4f58-a292-ba232ff7ec70" containerName="test-curl-7235-139910755327152" Jul 09 23:01:15.925837 ip-10-0-136-251 kubenswrapper[2579]: I0709 23:01:15.925810 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7235-139910755324208" Jul 09 23:01:15.931671 ip-10-0-136-251 kubenswrapper[2579]: I0709 23:01:15.931640 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/test-curl-7235-139910755324208"] Jul 09 23:01:15.994121 ip-10-0-136-251 kubenswrapper[2579]: I0709 23:01:15.994079 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wwvzl\" (UniqueName: \"kubernetes.io/projected/26a6971a-ac65-4f58-a292-ba232ff7ec70-kube-api-access-wwvzl\") on node \"ip-10-0-136-251.ec2.internal\" DevicePath \"\"" Jul 09 23:01:16.094730 ip-10-0-136-251 kubenswrapper[2579]: I0709 23:01:16.094687 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xwlqr\" (UniqueName: \"kubernetes.io/projected/e1af1e7e-fa2f-445d-b3c9-b651ec3c54ca-kube-api-access-xwlqr\") pod \"test-curl-7235-139910755324208\" (UID: \"e1af1e7e-fa2f-445d-b3c9-b651ec3c54ca\") " pod="openshift-ingress/test-curl-7235-139910755324208" Jul 09 23:01:16.196629 ip-10-0-136-251 kubenswrapper[2579]: I0709 23:01:16.196522 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xwlqr\" (UniqueName: \"kubernetes.io/projected/e1af1e7e-fa2f-445d-b3c9-b651ec3c54ca-kube-api-access-xwlqr\") pod \"test-curl-7235-139910755324208\" (UID: \"e1af1e7e-fa2f-445d-b3c9-b651ec3c54ca\") " pod="openshift-ingress/test-curl-7235-139910755324208" Jul 09 23:01:16.206418 ip-10-0-136-251 kubenswrapper[2579]: I0709 23:01:16.206390 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xwlqr\" (UniqueName: \"kubernetes.io/projected/e1af1e7e-fa2f-445d-b3c9-b651ec3c54ca-kube-api-access-xwlqr\") pod \"test-curl-7235-139910755324208\" (UID: \"e1af1e7e-fa2f-445d-b3c9-b651ec3c54ca\") " pod="openshift-ingress/test-curl-7235-139910755324208" Jul 09 23:01:16.237209 ip-10-0-136-251 kubenswrapper[2579]: I0709 23:01:16.237170 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7235-139910755324208" Jul 09 23:01:16.378132 ip-10-0-136-251 kubenswrapper[2579]: I0709 23:01:16.378106 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/test-curl-7235-139910755324208"] Jul 09 23:01:16.379813 ip-10-0-136-251 kubenswrapper[2579]: W0709 23:01:16.379789 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode1af1e7e_fa2f_445d_b3c9_b651ec3c54ca.slice/crio-56bf45bbf29ef2089db27c64beb5b8cd2d1a6da0dc6669a68bdfacd853abba64 WatchSource:0}: Error finding container 56bf45bbf29ef2089db27c64beb5b8cd2d1a6da0dc6669a68bdfacd853abba64: Status 404 returned error can't find the container with id 56bf45bbf29ef2089db27c64beb5b8cd2d1a6da0dc6669a68bdfacd853abba64 Jul 09 23:01:16.564517 ip-10-0-136-251 kubenswrapper[2579]: I0709 23:01:16.564482 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="26a6971a-ac65-4f58-a292-ba232ff7ec70" path="/var/lib/kubelet/pods/26a6971a-ac65-4f58-a292-ba232ff7ec70/volumes" Jul 09 23:01:16.578760 ip-10-0-136-251 kubenswrapper[2579]: I0709 23:01:16.578719 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/test-curl-7235-139910755324208" event={"ID":"e1af1e7e-fa2f-445d-b3c9-b651ec3c54ca","Type":"ContainerStarted","Data":"56bf45bbf29ef2089db27c64beb5b8cd2d1a6da0dc6669a68bdfacd853abba64"} Jul 09 23:01:16.580092 ip-10-0-136-251 kubenswrapper[2579]: I0709 23:01:16.580065 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7235-139910755327152" Jul 09 23:01:16.580224 ip-10-0-136-251 kubenswrapper[2579]: I0709 23:01:16.580095 2579 scope.go:117] "RemoveContainer" containerID="695c1dbf1ee0fe04de82e4a1f7e3b7389a3dd3790200a9f30fc4de38d96cd030" Jul 09 23:01:17.587730 ip-10-0-136-251 kubenswrapper[2579]: I0709 23:01:17.587695 2579 generic.go:356] "Generic (PLEG): container finished" podID="e1af1e7e-fa2f-445d-b3c9-b651ec3c54ca" containerID="ab42139189f02b9669a2876020716b4227cb7b3b7f3a9365d747d0be08126dfd" exitCode=0 Jul 09 23:01:17.588275 ip-10-0-136-251 kubenswrapper[2579]: I0709 23:01:17.587792 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/test-curl-7235-139910755324208" event={"ID":"e1af1e7e-fa2f-445d-b3c9-b651ec3c54ca","Type":"ContainerDied","Data":"ab42139189f02b9669a2876020716b4227cb7b3b7f3a9365d747d0be08126dfd"} Jul 09 23:01:18.747432 ip-10-0-136-251 kubenswrapper[2579]: I0709 23:01:18.747399 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7235-139910755324208" Jul 09 23:01:18.768683 ip-10-0-136-251 kubenswrapper[2579]: I0709 23:01:18.768651 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_test-curl-7235-139910755324208_e1af1e7e-fa2f-445d-b3c9-b651ec3c54ca/test-curl-7235-139910755324208/0.log" Jul 09 23:01:18.792895 ip-10-0-136-251 kubenswrapper[2579]: I0709 23:01:18.792863 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/test-curl-7235-139910755324208"] Jul 09 23:01:18.798041 ip-10-0-136-251 kubenswrapper[2579]: I0709 23:01:18.798014 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/test-curl-7235-139910755324208"] Jul 09 23:01:18.919848 ip-10-0-136-251 kubenswrapper[2579]: I0709 23:01:18.919751 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xwlqr\" (UniqueName: \"kubernetes.io/projected/e1af1e7e-fa2f-445d-b3c9-b651ec3c54ca-kube-api-access-xwlqr\") pod \"e1af1e7e-fa2f-445d-b3c9-b651ec3c54ca\" (UID: \"e1af1e7e-fa2f-445d-b3c9-b651ec3c54ca\") " Jul 09 23:01:18.921886 ip-10-0-136-251 kubenswrapper[2579]: I0709 23:01:18.921860 2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e1af1e7e-fa2f-445d-b3c9-b651ec3c54ca-kube-api-access-xwlqr" (OuterVolumeSpecName: "kube-api-access-xwlqr") pod "e1af1e7e-fa2f-445d-b3c9-b651ec3c54ca" (UID: "e1af1e7e-fa2f-445d-b3c9-b651ec3c54ca"). InnerVolumeSpecName "kube-api-access-xwlqr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 23:01:18.982452 ip-10-0-136-251 kubenswrapper[2579]: I0709 23:01:18.982415 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/test-curl-7235-139910759980608"] Jul 09 23:01:18.982831 ip-10-0-136-251 kubenswrapper[2579]: I0709 23:01:18.982817 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e1af1e7e-fa2f-445d-b3c9-b651ec3c54ca" containerName="test-curl-7235-139910755324208" Jul 09 23:01:18.982875 ip-10-0-136-251 kubenswrapper[2579]: I0709 23:01:18.982832 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="e1af1e7e-fa2f-445d-b3c9-b651ec3c54ca" containerName="test-curl-7235-139910755324208" Jul 09 23:01:18.982910 ip-10-0-136-251 kubenswrapper[2579]: I0709 23:01:18.982902 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="e1af1e7e-fa2f-445d-b3c9-b651ec3c54ca" containerName="test-curl-7235-139910755324208" Jul 09 23:01:18.986048 ip-10-0-136-251 kubenswrapper[2579]: I0709 23:01:18.985913 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7235-139910759980608" Jul 09 23:01:18.991886 ip-10-0-136-251 kubenswrapper[2579]: I0709 23:01:18.991861 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/test-curl-7235-139910759980608"] Jul 09 23:01:19.021254 ip-10-0-136-251 kubenswrapper[2579]: I0709 23:01:19.021218 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xwlqr\" (UniqueName: \"kubernetes.io/projected/e1af1e7e-fa2f-445d-b3c9-b651ec3c54ca-kube-api-access-xwlqr\") on node \"ip-10-0-136-251.ec2.internal\" DevicePath \"\"" Jul 09 23:01:19.121682 ip-10-0-136-251 kubenswrapper[2579]: I0709 23:01:19.121644 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d6km4\" (UniqueName: \"kubernetes.io/projected/de3c5a03-9cf9-43cf-87c5-9fa478346d5e-kube-api-access-d6km4\") pod \"test-curl-7235-139910759980608\" (UID: \"de3c5a03-9cf9-43cf-87c5-9fa478346d5e\") " pod="openshift-ingress/test-curl-7235-139910759980608" Jul 09 23:01:19.223342 ip-10-0-136-251 kubenswrapper[2579]: I0709 23:01:19.223229 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d6km4\" (UniqueName: \"kubernetes.io/projected/de3c5a03-9cf9-43cf-87c5-9fa478346d5e-kube-api-access-d6km4\") pod \"test-curl-7235-139910759980608\" (UID: \"de3c5a03-9cf9-43cf-87c5-9fa478346d5e\") " pod="openshift-ingress/test-curl-7235-139910759980608" Jul 09 23:01:19.232428 ip-10-0-136-251 kubenswrapper[2579]: I0709 23:01:19.232394 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d6km4\" (UniqueName: \"kubernetes.io/projected/de3c5a03-9cf9-43cf-87c5-9fa478346d5e-kube-api-access-d6km4\") pod \"test-curl-7235-139910759980608\" (UID: \"de3c5a03-9cf9-43cf-87c5-9fa478346d5e\") " pod="openshift-ingress/test-curl-7235-139910759980608" Jul 09 23:01:19.297111 ip-10-0-136-251 kubenswrapper[2579]: I0709 23:01:19.297069 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7235-139910759980608" Jul 09 23:01:19.440091 ip-10-0-136-251 kubenswrapper[2579]: I0709 23:01:19.440063 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/test-curl-7235-139910759980608"] Jul 09 23:01:19.441458 ip-10-0-136-251 kubenswrapper[2579]: W0709 23:01:19.441435 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podde3c5a03_9cf9_43cf_87c5_9fa478346d5e.slice/crio-f39fbded6e886aad2323f4abb9a42c86ad6a19540966102d19f4bdaa7db80c57 WatchSource:0}: Error finding container f39fbded6e886aad2323f4abb9a42c86ad6a19540966102d19f4bdaa7db80c57: Status 404 returned error can't find the container with id f39fbded6e886aad2323f4abb9a42c86ad6a19540966102d19f4bdaa7db80c57 Jul 09 23:01:19.600832 ip-10-0-136-251 kubenswrapper[2579]: I0709 23:01:19.600795 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/test-curl-7235-139910759980608" event={"ID":"de3c5a03-9cf9-43cf-87c5-9fa478346d5e","Type":"ContainerStarted","Data":"f39fbded6e886aad2323f4abb9a42c86ad6a19540966102d19f4bdaa7db80c57"} Jul 09 23:01:19.602008 ip-10-0-136-251 kubenswrapper[2579]: I0709 23:01:19.601985 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7235-139910755324208" Jul 09 23:01:19.602128 ip-10-0-136-251 kubenswrapper[2579]: I0709 23:01:19.602021 2579 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="56bf45bbf29ef2089db27c64beb5b8cd2d1a6da0dc6669a68bdfacd853abba64" Jul 09 23:01:19.612485 ip-10-0-136-251 kubenswrapper[2579]: E0709 23:01:19.612449 2579 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7235-139910755324208\" is forbidden: User \"system:node:ip-10-0-136-251.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-ingress\": no relationship found between node 'ip-10-0-136-251.ec2.internal' and this object" podUID="e1af1e7e-fa2f-445d-b3c9-b651ec3c54ca" pod="openshift-ingress/test-curl-7235-139910755324208" Jul 09 23:01:20.564961 ip-10-0-136-251 kubenswrapper[2579]: I0709 23:01:20.564911 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e1af1e7e-fa2f-445d-b3c9-b651ec3c54ca" path="/var/lib/kubelet/pods/e1af1e7e-fa2f-445d-b3c9-b651ec3c54ca/volumes" Jul 09 23:01:20.609147 ip-10-0-136-251 kubenswrapper[2579]: I0709 23:01:20.609113 2579 generic.go:356] "Generic (PLEG): container finished" podID="de3c5a03-9cf9-43cf-87c5-9fa478346d5e" containerID="e96394d4bc318f253e1c234f166cdba5aa1d977220173a06459dd3642a4b5dec" exitCode=0 Jul 09 23:01:20.609333 ip-10-0-136-251 kubenswrapper[2579]: I0709 23:01:20.609197 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/test-curl-7235-139910759980608" event={"ID":"de3c5a03-9cf9-43cf-87c5-9fa478346d5e","Type":"ContainerDied","Data":"e96394d4bc318f253e1c234f166cdba5aa1d977220173a06459dd3642a4b5dec"} Jul 09 23:01:21.764017 ip-10-0-136-251 kubenswrapper[2579]: I0709 23:01:21.763990 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7235-139910759980608" Jul 09 23:01:21.784315 ip-10-0-136-251 kubenswrapper[2579]: I0709 23:01:21.784288 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_test-curl-7235-139910759980608_de3c5a03-9cf9-43cf-87c5-9fa478346d5e/test-curl-7235-139910759980608/0.log" Jul 09 23:01:21.810232 ip-10-0-136-251 kubenswrapper[2579]: I0709 23:01:21.810192 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/test-curl-7235-139910759980608"] Jul 09 23:01:21.814551 ip-10-0-136-251 kubenswrapper[2579]: I0709 23:01:21.814522 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/test-curl-7235-139910759980608"] Jul 09 23:01:21.845366 ip-10-0-136-251 kubenswrapper[2579]: I0709 23:01:21.845316 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-d6km4\" (UniqueName: \"kubernetes.io/projected/de3c5a03-9cf9-43cf-87c5-9fa478346d5e-kube-api-access-d6km4\") pod \"de3c5a03-9cf9-43cf-87c5-9fa478346d5e\" (UID: \"de3c5a03-9cf9-43cf-87c5-9fa478346d5e\") " Jul 09 23:01:21.847387 ip-10-0-136-251 kubenswrapper[2579]: I0709 23:01:21.847363 2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/de3c5a03-9cf9-43cf-87c5-9fa478346d5e-kube-api-access-d6km4" (OuterVolumeSpecName: "kube-api-access-d6km4") pod "de3c5a03-9cf9-43cf-87c5-9fa478346d5e" (UID: "de3c5a03-9cf9-43cf-87c5-9fa478346d5e"). InnerVolumeSpecName "kube-api-access-d6km4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 23:01:21.946386 ip-10-0-136-251 kubenswrapper[2579]: I0709 23:01:21.946265 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d6km4\" (UniqueName: \"kubernetes.io/projected/de3c5a03-9cf9-43cf-87c5-9fa478346d5e-kube-api-access-d6km4\") on node \"ip-10-0-136-251.ec2.internal\" DevicePath \"\"" Jul 09 23:01:22.565591 ip-10-0-136-251 kubenswrapper[2579]: I0709 23:01:22.565553 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="de3c5a03-9cf9-43cf-87c5-9fa478346d5e" path="/var/lib/kubelet/pods/de3c5a03-9cf9-43cf-87c5-9fa478346d5e/volumes" Jul 09 23:01:22.623003 ip-10-0-136-251 kubenswrapper[2579]: I0709 23:01:22.622971 2579 scope.go:117] "RemoveContainer" containerID="e96394d4bc318f253e1c234f166cdba5aa1d977220173a06459dd3642a4b5dec" Jul 09 23:01:22.623204 ip-10-0-136-251 kubenswrapper[2579]: I0709 23:01:22.622972 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7235-139910759980608" Jul 09 23:01:24.139717 ip-10-0-136-251 kubenswrapper[2579]: I0709 23:01:24.139684 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-xdfrr"] Jul 09 23:01:25.128606 ip-10-0-136-251 kubenswrapper[2579]: I0709 23:01:25.128564 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/test-curl-7235-139910759980464"] Jul 09 23:01:25.129022 ip-10-0-136-251 kubenswrapper[2579]: I0709 23:01:25.129005 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="de3c5a03-9cf9-43cf-87c5-9fa478346d5e" containerName="test-curl-7235-139910759980608" Jul 09 23:01:25.129101 ip-10-0-136-251 kubenswrapper[2579]: I0709 23:01:25.129026 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="de3c5a03-9cf9-43cf-87c5-9fa478346d5e" containerName="test-curl-7235-139910759980608" Jul 09 23:01:25.129151 ip-10-0-136-251 kubenswrapper[2579]: I0709 23:01:25.129141 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="de3c5a03-9cf9-43cf-87c5-9fa478346d5e" containerName="test-curl-7235-139910759980608" Jul 09 23:01:25.133506 ip-10-0-136-251 kubenswrapper[2579]: I0709 23:01:25.133487 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7235-139910759980464" Jul 09 23:01:25.136783 ip-10-0-136-251 kubenswrapper[2579]: I0709 23:01:25.136759 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-dockercfg-2sl77\"" Jul 09 23:01:25.142793 ip-10-0-136-251 kubenswrapper[2579]: I0709 23:01:25.142767 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/test-curl-7235-139910759980464"] Jul 09 23:01:25.170486 ip-10-0-136-251 kubenswrapper[2579]: I0709 23:01:25.170458 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7slnv\" (UniqueName: \"kubernetes.io/projected/e2ef48f1-5e75-4881-96f2-12ddf10864aa-kube-api-access-7slnv\") pod \"test-curl-7235-139910759980464\" (UID: \"e2ef48f1-5e75-4881-96f2-12ddf10864aa\") " pod="openshift-ingress/test-curl-7235-139910759980464" Jul 09 23:01:25.271879 ip-10-0-136-251 kubenswrapper[2579]: I0709 23:01:25.271834 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7slnv\" (UniqueName: \"kubernetes.io/projected/e2ef48f1-5e75-4881-96f2-12ddf10864aa-kube-api-access-7slnv\") pod \"test-curl-7235-139910759980464\" (UID: \"e2ef48f1-5e75-4881-96f2-12ddf10864aa\") " pod="openshift-ingress/test-curl-7235-139910759980464" Jul 09 23:01:25.284973 ip-10-0-136-251 kubenswrapper[2579]: I0709 23:01:25.284908 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7slnv\" (UniqueName: \"kubernetes.io/projected/e2ef48f1-5e75-4881-96f2-12ddf10864aa-kube-api-access-7slnv\") pod \"test-curl-7235-139910759980464\" (UID: \"e2ef48f1-5e75-4881-96f2-12ddf10864aa\") " pod="openshift-ingress/test-curl-7235-139910759980464" Jul 09 23:01:25.444083 ip-10-0-136-251 kubenswrapper[2579]: I0709 23:01:25.443989 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7235-139910759980464" Jul 09 23:01:25.587971 ip-10-0-136-251 kubenswrapper[2579]: I0709 23:01:25.587939 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/test-curl-7235-139910759980464"] Jul 09 23:01:25.589613 ip-10-0-136-251 kubenswrapper[2579]: W0709 23:01:25.589581 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode2ef48f1_5e75_4881_96f2_12ddf10864aa.slice/crio-7e5e47111778ec0cda4a4a56f8c3738b7beebe78d7a9c917f803018e5ad2225b WatchSource:0}: Error finding container 7e5e47111778ec0cda4a4a56f8c3738b7beebe78d7a9c917f803018e5ad2225b: Status 404 returned error can't find the container with id 7e5e47111778ec0cda4a4a56f8c3738b7beebe78d7a9c917f803018e5ad2225b Jul 09 23:01:25.646913 ip-10-0-136-251 kubenswrapper[2579]: I0709 23:01:25.646877 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/test-curl-7235-139910759980464" event={"ID":"e2ef48f1-5e75-4881-96f2-12ddf10864aa","Type":"ContainerStarted","Data":"7e5e47111778ec0cda4a4a56f8c3738b7beebe78d7a9c917f803018e5ad2225b"} Jul 09 23:01:26.653709 ip-10-0-136-251 kubenswrapper[2579]: I0709 23:01:26.653622 2579 generic.go:356] "Generic (PLEG): container finished" podID="e2ef48f1-5e75-4881-96f2-12ddf10864aa" containerID="fb4d7189d1168140e33dbe99936fb28aa13cfa00b8325acf00e7395c8141697d" exitCode=0 Jul 09 23:01:26.654111 ip-10-0-136-251 kubenswrapper[2579]: I0709 23:01:26.653710 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/test-curl-7235-139910759980464" event={"ID":"e2ef48f1-5e75-4881-96f2-12ddf10864aa","Type":"ContainerDied","Data":"fb4d7189d1168140e33dbe99936fb28aa13cfa00b8325acf00e7395c8141697d"} Jul 09 23:01:27.145370 ip-10-0-136-251 kubenswrapper[2579]: I0709 23:01:27.145327 2579 kuberuntime_container.go:872] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-862c5d/rate-test-model-862c5d-kserve-84c477b7bc-5585s" podUID="5e4deff1-e164-4ef2-b202-f489af121ba2" containerName="main" containerID="cri-o://86260817dc05bfb91759446036d46157c1e60ee43b4d0cefa87c33e808309456" gracePeriod=45 Jul 09 23:01:27.416043 ip-10-0-136-251 kubenswrapper[2579]: I0709 23:01:27.416017 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-862c5d/rate-test-model-862c5d-kserve-84c477b7bc-5585s" Jul 09 23:01:27.492555 ip-10-0-136-251 kubenswrapper[2579]: I0709 23:01:27.492522 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5e4deff1-e164-4ef2-b202-f489af121ba2-home\") pod \"5e4deff1-e164-4ef2-b202-f489af121ba2\" (UID: \"5e4deff1-e164-4ef2-b202-f489af121ba2\") " Jul 09 23:01:27.492738 ip-10-0-136-251 kubenswrapper[2579]: I0709 23:01:27.492577 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5e4deff1-e164-4ef2-b202-f489af121ba2-model-cache\") pod \"5e4deff1-e164-4ef2-b202-f489af121ba2\" (UID: \"5e4deff1-e164-4ef2-b202-f489af121ba2\") " Jul 09 23:01:27.492738 ip-10-0-136-251 kubenswrapper[2579]: I0709 23:01:27.492617 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5e4deff1-e164-4ef2-b202-f489af121ba2-tls-certs\") pod \"5e4deff1-e164-4ef2-b202-f489af121ba2\" (UID: \"5e4deff1-e164-4ef2-b202-f489af121ba2\") " Jul 09 23:01:27.492738 ip-10-0-136-251 kubenswrapper[2579]: I0709 23:01:27.492669 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5e4deff1-e164-4ef2-b202-f489af121ba2-tmp-dir\") pod \"5e4deff1-e164-4ef2-b202-f489af121ba2\" (UID: \"5e4deff1-e164-4ef2-b202-f489af121ba2\") " Jul 09 23:01:27.492738 ip-10-0-136-251 kubenswrapper[2579]: I0709 23:01:27.492713 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9sjlh\" (UniqueName: \"kubernetes.io/projected/5e4deff1-e164-4ef2-b202-f489af121ba2-kube-api-access-9sjlh\") pod \"5e4deff1-e164-4ef2-b202-f489af121ba2\" (UID: \"5e4deff1-e164-4ef2-b202-f489af121ba2\") " Jul 09 23:01:27.492957 ip-10-0-136-251 kubenswrapper[2579]: I0709 23:01:27.492750 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5e4deff1-e164-4ef2-b202-f489af121ba2-dshm\") pod \"5e4deff1-e164-4ef2-b202-f489af121ba2\" (UID: \"5e4deff1-e164-4ef2-b202-f489af121ba2\") " Jul 09 23:01:27.492957 ip-10-0-136-251 kubenswrapper[2579]: I0709 23:01:27.492840 2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e4deff1-e164-4ef2-b202-f489af121ba2-model-cache" (OuterVolumeSpecName: "model-cache") pod "5e4deff1-e164-4ef2-b202-f489af121ba2" (UID: "5e4deff1-e164-4ef2-b202-f489af121ba2"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 23:01:27.492957 ip-10-0-136-251 kubenswrapper[2579]: I0709 23:01:27.492853 2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e4deff1-e164-4ef2-b202-f489af121ba2-home" (OuterVolumeSpecName: "home") pod "5e4deff1-e164-4ef2-b202-f489af121ba2" (UID: "5e4deff1-e164-4ef2-b202-f489af121ba2"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 23:01:27.493125 ip-10-0-136-251 kubenswrapper[2579]: I0709 23:01:27.492945 2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e4deff1-e164-4ef2-b202-f489af121ba2-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "5e4deff1-e164-4ef2-b202-f489af121ba2" (UID: "5e4deff1-e164-4ef2-b202-f489af121ba2"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 23:01:27.493125 ip-10-0-136-251 kubenswrapper[2579]: I0709 23:01:27.493075 2579 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5e4deff1-e164-4ef2-b202-f489af121ba2-tmp-dir\") on node \"ip-10-0-136-251.ec2.internal\" DevicePath \"\"" Jul 09 23:01:27.493125 ip-10-0-136-251 kubenswrapper[2579]: I0709 23:01:27.493087 2579 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5e4deff1-e164-4ef2-b202-f489af121ba2-home\") on node \"ip-10-0-136-251.ec2.internal\" DevicePath \"\"" Jul 09 23:01:27.493125 ip-10-0-136-251 kubenswrapper[2579]: I0709 23:01:27.493102 2579 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5e4deff1-e164-4ef2-b202-f489af121ba2-model-cache\") on node \"ip-10-0-136-251.ec2.internal\" DevicePath \"\"" Jul 09 23:01:27.494998 ip-10-0-136-251 kubenswrapper[2579]: I0709 23:01:27.494968 2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e4deff1-e164-4ef2-b202-f489af121ba2-dshm" (OuterVolumeSpecName: "dshm") pod "5e4deff1-e164-4ef2-b202-f489af121ba2" (UID: "5e4deff1-e164-4ef2-b202-f489af121ba2"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 23:01:27.494998 ip-10-0-136-251 kubenswrapper[2579]: I0709 23:01:27.494981 2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5e4deff1-e164-4ef2-b202-f489af121ba2-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "5e4deff1-e164-4ef2-b202-f489af121ba2" (UID: "5e4deff1-e164-4ef2-b202-f489af121ba2"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 09 23:01:27.495154 ip-10-0-136-251 kubenswrapper[2579]: I0709 23:01:27.495073 2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5e4deff1-e164-4ef2-b202-f489af121ba2-kube-api-access-9sjlh" (OuterVolumeSpecName: "kube-api-access-9sjlh") pod "5e4deff1-e164-4ef2-b202-f489af121ba2" (UID: "5e4deff1-e164-4ef2-b202-f489af121ba2"). InnerVolumeSpecName "kube-api-access-9sjlh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 23:01:27.593570 ip-10-0-136-251 kubenswrapper[2579]: I0709 23:01:27.593534 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9sjlh\" (UniqueName: \"kubernetes.io/projected/5e4deff1-e164-4ef2-b202-f489af121ba2-kube-api-access-9sjlh\") on node \"ip-10-0-136-251.ec2.internal\" DevicePath \"\"" Jul 09 23:01:27.593570 ip-10-0-136-251 kubenswrapper[2579]: I0709 23:01:27.593564 2579 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5e4deff1-e164-4ef2-b202-f489af121ba2-dshm\") on node \"ip-10-0-136-251.ec2.internal\" DevicePath \"\"" Jul 09 23:01:27.593570 ip-10-0-136-251 kubenswrapper[2579]: I0709 23:01:27.593573 2579 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5e4deff1-e164-4ef2-b202-f489af121ba2-tls-certs\") on node \"ip-10-0-136-251.ec2.internal\" DevicePath \"\"" Jul 09 23:01:27.661294 ip-10-0-136-251 kubenswrapper[2579]: I0709 23:01:27.661195 2579 generic.go:356] "Generic (PLEG): container finished" podID="5e4deff1-e164-4ef2-b202-f489af121ba2" containerID="86260817dc05bfb91759446036d46157c1e60ee43b4d0cefa87c33e808309456" exitCode=0 Jul 09 23:01:27.661294 ip-10-0-136-251 kubenswrapper[2579]: I0709 23:01:27.661240 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-862c5d/rate-test-model-862c5d-kserve-84c477b7bc-5585s" Jul 09 23:01:27.661843 ip-10-0-136-251 kubenswrapper[2579]: I0709 23:01:27.661295 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-862c5d/rate-test-model-862c5d-kserve-84c477b7bc-5585s" event={"ID":"5e4deff1-e164-4ef2-b202-f489af121ba2","Type":"ContainerDied","Data":"86260817dc05bfb91759446036d46157c1e60ee43b4d0cefa87c33e808309456"} Jul 09 23:01:27.661843 ip-10-0-136-251 kubenswrapper[2579]: I0709 23:01:27.661338 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-862c5d/rate-test-model-862c5d-kserve-84c477b7bc-5585s" event={"ID":"5e4deff1-e164-4ef2-b202-f489af121ba2","Type":"ContainerDied","Data":"e867fff7426c3f1a695ea86e8a2a4519c516d77caa021f3d76f431fa0a6c3dbe"} Jul 09 23:01:27.661843 ip-10-0-136-251 kubenswrapper[2579]: I0709 23:01:27.661361 2579 scope.go:117] "RemoveContainer" containerID="86260817dc05bfb91759446036d46157c1e60ee43b4d0cefa87c33e808309456" Jul 09 23:01:27.671552 ip-10-0-136-251 kubenswrapper[2579]: I0709 23:01:27.671529 2579 scope.go:117] "RemoveContainer" containerID="86260817dc05bfb91759446036d46157c1e60ee43b4d0cefa87c33e808309456" Jul 09 23:01:27.671871 ip-10-0-136-251 kubenswrapper[2579]: E0709 23:01:27.671853 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"86260817dc05bfb91759446036d46157c1e60ee43b4d0cefa87c33e808309456\": container with ID starting with 86260817dc05bfb91759446036d46157c1e60ee43b4d0cefa87c33e808309456 not found: ID does not exist" containerID="86260817dc05bfb91759446036d46157c1e60ee43b4d0cefa87c33e808309456" Jul 09 23:01:27.671945 ip-10-0-136-251 kubenswrapper[2579]: I0709 23:01:27.671883 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"86260817dc05bfb91759446036d46157c1e60ee43b4d0cefa87c33e808309456"} err="failed to get container status \"86260817dc05bfb91759446036d46157c1e60ee43b4d0cefa87c33e808309456\": rpc error: code = NotFound desc = could not find container \"86260817dc05bfb91759446036d46157c1e60ee43b4d0cefa87c33e808309456\": container with ID starting with 86260817dc05bfb91759446036d46157c1e60ee43b4d0cefa87c33e808309456 not found: ID does not exist" Jul 09 23:01:27.685282 ip-10-0-136-251 kubenswrapper[2579]: I0709 23:01:27.685245 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-862c5d/rate-test-model-862c5d-kserve-84c477b7bc-5585s"] Jul 09 23:01:27.689053 ip-10-0-136-251 kubenswrapper[2579]: I0709 23:01:27.688889 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-862c5d/rate-test-model-862c5d-kserve-84c477b7bc-5585s"] Jul 09 23:01:27.804745 ip-10-0-136-251 kubenswrapper[2579]: I0709 23:01:27.804718 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7235-139910759980464" Jul 09 23:01:27.825142 ip-10-0-136-251 kubenswrapper[2579]: I0709 23:01:27.825103 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_test-curl-7235-139910759980464_e2ef48f1-5e75-4881-96f2-12ddf10864aa/test-curl-7235-139910759980464/0.log" Jul 09 23:01:27.853770 ip-10-0-136-251 kubenswrapper[2579]: I0709 23:01:27.853728 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/test-curl-7235-139910759980464"] Jul 09 23:01:27.857531 ip-10-0-136-251 kubenswrapper[2579]: I0709 23:01:27.857499 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/test-curl-7235-139910759980464"] Jul 09 23:01:27.895807 ip-10-0-136-251 kubenswrapper[2579]: I0709 23:01:27.895766 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7slnv\" (UniqueName: \"kubernetes.io/projected/e2ef48f1-5e75-4881-96f2-12ddf10864aa-kube-api-access-7slnv\") pod \"e2ef48f1-5e75-4881-96f2-12ddf10864aa\" (UID: \"e2ef48f1-5e75-4881-96f2-12ddf10864aa\") " Jul 09 23:01:27.897908 ip-10-0-136-251 kubenswrapper[2579]: I0709 23:01:27.897877 2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e2ef48f1-5e75-4881-96f2-12ddf10864aa-kube-api-access-7slnv" (OuterVolumeSpecName: "kube-api-access-7slnv") pod "e2ef48f1-5e75-4881-96f2-12ddf10864aa" (UID: "e2ef48f1-5e75-4881-96f2-12ddf10864aa"). InnerVolumeSpecName "kube-api-access-7slnv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 23:01:27.997302 ip-10-0-136-251 kubenswrapper[2579]: I0709 23:01:27.997268 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7slnv\" (UniqueName: \"kubernetes.io/projected/e2ef48f1-5e75-4881-96f2-12ddf10864aa-kube-api-access-7slnv\") on node \"ip-10-0-136-251.ec2.internal\" DevicePath \"\"" Jul 09 23:01:28.565251 ip-10-0-136-251 kubenswrapper[2579]: I0709 23:01:28.565204 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5e4deff1-e164-4ef2-b202-f489af121ba2" path="/var/lib/kubelet/pods/5e4deff1-e164-4ef2-b202-f489af121ba2/volumes" Jul 09 23:01:28.565655 ip-10-0-136-251 kubenswrapper[2579]: I0709 23:01:28.565634 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e2ef48f1-5e75-4881-96f2-12ddf10864aa" path="/var/lib/kubelet/pods/e2ef48f1-5e75-4881-96f2-12ddf10864aa/volumes" Jul 09 23:01:28.668549 ip-10-0-136-251 kubenswrapper[2579]: I0709 23:01:28.668456 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7235-139910759980464" Jul 09 23:01:28.668549 ip-10-0-136-251 kubenswrapper[2579]: I0709 23:01:28.668515 2579 scope.go:117] "RemoveContainer" containerID="fb4d7189d1168140e33dbe99936fb28aa13cfa00b8325acf00e7395c8141697d" Jul 09 23:01:41.660173 ip-10-0-136-251 kubenswrapper[2579]: I0709 23:01:41.660132 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-8a6043-29727300-4rt7k"] Jul 09 23:01:41.663150 ip-10-0-136-251 kubenswrapper[2579]: I0709 23:01:41.663119 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-8a6043-29727300-4rt7k"] Jul 09 23:01:42.564645 ip-10-0-136-251 kubenswrapper[2579]: I0709 23:01:42.564608 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="834b2c8b-6388-43b8-ba59-a1ab73ddfb1f" path="/var/lib/kubelet/pods/834b2c8b-6388-43b8-ba59-a1ab73ddfb1f/volumes" Jul 09 23:01:45.219968 ip-10-0-136-251 kubenswrapper[2579]: I0709 23:01:45.219927 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-xdfrr"] Jul 09 23:01:57.003719 ip-10-0-136-251 kubenswrapper[2579]: I0709 23:01:57.003680 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-8a6043-openshift-default-9b5ff877f-tnrjf"] Jul 09 23:01:57.004197 ip-10-0-136-251 kubenswrapper[2579]: I0709 23:01:57.003906 2579 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-8a6043-openshift-default-9b5ff877f-tnrjf" podUID="5d38a047-94c4-4dfb-aef7-8c96c09c4b33" containerName="istio-proxy" containerID="cri-o://15576633fcec968bf6daa19d2164fc8b8742e456e1be4c04a7be6d5e242aa00b" gracePeriod=30 Jul 09 23:02:02.289125 ip-10-0-136-251 kubenswrapper[2579]: I0709 23:02:02.289095 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-8a6043-openshift-default-9b5ff877f-tnrjf" Jul 09 23:02:02.424973 ip-10-0-136-251 kubenswrapper[2579]: I0709 23:02:02.424936 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lb647\" (UniqueName: \"kubernetes.io/projected/5d38a047-94c4-4dfb-aef7-8c96c09c4b33-kube-api-access-lb647\") pod \"5d38a047-94c4-4dfb-aef7-8c96c09c4b33\" (UID: \"5d38a047-94c4-4dfb-aef7-8c96c09c4b33\") " Jul 09 23:02:02.425159 ip-10-0-136-251 kubenswrapper[2579]: I0709 23:02:02.425040 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d38a047-94c4-4dfb-aef7-8c96c09c4b33-workload-socket\") pod \"5d38a047-94c4-4dfb-aef7-8c96c09c4b33\" (UID: \"5d38a047-94c4-4dfb-aef7-8c96c09c4b33\") " Jul 09 23:02:02.425159 ip-10-0-136-251 kubenswrapper[2579]: I0709 23:02:02.425068 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5d38a047-94c4-4dfb-aef7-8c96c09c4b33-istiod-ca-cert\") pod \"5d38a047-94c4-4dfb-aef7-8c96c09c4b33\" (UID: \"5d38a047-94c4-4dfb-aef7-8c96c09c4b33\") " Jul 09 23:02:02.425159 ip-10-0-136-251 kubenswrapper[2579]: I0709 23:02:02.425093 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5d38a047-94c4-4dfb-aef7-8c96c09c4b33-istio-token\") pod \"5d38a047-94c4-4dfb-aef7-8c96c09c4b33\" (UID: \"5d38a047-94c4-4dfb-aef7-8c96c09c4b33\") " Jul 09 23:02:02.425159 ip-10-0-136-251 kubenswrapper[2579]: I0709 23:02:02.425136 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5d38a047-94c4-4dfb-aef7-8c96c09c4b33-istio-data\") pod \"5d38a047-94c4-4dfb-aef7-8c96c09c4b33\" (UID: \"5d38a047-94c4-4dfb-aef7-8c96c09c4b33\") " Jul 09 23:02:02.425377 ip-10-0-136-251 kubenswrapper[2579]: I0709 23:02:02.425161 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5d38a047-94c4-4dfb-aef7-8c96c09c4b33-istio-podinfo\") pod \"5d38a047-94c4-4dfb-aef7-8c96c09c4b33\" (UID: \"5d38a047-94c4-4dfb-aef7-8c96c09c4b33\") " Jul 09 23:02:02.425377 ip-10-0-136-251 kubenswrapper[2579]: I0709 23:02:02.425209 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5d38a047-94c4-4dfb-aef7-8c96c09c4b33-istio-envoy\") pod \"5d38a047-94c4-4dfb-aef7-8c96c09c4b33\" (UID: \"5d38a047-94c4-4dfb-aef7-8c96c09c4b33\") " Jul 09 23:02:02.425377 ip-10-0-136-251 kubenswrapper[2579]: I0709 23:02:02.425239 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5d38a047-94c4-4dfb-aef7-8c96c09c4b33-workload-certs\") pod \"5d38a047-94c4-4dfb-aef7-8c96c09c4b33\" (UID: \"5d38a047-94c4-4dfb-aef7-8c96c09c4b33\") " Jul 09 23:02:02.425377 ip-10-0-136-251 kubenswrapper[2579]: I0709 23:02:02.425272 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d38a047-94c4-4dfb-aef7-8c96c09c4b33-credential-socket\") pod \"5d38a047-94c4-4dfb-aef7-8c96c09c4b33\" (UID: \"5d38a047-94c4-4dfb-aef7-8c96c09c4b33\") " Jul 09 23:02:02.425377 ip-10-0-136-251 kubenswrapper[2579]: I0709 23:02:02.425356 2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d38a047-94c4-4dfb-aef7-8c96c09c4b33-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "5d38a047-94c4-4dfb-aef7-8c96c09c4b33" (UID: "5d38a047-94c4-4dfb-aef7-8c96c09c4b33"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 23:02:02.425649 ip-10-0-136-251 kubenswrapper[2579]: I0709 23:02:02.425472 2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5d38a047-94c4-4dfb-aef7-8c96c09c4b33-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "5d38a047-94c4-4dfb-aef7-8c96c09c4b33" (UID: "5d38a047-94c4-4dfb-aef7-8c96c09c4b33"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 09 23:02:02.425649 ip-10-0-136-251 kubenswrapper[2579]: I0709 23:02:02.425588 2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d38a047-94c4-4dfb-aef7-8c96c09c4b33-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "5d38a047-94c4-4dfb-aef7-8c96c09c4b33" (UID: "5d38a047-94c4-4dfb-aef7-8c96c09c4b33"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 23:02:02.425764 ip-10-0-136-251 kubenswrapper[2579]: I0709 23:02:02.425718 2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d38a047-94c4-4dfb-aef7-8c96c09c4b33-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "5d38a047-94c4-4dfb-aef7-8c96c09c4b33" (UID: "5d38a047-94c4-4dfb-aef7-8c96c09c4b33"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 23:02:02.425867 ip-10-0-136-251 kubenswrapper[2579]: I0709 23:02:02.425847 2579 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d38a047-94c4-4dfb-aef7-8c96c09c4b33-credential-socket\") on node \"ip-10-0-136-251.ec2.internal\" DevicePath \"\"" Jul 09 23:02:02.425950 ip-10-0-136-251 kubenswrapper[2579]: I0709 23:02:02.425934 2579 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d38a047-94c4-4dfb-aef7-8c96c09c4b33-workload-socket\") on node \"ip-10-0-136-251.ec2.internal\" DevicePath \"\"" Jul 09 23:02:02.426004 ip-10-0-136-251 kubenswrapper[2579]: I0709 23:02:02.425955 2579 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5d38a047-94c4-4dfb-aef7-8c96c09c4b33-istiod-ca-cert\") on node \"ip-10-0-136-251.ec2.internal\" DevicePath \"\"" Jul 09 23:02:02.426004 ip-10-0-136-251 kubenswrapper[2579]: I0709 23:02:02.425969 2579 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5d38a047-94c4-4dfb-aef7-8c96c09c4b33-workload-certs\") on node \"ip-10-0-136-251.ec2.internal\" DevicePath \"\"" Jul 09 23:02:02.426143 ip-10-0-136-251 kubenswrapper[2579]: I0709 23:02:02.426119 2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d38a047-94c4-4dfb-aef7-8c96c09c4b33-istio-data" (OuterVolumeSpecName: "istio-data") pod "5d38a047-94c4-4dfb-aef7-8c96c09c4b33" (UID: "5d38a047-94c4-4dfb-aef7-8c96c09c4b33"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 23:02:02.427304 ip-10-0-136-251 kubenswrapper[2579]: I0709 23:02:02.427275 2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5d38a047-94c4-4dfb-aef7-8c96c09c4b33-kube-api-access-lb647" (OuterVolumeSpecName: "kube-api-access-lb647") pod "5d38a047-94c4-4dfb-aef7-8c96c09c4b33" (UID: "5d38a047-94c4-4dfb-aef7-8c96c09c4b33"). InnerVolumeSpecName "kube-api-access-lb647". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 23:02:02.427436 ip-10-0-136-251 kubenswrapper[2579]: I0709 23:02:02.427377 2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5d38a047-94c4-4dfb-aef7-8c96c09c4b33-istio-token" (OuterVolumeSpecName: "istio-token") pod "5d38a047-94c4-4dfb-aef7-8c96c09c4b33" (UID: "5d38a047-94c4-4dfb-aef7-8c96c09c4b33"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 23:02:02.427661 ip-10-0-136-251 kubenswrapper[2579]: I0709 23:02:02.427641 2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5d38a047-94c4-4dfb-aef7-8c96c09c4b33-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "5d38a047-94c4-4dfb-aef7-8c96c09c4b33" (UID: "5d38a047-94c4-4dfb-aef7-8c96c09c4b33"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 09 23:02:02.427741 ip-10-0-136-251 kubenswrapper[2579]: I0709 23:02:02.427724 2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d38a047-94c4-4dfb-aef7-8c96c09c4b33-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "5d38a047-94c4-4dfb-aef7-8c96c09c4b33" (UID: "5d38a047-94c4-4dfb-aef7-8c96c09c4b33"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 23:02:02.526515 ip-10-0-136-251 kubenswrapper[2579]: I0709 23:02:02.526422 2579 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5d38a047-94c4-4dfb-aef7-8c96c09c4b33-istio-token\") on node \"ip-10-0-136-251.ec2.internal\" DevicePath \"\"" Jul 09 23:02:02.526515 ip-10-0-136-251 kubenswrapper[2579]: I0709 23:02:02.526455 2579 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5d38a047-94c4-4dfb-aef7-8c96c09c4b33-istio-data\") on node \"ip-10-0-136-251.ec2.internal\" DevicePath \"\"" Jul 09 23:02:02.526515 ip-10-0-136-251 kubenswrapper[2579]: I0709 23:02:02.526465 2579 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5d38a047-94c4-4dfb-aef7-8c96c09c4b33-istio-podinfo\") on node \"ip-10-0-136-251.ec2.internal\" DevicePath \"\"" Jul 09 23:02:02.526515 ip-10-0-136-251 kubenswrapper[2579]: I0709 23:02:02.526473 2579 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5d38a047-94c4-4dfb-aef7-8c96c09c4b33-istio-envoy\") on node \"ip-10-0-136-251.ec2.internal\" DevicePath \"\"" Jul 09 23:02:02.526515 ip-10-0-136-251 kubenswrapper[2579]: I0709 23:02:02.526482 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lb647\" (UniqueName: \"kubernetes.io/projected/5d38a047-94c4-4dfb-aef7-8c96c09c4b33-kube-api-access-lb647\") on node \"ip-10-0-136-251.ec2.internal\" DevicePath \"\"" Jul 09 23:02:02.860646 ip-10-0-136-251 kubenswrapper[2579]: I0709 23:02:02.860548 2579 generic.go:356] "Generic (PLEG): container finished" podID="5d38a047-94c4-4dfb-aef7-8c96c09c4b33" containerID="15576633fcec968bf6daa19d2164fc8b8742e456e1be4c04a7be6d5e242aa00b" exitCode=0 Jul 09 23:02:02.860646 ip-10-0-136-251 kubenswrapper[2579]: I0709 23:02:02.860600 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-8a6043-openshift-default-9b5ff877f-tnrjf" Jul 09 23:02:02.860859 ip-10-0-136-251 kubenswrapper[2579]: I0709 23:02:02.860640 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-8a6043-openshift-default-9b5ff877f-tnrjf" event={"ID":"5d38a047-94c4-4dfb-aef7-8c96c09c4b33","Type":"ContainerDied","Data":"15576633fcec968bf6daa19d2164fc8b8742e456e1be4c04a7be6d5e242aa00b"} Jul 09 23:02:02.860859 ip-10-0-136-251 kubenswrapper[2579]: I0709 23:02:02.860697 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-8a6043-openshift-default-9b5ff877f-tnrjf" event={"ID":"5d38a047-94c4-4dfb-aef7-8c96c09c4b33","Type":"ContainerDied","Data":"a9afbfbf91ea365a6290ef58d8c53c96ca5deee82e3d785371be273488f66557"} Jul 09 23:02:02.860859 ip-10-0-136-251 kubenswrapper[2579]: I0709 23:02:02.860723 2579 scope.go:117] "RemoveContainer" containerID="15576633fcec968bf6daa19d2164fc8b8742e456e1be4c04a7be6d5e242aa00b" Jul 09 23:02:02.870132 ip-10-0-136-251 kubenswrapper[2579]: I0709 23:02:02.870112 2579 scope.go:117] "RemoveContainer" containerID="b267e23e798d2be20660c11893703c1997b518c5ee8ebefcfb3d5d8f97b30929" Jul 09 23:02:02.879208 ip-10-0-136-251 kubenswrapper[2579]: I0709 23:02:02.879184 2579 scope.go:117] "RemoveContainer" containerID="15576633fcec968bf6daa19d2164fc8b8742e456e1be4c04a7be6d5e242aa00b" Jul 09 23:02:02.879534 ip-10-0-136-251 kubenswrapper[2579]: E0709 23:02:02.879510 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"15576633fcec968bf6daa19d2164fc8b8742e456e1be4c04a7be6d5e242aa00b\": container with ID starting with 15576633fcec968bf6daa19d2164fc8b8742e456e1be4c04a7be6d5e242aa00b not found: ID does not exist" containerID="15576633fcec968bf6daa19d2164fc8b8742e456e1be4c04a7be6d5e242aa00b" Jul 09 23:02:02.879629 ip-10-0-136-251 kubenswrapper[2579]: I0709 23:02:02.879548 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"15576633fcec968bf6daa19d2164fc8b8742e456e1be4c04a7be6d5e242aa00b"} err="failed to get container status \"15576633fcec968bf6daa19d2164fc8b8742e456e1be4c04a7be6d5e242aa00b\": rpc error: code = NotFound desc = could not find container \"15576633fcec968bf6daa19d2164fc8b8742e456e1be4c04a7be6d5e242aa00b\": container with ID starting with 15576633fcec968bf6daa19d2164fc8b8742e456e1be4c04a7be6d5e242aa00b not found: ID does not exist" Jul 09 23:02:02.879629 ip-10-0-136-251 kubenswrapper[2579]: I0709 23:02:02.879574 2579 scope.go:117] "RemoveContainer" containerID="b267e23e798d2be20660c11893703c1997b518c5ee8ebefcfb3d5d8f97b30929" Jul 09 23:02:02.879832 ip-10-0-136-251 kubenswrapper[2579]: E0709 23:02:02.879815 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b267e23e798d2be20660c11893703c1997b518c5ee8ebefcfb3d5d8f97b30929\": container with ID starting with b267e23e798d2be20660c11893703c1997b518c5ee8ebefcfb3d5d8f97b30929 not found: ID does not exist" containerID="b267e23e798d2be20660c11893703c1997b518c5ee8ebefcfb3d5d8f97b30929" Jul 09 23:02:02.879886 ip-10-0-136-251 kubenswrapper[2579]: I0709 23:02:02.879842 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b267e23e798d2be20660c11893703c1997b518c5ee8ebefcfb3d5d8f97b30929"} err="failed to get container status \"b267e23e798d2be20660c11893703c1997b518c5ee8ebefcfb3d5d8f97b30929\": rpc error: code = NotFound desc = could not find container \"b267e23e798d2be20660c11893703c1997b518c5ee8ebefcfb3d5d8f97b30929\": container with ID starting with b267e23e798d2be20660c11893703c1997b518c5ee8ebefcfb3d5d8f97b30929 not found: ID does not exist" Jul 09 23:02:02.880997 ip-10-0-136-251 kubenswrapper[2579]: I0709 23:02:02.880977 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-8a6043-openshift-default-9b5ff877f-tnrjf"] Jul 09 23:02:02.884748 ip-10-0-136-251 kubenswrapper[2579]: I0709 23:02:02.884723 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-8a6043-openshift-default-9b5ff877f-tnrjf"] Jul 09 23:02:04.564139 ip-10-0-136-251 kubenswrapper[2579]: I0709 23:02:04.564104 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5d38a047-94c4-4dfb-aef7-8c96c09c4b33" path="/var/lib/kubelet/pods/5d38a047-94c4-4dfb-aef7-8c96c09c4b33/volumes" Jul 09 23:02:07.877572 ip-10-0-136-251 kubenswrapper[2579]: I0709 23:02:07.877533 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-862c5d-29727300-78j45"] Jul 09 23:02:07.882003 ip-10-0-136-251 kubenswrapper[2579]: I0709 23:02:07.881974 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-862c5d-29727300-78j45"] Jul 09 23:02:08.565347 ip-10-0-136-251 kubenswrapper[2579]: I0709 23:02:08.565311 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f5ddc88c-bc30-4c14-aa1f-dd54f4a8e633" path="/var/lib/kubelet/pods/f5ddc88c-bc30-4c14-aa1f-dd54f4a8e633/volumes" Jul 09 23:02:11.013491 ip-10-0-136-251 kubenswrapper[2579]: I0709 23:02:11.013454 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-xdfrr"] Jul 09 23:02:23.168736 ip-10-0-136-251 kubenswrapper[2579]: I0709 23:02:23.168701 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-862c5d-openshift-default-f4fdbd455-xf87k"] Jul 09 23:02:23.169416 ip-10-0-136-251 kubenswrapper[2579]: I0709 23:02:23.168937 2579 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-862c5d-openshift-default-f4fdbd455-xf87k" podUID="a3eee859-2f99-438f-b4ad-ca6575ecff6e" containerName="istio-proxy" containerID="cri-o://c4c6267c192d50f7d3d41c00c94f3526281c2cb44b384a699125396e657508fc" gracePeriod=30 Jul 09 23:02:28.434487 ip-10-0-136-251 kubenswrapper[2579]: I0709 23:02:28.434416 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-862c5d-openshift-default-f4fdbd455-xf87k" Jul 09 23:02:28.563167 ip-10-0-136-251 kubenswrapper[2579]: I0709 23:02:28.563135 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a3eee859-2f99-438f-b4ad-ca6575ecff6e-istiod-ca-cert\") pod \"a3eee859-2f99-438f-b4ad-ca6575ecff6e\" (UID: \"a3eee859-2f99-438f-b4ad-ca6575ecff6e\") " Jul 09 23:02:28.563357 ip-10-0-136-251 kubenswrapper[2579]: I0709 23:02:28.563223 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3eee859-2f99-438f-b4ad-ca6575ecff6e-workload-socket\") pod \"a3eee859-2f99-438f-b4ad-ca6575ecff6e\" (UID: \"a3eee859-2f99-438f-b4ad-ca6575ecff6e\") " Jul 09 23:02:28.563357 ip-10-0-136-251 kubenswrapper[2579]: I0709 23:02:28.563254 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a3eee859-2f99-438f-b4ad-ca6575ecff6e-workload-certs\") pod \"a3eee859-2f99-438f-b4ad-ca6575ecff6e\" (UID: \"a3eee859-2f99-438f-b4ad-ca6575ecff6e\") " Jul 09 23:02:28.563357 ip-10-0-136-251 kubenswrapper[2579]: I0709 23:02:28.563280 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a3eee859-2f99-438f-b4ad-ca6575ecff6e-istio-data\") pod \"a3eee859-2f99-438f-b4ad-ca6575ecff6e\" (UID: \"a3eee859-2f99-438f-b4ad-ca6575ecff6e\") " Jul 09 23:02:28.563357 ip-10-0-136-251 kubenswrapper[2579]: I0709 23:02:28.563307 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3eee859-2f99-438f-b4ad-ca6575ecff6e-credential-socket\") pod \"a3eee859-2f99-438f-b4ad-ca6575ecff6e\" (UID: \"a3eee859-2f99-438f-b4ad-ca6575ecff6e\") " Jul 09 23:02:28.563357 ip-10-0-136-251 kubenswrapper[2579]: I0709 23:02:28.563333 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a3eee859-2f99-438f-b4ad-ca6575ecff6e-istio-envoy\") pod \"a3eee859-2f99-438f-b4ad-ca6575ecff6e\" (UID: \"a3eee859-2f99-438f-b4ad-ca6575ecff6e\") " Jul 09 23:02:28.563828 ip-10-0-136-251 kubenswrapper[2579]: I0709 23:02:28.563362 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a3eee859-2f99-438f-b4ad-ca6575ecff6e-istio-podinfo\") pod \"a3eee859-2f99-438f-b4ad-ca6575ecff6e\" (UID: \"a3eee859-2f99-438f-b4ad-ca6575ecff6e\") " Jul 09 23:02:28.563828 ip-10-0-136-251 kubenswrapper[2579]: I0709 23:02:28.563397 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a3eee859-2f99-438f-b4ad-ca6575ecff6e-istio-token\") pod \"a3eee859-2f99-438f-b4ad-ca6575ecff6e\" (UID: \"a3eee859-2f99-438f-b4ad-ca6575ecff6e\") " Jul 09 23:02:28.563828 ip-10-0-136-251 kubenswrapper[2579]: I0709 23:02:28.563442 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-v5xqf\" (UniqueName: \"kubernetes.io/projected/a3eee859-2f99-438f-b4ad-ca6575ecff6e-kube-api-access-v5xqf\") pod \"a3eee859-2f99-438f-b4ad-ca6575ecff6e\" (UID: \"a3eee859-2f99-438f-b4ad-ca6575ecff6e\") " Jul 09 23:02:28.563828 ip-10-0-136-251 kubenswrapper[2579]: I0709 23:02:28.563591 2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a3eee859-2f99-438f-b4ad-ca6575ecff6e-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "a3eee859-2f99-438f-b4ad-ca6575ecff6e" (UID: "a3eee859-2f99-438f-b4ad-ca6575ecff6e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 09 23:02:28.563828 ip-10-0-136-251 kubenswrapper[2579]: I0709 23:02:28.563766 2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a3eee859-2f99-438f-b4ad-ca6575ecff6e-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "a3eee859-2f99-438f-b4ad-ca6575ecff6e" (UID: "a3eee859-2f99-438f-b4ad-ca6575ecff6e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 23:02:28.564126 ip-10-0-136-251 kubenswrapper[2579]: I0709 23:02:28.563594 2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a3eee859-2f99-438f-b4ad-ca6575ecff6e-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "a3eee859-2f99-438f-b4ad-ca6575ecff6e" (UID: "a3eee859-2f99-438f-b4ad-ca6575ecff6e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 23:02:28.564126 ip-10-0-136-251 kubenswrapper[2579]: I0709 23:02:28.563963 2579 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a3eee859-2f99-438f-b4ad-ca6575ecff6e-istiod-ca-cert\") on node \"ip-10-0-136-251.ec2.internal\" DevicePath \"\"" Jul 09 23:02:28.564126 ip-10-0-136-251 kubenswrapper[2579]: I0709 23:02:28.563991 2579 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3eee859-2f99-438f-b4ad-ca6575ecff6e-workload-socket\") on node \"ip-10-0-136-251.ec2.internal\" DevicePath \"\"" Jul 09 23:02:28.564126 ip-10-0-136-251 kubenswrapper[2579]: I0709 23:02:28.564005 2579 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a3eee859-2f99-438f-b4ad-ca6575ecff6e-workload-certs\") on node \"ip-10-0-136-251.ec2.internal\" DevicePath \"\"" Jul 09 23:02:28.564126 ip-10-0-136-251 kubenswrapper[2579]: I0709 23:02:28.563958 2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a3eee859-2f99-438f-b4ad-ca6575ecff6e-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "a3eee859-2f99-438f-b4ad-ca6575ecff6e" (UID: "a3eee859-2f99-438f-b4ad-ca6575ecff6e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 23:02:28.564661 ip-10-0-136-251 kubenswrapper[2579]: I0709 23:02:28.564621 2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a3eee859-2f99-438f-b4ad-ca6575ecff6e-istio-data" (OuterVolumeSpecName: "istio-data") pod "a3eee859-2f99-438f-b4ad-ca6575ecff6e" (UID: "a3eee859-2f99-438f-b4ad-ca6575ecff6e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 23:02:28.565838 ip-10-0-136-251 kubenswrapper[2579]: I0709 23:02:28.565814 2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a3eee859-2f99-438f-b4ad-ca6575ecff6e-istio-token" (OuterVolumeSpecName: "istio-token") pod "a3eee859-2f99-438f-b4ad-ca6575ecff6e" (UID: "a3eee859-2f99-438f-b4ad-ca6575ecff6e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 23:02:28.565979 ip-10-0-136-251 kubenswrapper[2579]: I0709 23:02:28.565863 2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a3eee859-2f99-438f-b4ad-ca6575ecff6e-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "a3eee859-2f99-438f-b4ad-ca6575ecff6e" (UID: "a3eee859-2f99-438f-b4ad-ca6575ecff6e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 09 23:02:28.566108 ip-10-0-136-251 kubenswrapper[2579]: I0709 23:02:28.566089 2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a3eee859-2f99-438f-b4ad-ca6575ecff6e-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "a3eee859-2f99-438f-b4ad-ca6575ecff6e" (UID: "a3eee859-2f99-438f-b4ad-ca6575ecff6e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 09 23:02:28.566221 ip-10-0-136-251 kubenswrapper[2579]: I0709 23:02:28.566195 2579 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a3eee859-2f99-438f-b4ad-ca6575ecff6e-kube-api-access-v5xqf" (OuterVolumeSpecName: "kube-api-access-v5xqf") pod "a3eee859-2f99-438f-b4ad-ca6575ecff6e" (UID: "a3eee859-2f99-438f-b4ad-ca6575ecff6e"). InnerVolumeSpecName "kube-api-access-v5xqf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 09 23:02:28.665483 ip-10-0-136-251 kubenswrapper[2579]: I0709 23:02:28.665441 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v5xqf\" (UniqueName: \"kubernetes.io/projected/a3eee859-2f99-438f-b4ad-ca6575ecff6e-kube-api-access-v5xqf\") on node \"ip-10-0-136-251.ec2.internal\" DevicePath \"\"" Jul 09 23:02:28.665483 ip-10-0-136-251 kubenswrapper[2579]: I0709 23:02:28.665476 2579 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a3eee859-2f99-438f-b4ad-ca6575ecff6e-istio-data\") on node \"ip-10-0-136-251.ec2.internal\" DevicePath \"\"" Jul 09 23:02:28.665483 ip-10-0-136-251 kubenswrapper[2579]: I0709 23:02:28.665486 2579 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3eee859-2f99-438f-b4ad-ca6575ecff6e-credential-socket\") on node \"ip-10-0-136-251.ec2.internal\" DevicePath \"\"" Jul 09 23:02:28.665483 ip-10-0-136-251 kubenswrapper[2579]: I0709 23:02:28.665494 2579 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a3eee859-2f99-438f-b4ad-ca6575ecff6e-istio-envoy\") on node \"ip-10-0-136-251.ec2.internal\" DevicePath \"\"" Jul 09 23:02:28.665852 ip-10-0-136-251 kubenswrapper[2579]: I0709 23:02:28.665503 2579 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a3eee859-2f99-438f-b4ad-ca6575ecff6e-istio-podinfo\") on node \"ip-10-0-136-251.ec2.internal\" DevicePath \"\"" Jul 09 23:02:28.665852 ip-10-0-136-251 kubenswrapper[2579]: I0709 23:02:28.665511 2579 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a3eee859-2f99-438f-b4ad-ca6575ecff6e-istio-token\") on node \"ip-10-0-136-251.ec2.internal\" DevicePath \"\"" Jul 09 23:02:29.013272 ip-10-0-136-251 kubenswrapper[2579]: I0709 23:02:29.013232 2579 generic.go:356] "Generic (PLEG): container finished" podID="a3eee859-2f99-438f-b4ad-ca6575ecff6e" containerID="c4c6267c192d50f7d3d41c00c94f3526281c2cb44b384a699125396e657508fc" exitCode=0 Jul 09 23:02:29.013272 ip-10-0-136-251 kubenswrapper[2579]: I0709 23:02:29.013277 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-862c5d-openshift-default-f4fdbd455-xf87k" Jul 09 23:02:29.013548 ip-10-0-136-251 kubenswrapper[2579]: I0709 23:02:29.013290 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-862c5d-openshift-default-f4fdbd455-xf87k" event={"ID":"a3eee859-2f99-438f-b4ad-ca6575ecff6e","Type":"ContainerDied","Data":"c4c6267c192d50f7d3d41c00c94f3526281c2cb44b384a699125396e657508fc"} Jul 09 23:02:29.013548 ip-10-0-136-251 kubenswrapper[2579]: I0709 23:02:29.013345 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-862c5d-openshift-default-f4fdbd455-xf87k" event={"ID":"a3eee859-2f99-438f-b4ad-ca6575ecff6e","Type":"ContainerDied","Data":"5122d7b22e0db70c9f99c7ac46c22d8979cdf2a0e0bd1c3bab3830bf084c0cc0"} Jul 09 23:02:29.013548 ip-10-0-136-251 kubenswrapper[2579]: I0709 23:02:29.013374 2579 scope.go:117] "RemoveContainer" containerID="c4c6267c192d50f7d3d41c00c94f3526281c2cb44b384a699125396e657508fc" Jul 09 23:02:29.023475 ip-10-0-136-251 kubenswrapper[2579]: I0709 23:02:29.023459 2579 scope.go:117] "RemoveContainer" containerID="a5e11bb3de725cfb61f6103f960121ce6e549ebb4858c9298485ab93f5fb9ae7" Jul 09 23:02:29.029784 ip-10-0-136-251 kubenswrapper[2579]: I0709 23:02:29.029763 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-b6tv8_6a02a146-e107-4650-828d-e092ad914d87/manager/0.log" Jul 09 23:02:29.032600 ip-10-0-136-251 kubenswrapper[2579]: I0709 23:02:29.032582 2579 scope.go:117] "RemoveContainer" containerID="c4c6267c192d50f7d3d41c00c94f3526281c2cb44b384a699125396e657508fc" Jul 09 23:02:29.032832 ip-10-0-136-251 kubenswrapper[2579]: E0709 23:02:29.032811 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c4c6267c192d50f7d3d41c00c94f3526281c2cb44b384a699125396e657508fc\": container with ID starting with c4c6267c192d50f7d3d41c00c94f3526281c2cb44b384a699125396e657508fc not found: ID does not exist" containerID="c4c6267c192d50f7d3d41c00c94f3526281c2cb44b384a699125396e657508fc" Jul 09 23:02:29.032902 ip-10-0-136-251 kubenswrapper[2579]: I0709 23:02:29.032845 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c4c6267c192d50f7d3d41c00c94f3526281c2cb44b384a699125396e657508fc"} err="failed to get container status \"c4c6267c192d50f7d3d41c00c94f3526281c2cb44b384a699125396e657508fc\": rpc error: code = NotFound desc = could not find container \"c4c6267c192d50f7d3d41c00c94f3526281c2cb44b384a699125396e657508fc\": container with ID starting with c4c6267c192d50f7d3d41c00c94f3526281c2cb44b384a699125396e657508fc not found: ID does not exist" Jul 09 23:02:29.032902 ip-10-0-136-251 kubenswrapper[2579]: I0709 23:02:29.032868 2579 scope.go:117] "RemoveContainer" containerID="a5e11bb3de725cfb61f6103f960121ce6e549ebb4858c9298485ab93f5fb9ae7" Jul 09 23:02:29.033155 ip-10-0-136-251 kubenswrapper[2579]: E0709 23:02:29.033136 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a5e11bb3de725cfb61f6103f960121ce6e549ebb4858c9298485ab93f5fb9ae7\": container with ID starting with a5e11bb3de725cfb61f6103f960121ce6e549ebb4858c9298485ab93f5fb9ae7 not found: ID does not exist" containerID="a5e11bb3de725cfb61f6103f960121ce6e549ebb4858c9298485ab93f5fb9ae7" Jul 09 23:02:29.033200 ip-10-0-136-251 kubenswrapper[2579]: I0709 23:02:29.033162 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a5e11bb3de725cfb61f6103f960121ce6e549ebb4858c9298485ab93f5fb9ae7"} err="failed to get container status \"a5e11bb3de725cfb61f6103f960121ce6e549ebb4858c9298485ab93f5fb9ae7\": rpc error: code = NotFound desc = could not find container \"a5e11bb3de725cfb61f6103f960121ce6e549ebb4858c9298485ab93f5fb9ae7\": container with ID starting with a5e11bb3de725cfb61f6103f960121ce6e549ebb4858c9298485ab93f5fb9ae7 not found: ID does not exist" Jul 09 23:02:29.038250 ip-10-0-136-251 kubenswrapper[2579]: I0709 23:02:29.038228 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-862c5d-openshift-default-f4fdbd455-xf87k"] Jul 09 23:02:29.043457 ip-10-0-136-251 kubenswrapper[2579]: I0709 23:02:29.043429 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-862c5d-openshift-default-f4fdbd455-xf87k"] Jul 09 23:02:29.154676 ip-10-0-136-251 kubenswrapper[2579]: I0709 23:02:29.154643 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-btzms_29c83de2-afa8-48fa-b975-93747a089549/manager/0.log" Jul 09 23:02:29.816706 ip-10-0-136-251 kubenswrapper[2579]: I0709 23:02:29.816669 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-5cf875db85-78gjk_0083cb04-279c-4d25-bd33-b639a0b1a0f4/copy-manifests/0.log" Jul 09 23:02:29.829566 ip-10-0-136-251 kubenswrapper[2579]: I0709 23:02:29.829541 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-5cf875db85-78gjk_0083cb04-279c-4d25-bd33-b639a0b1a0f4/manager/0.log" Jul 09 23:02:30.456583 ip-10-0-136-251 kubenswrapper[2579]: I0709 23:02:30.456556 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29727270-zzjnb_0b468c92-33c1-40d7-81ad-d3fdeed9c632/cleanup/0.log" Jul 09 23:02:30.564257 ip-10-0-136-251 kubenswrapper[2579]: I0709 23:02:30.564181 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a3eee859-2f99-438f-b4ad-ca6575ecff6e" path="/var/lib/kubelet/pods/a3eee859-2f99-438f-b4ad-ca6575ecff6e/volumes" Jul 09 23:02:30.568170 ip-10-0-136-251 kubenswrapper[2579]: I0709 23:02:30.568147 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29727285-grs86_413b1426-ad61-40dd-98f1-289859e64232/cleanup/0.log" Jul 09 23:02:30.699480 ip-10-0-136-251 kubenswrapper[2579]: I0709 23:02:30.699449 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29727300-mfvpq_f3972d63-c4f4-4011-bea3-681bc562086d/cleanup/0.log" Jul 09 23:02:30.815294 ip-10-0-136-251 kubenswrapper[2579]: I0709 23:02:30.815266 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-keys-e2e-ait-489a4fc6-mt8s9_d5613506-b584-4cfd-954e-eb6a5605350e/revoke-keys/0.log" Jul 09 23:02:30.927442 ip-10-0-136-251 kubenswrapper[2579]: I0709 23:02:30.927413 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-keys-e2e-ait-fdfdaecd-n6thq_08b884e8-79e7-4e06-98ea-b76a552d2fce/revoke-keys/1.log" Jul 09 23:02:31.288744 ip-10-0-136-251 kubenswrapper[2579]: I0709 23:02:31.288716 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-keys-e2e-derive-e08843d5-czkpv_d6bc0f38-299a-485a-8484-aafc7f04fca9/revoke-keys/0.log" Jul 09 23:02:31.406351 ip-10-0-136-251 kubenswrapper[2579]: I0709 23:02:31.406322 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-keys-e2e-inf-a-288815-n4kpm_9bc7fad2-3dad-4e26-a232-cfb0a2739fc4/revoke-keys/0.log" Jul 09 23:02:31.672291 ip-10-0-136-251 kubenswrapper[2579]: I0709 23:02:31.672207 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-keys-e2e-mt-67cb547f-q8dqt_f9be0827-40f8-4eef-be83-3abc36f47b7f/revoke-keys/0.log" Jul 09 23:02:31.793016 ip-10-0-136-251 kubenswrapper[2579]: I0709 23:02:31.792989 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-keys-e2e-mt-eb6719bd-722zs_5e177509-1d82-4fee-89b3-c306b771165e/revoke-keys/0.log" Jul 09 23:02:33.657500 ip-10-0-136-251 kubenswrapper[2579]: I0709 23:02:33.657457 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-75cc8d5cb7-zm6ft_c97c2fa5-5376-4f0a-bdc3-c8d69720ab75/manager/0.log" Jul 09 23:02:33.771851 ip-10-0-136-251 kubenswrapper[2579]: I0709 23:02:33.771820 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-864dd78c4c-q6sw8_af0b3a7a-21c8-43ae-95d0-2f09e3dee7f4/manager/0.log" Jul 09 23:02:34.293271 ip-10-0-136-251 kubenswrapper[2579]: I0709 23:02:34.293244 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-54d6d988d5-xdfrr_35f67383-3d5f-44fa-99d0-57f2c2a927bb/limitador/0.log" Jul 09 23:02:34.807150 ip-10-0-136-251 kubenswrapper[2579]: I0709 23:02:34.807120 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-6576859cf4x6nwz_bc747b34-dcf6-43d4-989f-35f49f02b32d/istio-proxy/0.log" Jul 09 23:02:35.325430 ip-10-0-136-251 kubenswrapper[2579]: I0709 23:02:35.325397 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-j9x6z_9a0d3041-24b6-42f9-9992-32cc62e3810e/istio-proxy/1.log" Jul 09 23:02:35.742212 ip-10-0-136-251 kubenswrapper[2579]: I0709 23:02:35.742167 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-854859444c-bhmbj_31587075-3ee2-43c7-9d0e-5340de9912a1/router/0.log" Jul 09 23:02:43.469770 ip-10-0-136-251 kubenswrapper[2579]: I0709 23:02:43.469737 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-bwdgc_b5e569ee-1aea-4eeb-8235-93438e648102/global-pull-secret-syncer/0.log" Jul 09 23:02:43.646722 ip-10-0-136-251 kubenswrapper[2579]: I0709 23:02:43.646690 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-gcmz7_5946d6db-f1f5-4fee-827b-1427b1bdd90f/konnectivity-agent/0.log" Jul 09 23:02:43.734484 ip-10-0-136-251 kubenswrapper[2579]: I0709 23:02:43.734404 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-136-251.ec2.internal_50613fbded229913a0a9abdd811c3ca4/haproxy/0.log" Jul 09 23:02:49.730576 ip-10-0-136-251 kubenswrapper[2579]: I0709 23:02:49.730539 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-75cc8d5cb7-zm6ft_c97c2fa5-5376-4f0a-bdc3-c8d69720ab75/manager/0.log" Jul 09 23:02:49.766823 ip-10-0-136-251 kubenswrapper[2579]: I0709 23:02:49.766789 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-864dd78c4c-q6sw8_af0b3a7a-21c8-43ae-95d0-2f09e3dee7f4/manager/0.log" Jul 09 23:02:50.075067 ip-10-0-136-251 kubenswrapper[2579]: I0709 23:02:50.075036 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-54d6d988d5-xdfrr_35f67383-3d5f-44fa-99d0-57f2c2a927bb/limitador/0.log" Jul 09 23:02:52.266710 ip-10-0-136-251 kubenswrapper[2579]: I0709 23:02:52.266673 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-d5dcff7cf-d6q56_91037a42-3ca0-4174-9ce2-ade5caadea9f/monitoring-plugin/0.log" Jul 09 23:02:52.412738 ip-10-0-136-251 kubenswrapper[2579]: I0709 23:02:52.412704 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-qcsd5_f9a132fa-2995-418a-9f53-19ecdf6ec341/node-exporter/0.log" Jul 09 23:02:52.457378 ip-10-0-136-251 kubenswrapper[2579]: I0709 23:02:52.457352 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-qcsd5_f9a132fa-2995-418a-9f53-19ecdf6ec341/kube-rbac-proxy/0.log" Jul 09 23:02:52.501158 ip-10-0-136-251 kubenswrapper[2579]: I0709 23:02:52.501130 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-qcsd5_f9a132fa-2995-418a-9f53-19ecdf6ec341/init-textfile/0.log" Jul 09 23:02:52.742391 ip-10-0-136-251 kubenswrapper[2579]: I0709 23:02:52.742356 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_c3770ca5-9562-4939-bf80-3ff488746902/prometheus/0.log" Jul 09 23:02:52.780829 ip-10-0-136-251 kubenswrapper[2579]: I0709 23:02:52.780799 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_c3770ca5-9562-4939-bf80-3ff488746902/config-reloader/0.log" Jul 09 23:02:52.821886 ip-10-0-136-251 kubenswrapper[2579]: I0709 23:02:52.821862 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_c3770ca5-9562-4939-bf80-3ff488746902/thanos-sidecar/0.log" Jul 09 23:02:52.866672 ip-10-0-136-251 kubenswrapper[2579]: I0709 23:02:52.866644 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_c3770ca5-9562-4939-bf80-3ff488746902/kube-rbac-proxy-web/0.log" Jul 09 23:02:52.904729 ip-10-0-136-251 kubenswrapper[2579]: I0709 23:02:52.904698 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_c3770ca5-9562-4939-bf80-3ff488746902/kube-rbac-proxy/0.log" Jul 09 23:02:52.931525 ip-10-0-136-251 kubenswrapper[2579]: I0709 23:02:52.931500 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_c3770ca5-9562-4939-bf80-3ff488746902/kube-rbac-proxy-thanos/0.log" Jul 09 23:02:52.954661 ip-10-0-136-251 kubenswrapper[2579]: I0709 23:02:52.954637 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_c3770ca5-9562-4939-bf80-3ff488746902/init-config-reloader/0.log" Jul 09 23:02:52.984178 ip-10-0-136-251 kubenswrapper[2579]: I0709 23:02:52.984148 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-567f7b8b7c-d8kq7_04192941-26ac-4e9e-a52a-381d544238d2/prometheus-operator/0.log" Jul 09 23:02:53.004308 ip-10-0-136-251 kubenswrapper[2579]: I0709 23:02:53.004229 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-567f7b8b7c-d8kq7_04192941-26ac-4e9e-a52a-381d544238d2/kube-rbac-proxy/0.log" Jul 09 23:02:53.172774 ip-10-0-136-251 kubenswrapper[2579]: I0709 23:02:53.172742 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-74f4c8dc89-jclzs_f27c5db7-84bc-49ca-8654-64590044faa0/thanos-query/0.log" Jul 09 23:02:53.226067 ip-10-0-136-251 kubenswrapper[2579]: I0709 23:02:53.226036 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-74f4c8dc89-jclzs_f27c5db7-84bc-49ca-8654-64590044faa0/kube-rbac-proxy-web/0.log" Jul 09 23:02:53.248543 ip-10-0-136-251 kubenswrapper[2579]: I0709 23:02:53.248520 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-74f4c8dc89-jclzs_f27c5db7-84bc-49ca-8654-64590044faa0/kube-rbac-proxy/0.log" Jul 09 23:02:53.280183 ip-10-0-136-251 kubenswrapper[2579]: I0709 23:02:53.280106 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-74f4c8dc89-jclzs_f27c5db7-84bc-49ca-8654-64590044faa0/prom-label-proxy/0.log" Jul 09 23:02:53.305229 ip-10-0-136-251 kubenswrapper[2579]: I0709 23:02:53.305199 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-74f4c8dc89-jclzs_f27c5db7-84bc-49ca-8654-64590044faa0/kube-rbac-proxy-rules/0.log" Jul 09 23:02:53.330558 ip-10-0-136-251 kubenswrapper[2579]: I0709 23:02:53.330527 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-74f4c8dc89-jclzs_f27c5db7-84bc-49ca-8654-64590044faa0/kube-rbac-proxy-metrics/0.log" Jul 09 23:02:53.520393 ip-10-0-136-251 kubenswrapper[2579]: I0709 23:02:53.520355 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-5rfjs/perf-node-gather-daemonset-4jcj2"] Jul 09 23:02:53.520739 ip-10-0-136-251 kubenswrapper[2579]: I0709 23:02:53.520727 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5d38a047-94c4-4dfb-aef7-8c96c09c4b33" containerName="istio-proxy" Jul 09 23:02:53.520783 ip-10-0-136-251 kubenswrapper[2579]: I0709 23:02:53.520740 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="5d38a047-94c4-4dfb-aef7-8c96c09c4b33" containerName="istio-proxy" Jul 09 23:02:53.520783 ip-10-0-136-251 kubenswrapper[2579]: I0709 23:02:53.520752 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a3eee859-2f99-438f-b4ad-ca6575ecff6e" containerName="istio-proxy" Jul 09 23:02:53.520783 ip-10-0-136-251 kubenswrapper[2579]: I0709 23:02:53.520758 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="a3eee859-2f99-438f-b4ad-ca6575ecff6e" containerName="istio-proxy" Jul 09 23:02:53.520783 ip-10-0-136-251 kubenswrapper[2579]: I0709 23:02:53.520766 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5e4deff1-e164-4ef2-b202-f489af121ba2" containerName="main" Jul 09 23:02:53.520783 ip-10-0-136-251 kubenswrapper[2579]: I0709 23:02:53.520771 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="5e4deff1-e164-4ef2-b202-f489af121ba2" containerName="main" Jul 09 23:02:53.520965 ip-10-0-136-251 kubenswrapper[2579]: I0709 23:02:53.520787 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e2ef48f1-5e75-4881-96f2-12ddf10864aa" containerName="test-curl-7235-139910759980464" Jul 09 23:02:53.520965 ip-10-0-136-251 kubenswrapper[2579]: I0709 23:02:53.520793 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="e2ef48f1-5e75-4881-96f2-12ddf10864aa" containerName="test-curl-7235-139910759980464" Jul 09 23:02:53.520965 ip-10-0-136-251 kubenswrapper[2579]: I0709 23:02:53.520858 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="5d38a047-94c4-4dfb-aef7-8c96c09c4b33" containerName="istio-proxy" Jul 09 23:02:53.520965 ip-10-0-136-251 kubenswrapper[2579]: I0709 23:02:53.520866 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="5e4deff1-e164-4ef2-b202-f489af121ba2" containerName="main" Jul 09 23:02:53.520965 ip-10-0-136-251 kubenswrapper[2579]: I0709 23:02:53.520873 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="5d38a047-94c4-4dfb-aef7-8c96c09c4b33" containerName="istio-proxy" Jul 09 23:02:53.520965 ip-10-0-136-251 kubenswrapper[2579]: I0709 23:02:53.520879 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="e2ef48f1-5e75-4881-96f2-12ddf10864aa" containerName="test-curl-7235-139910759980464" Jul 09 23:02:53.520965 ip-10-0-136-251 kubenswrapper[2579]: I0709 23:02:53.520885 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="a3eee859-2f99-438f-b4ad-ca6575ecff6e" containerName="istio-proxy" Jul 09 23:02:53.520965 ip-10-0-136-251 kubenswrapper[2579]: I0709 23:02:53.520891 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="a3eee859-2f99-438f-b4ad-ca6575ecff6e" containerName="istio-proxy" Jul 09 23:02:53.524197 ip-10-0-136-251 kubenswrapper[2579]: I0709 23:02:53.524175 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-5rfjs/perf-node-gather-daemonset-4jcj2" Jul 09 23:02:53.526782 ip-10-0-136-251 kubenswrapper[2579]: I0709 23:02:53.526761 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-5rfjs\"/\"openshift-service-ca.crt\"" Jul 09 23:02:53.526896 ip-10-0-136-251 kubenswrapper[2579]: I0709 23:02:53.526762 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-5rfjs\"/\"default-dockercfg-dbb7w\"" Jul 09 23:02:53.526896 ip-10-0-136-251 kubenswrapper[2579]: I0709 23:02:53.526830 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-5rfjs\"/\"kube-root-ca.crt\"" Jul 09 23:02:53.534760 ip-10-0-136-251 kubenswrapper[2579]: I0709 23:02:53.534733 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-5rfjs/perf-node-gather-daemonset-4jcj2"] Jul 09 23:02:53.695717 ip-10-0-136-251 kubenswrapper[2579]: I0709 23:02:53.695679 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/d326b5d7-6b28-44dc-a0b4-bd4d5b4ab63d-podres\") pod \"perf-node-gather-daemonset-4jcj2\" (UID: \"d326b5d7-6b28-44dc-a0b4-bd4d5b4ab63d\") " pod="openshift-must-gather-5rfjs/perf-node-gather-daemonset-4jcj2" Jul 09 23:02:53.695717 ip-10-0-136-251 kubenswrapper[2579]: I0709 23:02:53.695719 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d326b5d7-6b28-44dc-a0b4-bd4d5b4ab63d-lib-modules\") pod \"perf-node-gather-daemonset-4jcj2\" (UID: \"d326b5d7-6b28-44dc-a0b4-bd4d5b4ab63d\") " pod="openshift-must-gather-5rfjs/perf-node-gather-daemonset-4jcj2" Jul 09 23:02:53.696000 ip-10-0-136-251 kubenswrapper[2579]: I0709 23:02:53.695748 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d326b5d7-6b28-44dc-a0b4-bd4d5b4ab63d-sys\") pod \"perf-node-gather-daemonset-4jcj2\" (UID: \"d326b5d7-6b28-44dc-a0b4-bd4d5b4ab63d\") " pod="openshift-must-gather-5rfjs/perf-node-gather-daemonset-4jcj2" Jul 09 23:02:53.696000 ip-10-0-136-251 kubenswrapper[2579]: I0709 23:02:53.695884 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r9889\" (UniqueName: \"kubernetes.io/projected/d326b5d7-6b28-44dc-a0b4-bd4d5b4ab63d-kube-api-access-r9889\") pod \"perf-node-gather-daemonset-4jcj2\" (UID: \"d326b5d7-6b28-44dc-a0b4-bd4d5b4ab63d\") " pod="openshift-must-gather-5rfjs/perf-node-gather-daemonset-4jcj2" Jul 09 23:02:53.696000 ip-10-0-136-251 kubenswrapper[2579]: I0709 23:02:53.695902 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/d326b5d7-6b28-44dc-a0b4-bd4d5b4ab63d-proc\") pod \"perf-node-gather-daemonset-4jcj2\" (UID: \"d326b5d7-6b28-44dc-a0b4-bd4d5b4ab63d\") " pod="openshift-must-gather-5rfjs/perf-node-gather-daemonset-4jcj2" Jul 09 23:02:53.797333 ip-10-0-136-251 kubenswrapper[2579]: I0709 23:02:53.797211 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d326b5d7-6b28-44dc-a0b4-bd4d5b4ab63d-sys\") pod \"perf-node-gather-daemonset-4jcj2\" (UID: \"d326b5d7-6b28-44dc-a0b4-bd4d5b4ab63d\") " pod="openshift-must-gather-5rfjs/perf-node-gather-daemonset-4jcj2" Jul 09 23:02:53.797333 ip-10-0-136-251 kubenswrapper[2579]: I0709 23:02:53.797286 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r9889\" (UniqueName: \"kubernetes.io/projected/d326b5d7-6b28-44dc-a0b4-bd4d5b4ab63d-kube-api-access-r9889\") pod \"perf-node-gather-daemonset-4jcj2\" (UID: \"d326b5d7-6b28-44dc-a0b4-bd4d5b4ab63d\") " pod="openshift-must-gather-5rfjs/perf-node-gather-daemonset-4jcj2" Jul 09 23:02:53.797333 ip-10-0-136-251 kubenswrapper[2579]: I0709 23:02:53.797318 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/d326b5d7-6b28-44dc-a0b4-bd4d5b4ab63d-proc\") pod \"perf-node-gather-daemonset-4jcj2\" (UID: \"d326b5d7-6b28-44dc-a0b4-bd4d5b4ab63d\") " pod="openshift-must-gather-5rfjs/perf-node-gather-daemonset-4jcj2" Jul 09 23:02:53.797648 ip-10-0-136-251 kubenswrapper[2579]: I0709 23:02:53.797345 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d326b5d7-6b28-44dc-a0b4-bd4d5b4ab63d-sys\") pod \"perf-node-gather-daemonset-4jcj2\" (UID: \"d326b5d7-6b28-44dc-a0b4-bd4d5b4ab63d\") " pod="openshift-must-gather-5rfjs/perf-node-gather-daemonset-4jcj2" Jul 09 23:02:53.797648 ip-10-0-136-251 kubenswrapper[2579]: I0709 23:02:53.797420 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/d326b5d7-6b28-44dc-a0b4-bd4d5b4ab63d-proc\") pod \"perf-node-gather-daemonset-4jcj2\" (UID: \"d326b5d7-6b28-44dc-a0b4-bd4d5b4ab63d\") " pod="openshift-must-gather-5rfjs/perf-node-gather-daemonset-4jcj2" Jul 09 23:02:53.797648 ip-10-0-136-251 kubenswrapper[2579]: I0709 23:02:53.797429 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/d326b5d7-6b28-44dc-a0b4-bd4d5b4ab63d-podres\") pod \"perf-node-gather-daemonset-4jcj2\" (UID: \"d326b5d7-6b28-44dc-a0b4-bd4d5b4ab63d\") " pod="openshift-must-gather-5rfjs/perf-node-gather-daemonset-4jcj2" Jul 09 23:02:53.797648 ip-10-0-136-251 kubenswrapper[2579]: I0709 23:02:53.797466 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d326b5d7-6b28-44dc-a0b4-bd4d5b4ab63d-lib-modules\") pod \"perf-node-gather-daemonset-4jcj2\" (UID: \"d326b5d7-6b28-44dc-a0b4-bd4d5b4ab63d\") " pod="openshift-must-gather-5rfjs/perf-node-gather-daemonset-4jcj2" Jul 09 23:02:53.797648 ip-10-0-136-251 kubenswrapper[2579]: I0709 23:02:53.797509 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/d326b5d7-6b28-44dc-a0b4-bd4d5b4ab63d-podres\") pod \"perf-node-gather-daemonset-4jcj2\" (UID: \"d326b5d7-6b28-44dc-a0b4-bd4d5b4ab63d\") " pod="openshift-must-gather-5rfjs/perf-node-gather-daemonset-4jcj2" Jul 09 23:02:53.797648 ip-10-0-136-251 kubenswrapper[2579]: I0709 23:02:53.797614 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d326b5d7-6b28-44dc-a0b4-bd4d5b4ab63d-lib-modules\") pod \"perf-node-gather-daemonset-4jcj2\" (UID: \"d326b5d7-6b28-44dc-a0b4-bd4d5b4ab63d\") " pod="openshift-must-gather-5rfjs/perf-node-gather-daemonset-4jcj2" Jul 09 23:02:53.807162 ip-10-0-136-251 kubenswrapper[2579]: I0709 23:02:53.807123 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r9889\" (UniqueName: \"kubernetes.io/projected/d326b5d7-6b28-44dc-a0b4-bd4d5b4ab63d-kube-api-access-r9889\") pod \"perf-node-gather-daemonset-4jcj2\" (UID: \"d326b5d7-6b28-44dc-a0b4-bd4d5b4ab63d\") " pod="openshift-must-gather-5rfjs/perf-node-gather-daemonset-4jcj2" Jul 09 23:02:53.835987 ip-10-0-136-251 kubenswrapper[2579]: I0709 23:02:53.835950 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-5rfjs/perf-node-gather-daemonset-4jcj2" Jul 09 23:02:53.978576 ip-10-0-136-251 kubenswrapper[2579]: I0709 23:02:53.978548 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-5rfjs/perf-node-gather-daemonset-4jcj2"] Jul 09 23:02:53.981057 ip-10-0-136-251 kubenswrapper[2579]: W0709 23:02:53.981028 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podd326b5d7_6b28_44dc_a0b4_bd4d5b4ab63d.slice/crio-f80807cb6ad242c9b999bb40cd7fd1efaa164e0f6520d7f0e92a562746095f66 WatchSource:0}: Error finding container f80807cb6ad242c9b999bb40cd7fd1efaa164e0f6520d7f0e92a562746095f66: Status 404 returned error can't find the container with id f80807cb6ad242c9b999bb40cd7fd1efaa164e0f6520d7f0e92a562746095f66 Jul 09 23:02:54.149726 ip-10-0-136-251 kubenswrapper[2579]: I0709 23:02:54.149686 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-5rfjs/perf-node-gather-daemonset-4jcj2" event={"ID":"d326b5d7-6b28-44dc-a0b4-bd4d5b4ab63d","Type":"ContainerStarted","Data":"579537523fa4f633e4f729e49e40bf781da9e447963671d95f2634dd3099867d"} Jul 09 23:02:54.149969 ip-10-0-136-251 kubenswrapper[2579]: I0709 23:02:54.149747 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-5rfjs/perf-node-gather-daemonset-4jcj2" event={"ID":"d326b5d7-6b28-44dc-a0b4-bd4d5b4ab63d","Type":"ContainerStarted","Data":"f80807cb6ad242c9b999bb40cd7fd1efaa164e0f6520d7f0e92a562746095f66"} Jul 09 23:02:54.149969 ip-10-0-136-251 kubenswrapper[2579]: I0709 23:02:54.149792 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-5rfjs/perf-node-gather-daemonset-4jcj2" Jul 09 23:02:54.166491 ip-10-0-136-251 kubenswrapper[2579]: I0709 23:02:54.166340 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-5rfjs/perf-node-gather-daemonset-4jcj2" podStartSLOduration=1.166318776 podStartE2EDuration="1.166318776s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 23:02:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 23:02:54.165864411 +0000 UTC m=+3166.077317138" watchObservedRunningTime="2026-07-09 23:02:54.166318776 +0000 UTC m=+3166.077771477" Jul 09 23:02:54.510364 ip-10-0-136-251 kubenswrapper[2579]: I0709 23:02:54.510332 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-6bffdf9778-kggxk_2b7da825-82cf-48d9-a4fb-50c6adbc34e0/networking-console-plugin/0.log" Jul 09 23:02:55.552360 ip-10-0-136-251 kubenswrapper[2579]: I0709 23:02:55.552325 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-685bfb5f64-4jqjt_4d3d7a7e-833c-4557-9833-f03aeec52a95/download-server/0.log" Jul 09 23:02:57.305859 ip-10-0-136-251 kubenswrapper[2579]: I0709 23:02:57.305827 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-vptbn_5a6d8c82-ca6e-49b1-b660-78501b981273/dns/0.log" Jul 09 23:02:57.326121 ip-10-0-136-251 kubenswrapper[2579]: I0709 23:02:57.326081 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-vptbn_5a6d8c82-ca6e-49b1-b660-78501b981273/kube-rbac-proxy/0.log" Jul 09 23:02:57.372455 ip-10-0-136-251 kubenswrapper[2579]: I0709 23:02:57.372406 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-nr4cq_6807e2e6-d9a4-475a-9fb4-4b958127840f/dns-node-resolver/0.log" Jul 09 23:02:57.997574 ip-10-0-136-251 kubenswrapper[2579]: I0709 23:02:57.997542 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-l578c_af0be020-5b8b-4828-bf98-531265c795b2/node-ca/0.log" Jul 09 23:02:58.999083 ip-10-0-136-251 kubenswrapper[2579]: I0709 23:02:58.999039 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-6576859cf4x6nwz_bc747b34-dcf6-43d4-989f-35f49f02b32d/istio-proxy/0.log" Jul 09 23:02:59.285425 ip-10-0-136-251 kubenswrapper[2579]: I0709 23:02:59.285321 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-j9x6z_9a0d3041-24b6-42f9-9992-32cc62e3810e/istio-proxy/0.log" Jul 09 23:02:59.315377 ip-10-0-136-251 kubenswrapper[2579]: I0709 23:02:59.315342 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-j9x6z_9a0d3041-24b6-42f9-9992-32cc62e3810e/istio-proxy/1.log" Jul 09 23:02:59.416340 ip-10-0-136-251 kubenswrapper[2579]: I0709 23:02:59.416308 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-854859444c-bhmbj_31587075-3ee2-43c7-9d0e-5340de9912a1/router/0.log" Jul 09 23:03:00.166015 ip-10-0-136-251 kubenswrapper[2579]: I0709 23:03:00.165989 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-5rfjs/perf-node-gather-daemonset-4jcj2" Jul 09 23:03:00.201817 ip-10-0-136-251 kubenswrapper[2579]: I0709 23:03:00.201780 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-tnwg2_5afda08d-c04b-46ee-b245-6c884c49c78a/serve-healthcheck-canary/0.log" Jul 09 23:03:00.801093 ip-10-0-136-251 kubenswrapper[2579]: I0709 23:03:00.801065 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-7fv2q_c1c63bd0-6360-4f30-9a66-f4255a633916/kube-rbac-proxy/0.log" Jul 09 23:03:00.822855 ip-10-0-136-251 kubenswrapper[2579]: I0709 23:03:00.822829 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-7fv2q_c1c63bd0-6360-4f30-9a66-f4255a633916/exporter/0.log" Jul 09 23:03:00.875038 ip-10-0-136-251 kubenswrapper[2579]: I0709 23:03:00.875012 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-7fv2q_c1c63bd0-6360-4f30-9a66-f4255a633916/extractor/0.log" Jul 09 23:03:03.538420 ip-10-0-136-251 kubenswrapper[2579]: I0709 23:03:03.538393 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-b6tv8_6a02a146-e107-4650-828d-e092ad914d87/manager/0.log" Jul 09 23:03:03.686332 ip-10-0-136-251 kubenswrapper[2579]: I0709 23:03:03.686300 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-btzms_29c83de2-afa8-48fa-b975-93747a089549/manager/0.log" Jul 09 23:03:04.121030 ip-10-0-136-251 kubenswrapper[2579]: I0709 23:03:04.120987 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-5cf875db85-78gjk_0083cb04-279c-4d25-bd33-b639a0b1a0f4/manager/0.log" Jul 09 23:03:04.137810 ip-10-0-136-251 kubenswrapper[2579]: I0709 23:03:04.137783 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-5cf875db85-78gjk_0083cb04-279c-4d25-bd33-b639a0b1a0f4/copy-manifests/0.log" Jul 09 23:03:04.455300 ip-10-0-136-251 kubenswrapper[2579]: I0709 23:03:04.455259 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-6576859cf4x6nwz_bc747b34-dcf6-43d4-989f-35f49f02b32d/istio-proxy/0.log" Jul 09 23:03:04.713252 ip-10-0-136-251 kubenswrapper[2579]: I0709 23:03:04.713207 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-j9x6z_9a0d3041-24b6-42f9-9992-32cc62e3810e/istio-proxy/0.log" Jul 09 23:03:04.745429 ip-10-0-136-251 kubenswrapper[2579]: I0709 23:03:04.745396 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-j9x6z_9a0d3041-24b6-42f9-9992-32cc62e3810e/istio-proxy/1.log" Jul 09 23:03:04.822212 ip-10-0-136-251 kubenswrapper[2579]: I0709 23:03:04.822181 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-854859444c-bhmbj_31587075-3ee2-43c7-9d0e-5340de9912a1/router/0.log" Jul 09 23:03:05.411768 ip-10-0-136-251 kubenswrapper[2579]: I0709 23:03:05.411739 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-76dfc7bb46-sxtvx_7994ef1d-61df-4510-9779-a5b6ff8e723c/manager/0.log" Jul 09 23:03:05.500128 ip-10-0-136-251 kubenswrapper[2579]: I0709 23:03:05.500102 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_openshift-lws-operator-fd8ccff4c-99qx4_b4e0a4e6-0a52-4def-9fde-0d3b5ab3805f/openshift-lws-operator/0.log" Jul 09 23:03:06.084016 ip-10-0-136-251 kubenswrapper[2579]: I0709 23:03:06.083972 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-6576859cf4x6nwz_bc747b34-dcf6-43d4-989f-35f49f02b32d/istio-proxy/0.log" Jul 09 23:03:06.442542 ip-10-0-136-251 kubenswrapper[2579]: I0709 23:03:06.442405 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-j9x6z_9a0d3041-24b6-42f9-9992-32cc62e3810e/istio-proxy/0.log" Jul 09 23:03:06.474095 ip-10-0-136-251 kubenswrapper[2579]: I0709 23:03:06.474062 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-j9x6z_9a0d3041-24b6-42f9-9992-32cc62e3810e/istio-proxy/1.log" Jul 09 23:03:06.679430 ip-10-0-136-251 kubenswrapper[2579]: I0709 23:03:06.679401 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-854859444c-bhmbj_31587075-3ee2-43c7-9d0e-5340de9912a1/router/0.log"