Jul 19 14:57:37.657074 ip-10-0-138-220 systemd[1]: Starting Kubernetes Kubelet... Jul 19 14:57:38.103778 ip-10-0-138-220 kubenswrapper[2558]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 19 14:57:38.103778 ip-10-0-138-220 kubenswrapper[2558]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Jul 19 14:57:38.103778 ip-10-0-138-220 kubenswrapper[2558]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 19 14:57:38.103778 ip-10-0-138-220 kubenswrapper[2558]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Jul 19 14:57:38.103778 ip-10-0-138-220 kubenswrapper[2558]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 19 14:57:38.104911 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:38.104750 2558 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Jul 19 14:57:38.110163 ip-10-0-138-220 kubenswrapper[2558]: W0719 14:57:38.110148 2558 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 19 14:57:38.110163 ip-10-0-138-220 kubenswrapper[2558]: W0719 14:57:38.110164 2558 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 19 14:57:38.110228 ip-10-0-138-220 kubenswrapper[2558]: W0719 14:57:38.110169 2558 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 19 14:57:38.110228 ip-10-0-138-220 kubenswrapper[2558]: W0719 14:57:38.110172 2558 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 19 14:57:38.110228 ip-10-0-138-220 kubenswrapper[2558]: W0719 14:57:38.110176 2558 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 19 14:57:38.110228 ip-10-0-138-220 kubenswrapper[2558]: W0719 14:57:38.110179 2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 19 14:57:38.110228 ip-10-0-138-220 kubenswrapper[2558]: W0719 14:57:38.110181 2558 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 19 14:57:38.110228 ip-10-0-138-220 kubenswrapper[2558]: W0719 14:57:38.110184 2558 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 19 14:57:38.110228 ip-10-0-138-220 kubenswrapper[2558]: W0719 14:57:38.110187 2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 19 14:57:38.110228 ip-10-0-138-220 kubenswrapper[2558]: W0719 14:57:38.110190 2558 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 19 14:57:38.110228 ip-10-0-138-220 kubenswrapper[2558]: W0719 14:57:38.110193 2558 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 19 14:57:38.110228 ip-10-0-138-220 kubenswrapper[2558]: W0719 14:57:38.110197 2558 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 19 14:57:38.110228 ip-10-0-138-220 kubenswrapper[2558]: W0719 14:57:38.110201 2558 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 19 14:57:38.110228 ip-10-0-138-220 kubenswrapper[2558]: W0719 14:57:38.110204 2558 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 19 14:57:38.110228 ip-10-0-138-220 kubenswrapper[2558]: W0719 14:57:38.110206 2558 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 19 14:57:38.110228 ip-10-0-138-220 kubenswrapper[2558]: W0719 14:57:38.110209 2558 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 19 14:57:38.110228 ip-10-0-138-220 kubenswrapper[2558]: W0719 14:57:38.110211 2558 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 19 14:57:38.110228 ip-10-0-138-220 kubenswrapper[2558]: W0719 14:57:38.110215 2558 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 19 14:57:38.110228 ip-10-0-138-220 kubenswrapper[2558]: W0719 14:57:38.110217 2558 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 19 14:57:38.110228 ip-10-0-138-220 kubenswrapper[2558]: W0719 14:57:38.110230 2558 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 19 14:57:38.110228 ip-10-0-138-220 kubenswrapper[2558]: W0719 14:57:38.110233 2558 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 19 14:57:38.110228 ip-10-0-138-220 kubenswrapper[2558]: W0719 14:57:38.110236 2558 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 19 14:57:38.110694 ip-10-0-138-220 kubenswrapper[2558]: W0719 14:57:38.110239 2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 19 14:57:38.110694 ip-10-0-138-220 kubenswrapper[2558]: W0719 14:57:38.110242 2558 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 19 14:57:38.110694 ip-10-0-138-220 kubenswrapper[2558]: W0719 14:57:38.110244 2558 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 19 14:57:38.110694 ip-10-0-138-220 kubenswrapper[2558]: W0719 14:57:38.110248 2558 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 19 14:57:38.110694 ip-10-0-138-220 kubenswrapper[2558]: W0719 14:57:38.110250 2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 19 14:57:38.110694 ip-10-0-138-220 kubenswrapper[2558]: W0719 14:57:38.110253 2558 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 19 14:57:38.110694 ip-10-0-138-220 kubenswrapper[2558]: W0719 14:57:38.110255 2558 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 19 14:57:38.110694 ip-10-0-138-220 kubenswrapper[2558]: W0719 14:57:38.110258 2558 feature_gate.go:328] unrecognized feature gate: Example2 Jul 19 14:57:38.110694 ip-10-0-138-220 kubenswrapper[2558]: W0719 14:57:38.110260 2558 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 19 14:57:38.110694 ip-10-0-138-220 kubenswrapper[2558]: W0719 14:57:38.110263 2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 19 14:57:38.110694 ip-10-0-138-220 kubenswrapper[2558]: W0719 14:57:38.110266 2558 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 19 14:57:38.110694 ip-10-0-138-220 kubenswrapper[2558]: W0719 14:57:38.110268 2558 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 19 14:57:38.110694 ip-10-0-138-220 kubenswrapper[2558]: W0719 14:57:38.110271 2558 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 19 14:57:38.110694 ip-10-0-138-220 kubenswrapper[2558]: W0719 14:57:38.110273 2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 19 14:57:38.110694 ip-10-0-138-220 kubenswrapper[2558]: W0719 14:57:38.110276 2558 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 19 14:57:38.110694 ip-10-0-138-220 kubenswrapper[2558]: W0719 14:57:38.110278 2558 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 19 14:57:38.110694 ip-10-0-138-220 kubenswrapper[2558]: W0719 14:57:38.110281 2558 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 19 14:57:38.110694 ip-10-0-138-220 kubenswrapper[2558]: W0719 14:57:38.110283 2558 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 19 14:57:38.110694 ip-10-0-138-220 kubenswrapper[2558]: W0719 14:57:38.110285 2558 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 19 14:57:38.110694 ip-10-0-138-220 kubenswrapper[2558]: W0719 14:57:38.110288 2558 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 19 14:57:38.111318 ip-10-0-138-220 kubenswrapper[2558]: W0719 14:57:38.110291 2558 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 19 14:57:38.111318 ip-10-0-138-220 kubenswrapper[2558]: W0719 14:57:38.110293 2558 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 19 14:57:38.111318 ip-10-0-138-220 kubenswrapper[2558]: W0719 14:57:38.110295 2558 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 19 14:57:38.111318 ip-10-0-138-220 kubenswrapper[2558]: W0719 14:57:38.110298 2558 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 19 14:57:38.111318 ip-10-0-138-220 kubenswrapper[2558]: W0719 14:57:38.110300 2558 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 19 14:57:38.111318 ip-10-0-138-220 kubenswrapper[2558]: W0719 14:57:38.110302 2558 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 19 14:57:38.111318 ip-10-0-138-220 kubenswrapper[2558]: W0719 14:57:38.110305 2558 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 19 14:57:38.111318 ip-10-0-138-220 kubenswrapper[2558]: W0719 14:57:38.110307 2558 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 19 14:57:38.111318 ip-10-0-138-220 kubenswrapper[2558]: W0719 14:57:38.110310 2558 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 19 14:57:38.111318 ip-10-0-138-220 kubenswrapper[2558]: W0719 14:57:38.110312 2558 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 19 14:57:38.111318 ip-10-0-138-220 kubenswrapper[2558]: W0719 14:57:38.110321 2558 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 19 14:57:38.111318 ip-10-0-138-220 kubenswrapper[2558]: W0719 14:57:38.110324 2558 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 19 14:57:38.111318 ip-10-0-138-220 kubenswrapper[2558]: W0719 14:57:38.110327 2558 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 19 14:57:38.111318 ip-10-0-138-220 kubenswrapper[2558]: W0719 14:57:38.110330 2558 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 19 14:57:38.111318 ip-10-0-138-220 kubenswrapper[2558]: W0719 14:57:38.110333 2558 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 19 14:57:38.111318 ip-10-0-138-220 kubenswrapper[2558]: W0719 14:57:38.110335 2558 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 19 14:57:38.111318 ip-10-0-138-220 kubenswrapper[2558]: W0719 14:57:38.110338 2558 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 19 14:57:38.111318 ip-10-0-138-220 kubenswrapper[2558]: W0719 14:57:38.110340 2558 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 19 14:57:38.111318 ip-10-0-138-220 kubenswrapper[2558]: W0719 14:57:38.110343 2558 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 19 14:57:38.111318 ip-10-0-138-220 kubenswrapper[2558]: W0719 14:57:38.110346 2558 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 19 14:57:38.111812 ip-10-0-138-220 kubenswrapper[2558]: W0719 14:57:38.110348 2558 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 19 14:57:38.111812 ip-10-0-138-220 kubenswrapper[2558]: W0719 14:57:38.110351 2558 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 19 14:57:38.111812 ip-10-0-138-220 kubenswrapper[2558]: W0719 14:57:38.110353 2558 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 19 14:57:38.111812 ip-10-0-138-220 kubenswrapper[2558]: W0719 14:57:38.110356 2558 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 19 14:57:38.111812 ip-10-0-138-220 kubenswrapper[2558]: W0719 14:57:38.110358 2558 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 19 14:57:38.111812 ip-10-0-138-220 kubenswrapper[2558]: W0719 14:57:38.110360 2558 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 19 14:57:38.111812 ip-10-0-138-220 kubenswrapper[2558]: W0719 14:57:38.110363 2558 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 19 14:57:38.111812 ip-10-0-138-220 kubenswrapper[2558]: W0719 14:57:38.110366 2558 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 19 14:57:38.111812 ip-10-0-138-220 kubenswrapper[2558]: W0719 14:57:38.110369 2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 19 14:57:38.111812 ip-10-0-138-220 kubenswrapper[2558]: W0719 14:57:38.110371 2558 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 19 14:57:38.111812 ip-10-0-138-220 kubenswrapper[2558]: W0719 14:57:38.110374 2558 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 19 14:57:38.111812 ip-10-0-138-220 kubenswrapper[2558]: W0719 14:57:38.110376 2558 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 19 14:57:38.111812 ip-10-0-138-220 kubenswrapper[2558]: W0719 14:57:38.110379 2558 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 19 14:57:38.111812 ip-10-0-138-220 kubenswrapper[2558]: W0719 14:57:38.110381 2558 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 19 14:57:38.111812 ip-10-0-138-220 kubenswrapper[2558]: W0719 14:57:38.110384 2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 19 14:57:38.111812 ip-10-0-138-220 kubenswrapper[2558]: W0719 14:57:38.110386 2558 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 19 14:57:38.111812 ip-10-0-138-220 kubenswrapper[2558]: W0719 14:57:38.110389 2558 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 19 14:57:38.111812 ip-10-0-138-220 kubenswrapper[2558]: W0719 14:57:38.110391 2558 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 19 14:57:38.111812 ip-10-0-138-220 kubenswrapper[2558]: W0719 14:57:38.110395 2558 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 19 14:57:38.112266 ip-10-0-138-220 kubenswrapper[2558]: W0719 14:57:38.110398 2558 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 19 14:57:38.112266 ip-10-0-138-220 kubenswrapper[2558]: W0719 14:57:38.110401 2558 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 19 14:57:38.112266 ip-10-0-138-220 kubenswrapper[2558]: W0719 14:57:38.110404 2558 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 19 14:57:38.112266 ip-10-0-138-220 kubenswrapper[2558]: W0719 14:57:38.110407 2558 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 19 14:57:38.112266 ip-10-0-138-220 kubenswrapper[2558]: W0719 14:57:38.110410 2558 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 19 14:57:38.112266 ip-10-0-138-220 kubenswrapper[2558]: W0719 14:57:38.110413 2558 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 19 14:57:38.112266 ip-10-0-138-220 kubenswrapper[2558]: W0719 14:57:38.110416 2558 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 19 14:57:38.112266 ip-10-0-138-220 kubenswrapper[2558]: W0719 14:57:38.110419 2558 feature_gate.go:328] unrecognized feature gate: Example Jul 19 14:57:38.112266 ip-10-0-138-220 kubenswrapper[2558]: W0719 14:57:38.110422 2558 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 19 14:57:38.112266 ip-10-0-138-220 kubenswrapper[2558]: W0719 14:57:38.110424 2558 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 19 14:57:38.112266 ip-10-0-138-220 kubenswrapper[2558]: W0719 14:57:38.110427 2558 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 19 14:57:38.112266 ip-10-0-138-220 kubenswrapper[2558]: W0719 14:57:38.110429 2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 19 14:57:38.112266 ip-10-0-138-220 kubenswrapper[2558]: W0719 14:57:38.110432 2558 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 19 14:57:38.112266 ip-10-0-138-220 kubenswrapper[2558]: W0719 14:57:38.110434 2558 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 19 14:57:38.112266 ip-10-0-138-220 kubenswrapper[2558]: W0719 14:57:38.110437 2558 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 19 14:57:38.112266 ip-10-0-138-220 kubenswrapper[2558]: W0719 14:57:38.110439 2558 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 19 14:57:38.112266 ip-10-0-138-220 kubenswrapper[2558]: W0719 14:57:38.110441 2558 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 19 14:57:38.112266 ip-10-0-138-220 kubenswrapper[2558]: W0719 14:57:38.110444 2558 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 19 14:57:38.113298 ip-10-0-138-220 kubenswrapper[2558]: W0719 14:57:38.113285 2558 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 19 14:57:38.113298 ip-10-0-138-220 kubenswrapper[2558]: W0719 14:57:38.113298 2558 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 19 14:57:38.113356 ip-10-0-138-220 kubenswrapper[2558]: W0719 14:57:38.113301 2558 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 19 14:57:38.113356 ip-10-0-138-220 kubenswrapper[2558]: W0719 14:57:38.113306 2558 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 19 14:57:38.113356 ip-10-0-138-220 kubenswrapper[2558]: W0719 14:57:38.113310 2558 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 19 14:57:38.113356 ip-10-0-138-220 kubenswrapper[2558]: W0719 14:57:38.113313 2558 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 19 14:57:38.113356 ip-10-0-138-220 kubenswrapper[2558]: W0719 14:57:38.113316 2558 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 19 14:57:38.113356 ip-10-0-138-220 kubenswrapper[2558]: W0719 14:57:38.113319 2558 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 19 14:57:38.113356 ip-10-0-138-220 kubenswrapper[2558]: W0719 14:57:38.113322 2558 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 19 14:57:38.113356 ip-10-0-138-220 kubenswrapper[2558]: W0719 14:57:38.113324 2558 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 19 14:57:38.113356 ip-10-0-138-220 kubenswrapper[2558]: W0719 14:57:38.113327 2558 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 19 14:57:38.113356 ip-10-0-138-220 kubenswrapper[2558]: W0719 14:57:38.113330 2558 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 19 14:57:38.113356 ip-10-0-138-220 kubenswrapper[2558]: W0719 14:57:38.113332 2558 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 19 14:57:38.113356 ip-10-0-138-220 kubenswrapper[2558]: W0719 14:57:38.113335 2558 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 19 14:57:38.113356 ip-10-0-138-220 kubenswrapper[2558]: W0719 14:57:38.113337 2558 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 19 14:57:38.113356 ip-10-0-138-220 kubenswrapper[2558]: W0719 14:57:38.113340 2558 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 19 14:57:38.113356 ip-10-0-138-220 kubenswrapper[2558]: W0719 14:57:38.113343 2558 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 19 14:57:38.113356 ip-10-0-138-220 kubenswrapper[2558]: W0719 14:57:38.113346 2558 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 19 14:57:38.113356 ip-10-0-138-220 kubenswrapper[2558]: W0719 14:57:38.113348 2558 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 19 14:57:38.113356 ip-10-0-138-220 kubenswrapper[2558]: W0719 14:57:38.113351 2558 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 19 14:57:38.113356 ip-10-0-138-220 kubenswrapper[2558]: W0719 14:57:38.113354 2558 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 19 14:57:38.113835 ip-10-0-138-220 kubenswrapper[2558]: W0719 14:57:38.113357 2558 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 19 14:57:38.113835 ip-10-0-138-220 kubenswrapper[2558]: W0719 14:57:38.113360 2558 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 19 14:57:38.113835 ip-10-0-138-220 kubenswrapper[2558]: W0719 14:57:38.113363 2558 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 19 14:57:38.113835 ip-10-0-138-220 kubenswrapper[2558]: W0719 14:57:38.113365 2558 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 19 14:57:38.113835 ip-10-0-138-220 kubenswrapper[2558]: W0719 14:57:38.113368 2558 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 19 14:57:38.113835 ip-10-0-138-220 kubenswrapper[2558]: W0719 14:57:38.113371 2558 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 19 14:57:38.113835 ip-10-0-138-220 kubenswrapper[2558]: W0719 14:57:38.113373 2558 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 19 14:57:38.113835 ip-10-0-138-220 kubenswrapper[2558]: W0719 14:57:38.113375 2558 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 19 14:57:38.113835 ip-10-0-138-220 kubenswrapper[2558]: W0719 14:57:38.113378 2558 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 19 14:57:38.113835 ip-10-0-138-220 kubenswrapper[2558]: W0719 14:57:38.113380 2558 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 19 14:57:38.113835 ip-10-0-138-220 kubenswrapper[2558]: W0719 14:57:38.113383 2558 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 19 14:57:38.113835 ip-10-0-138-220 kubenswrapper[2558]: W0719 14:57:38.113385 2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 19 14:57:38.113835 ip-10-0-138-220 kubenswrapper[2558]: W0719 14:57:38.113387 2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 19 14:57:38.113835 ip-10-0-138-220 kubenswrapper[2558]: W0719 14:57:38.113390 2558 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 19 14:57:38.113835 ip-10-0-138-220 kubenswrapper[2558]: W0719 14:57:38.113392 2558 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 19 14:57:38.113835 ip-10-0-138-220 kubenswrapper[2558]: W0719 14:57:38.113395 2558 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 19 14:57:38.113835 ip-10-0-138-220 kubenswrapper[2558]: W0719 14:57:38.113397 2558 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 19 14:57:38.113835 ip-10-0-138-220 kubenswrapper[2558]: W0719 14:57:38.113400 2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 19 14:57:38.113835 ip-10-0-138-220 kubenswrapper[2558]: W0719 14:57:38.113402 2558 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 19 14:57:38.113835 ip-10-0-138-220 kubenswrapper[2558]: W0719 14:57:38.113405 2558 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 19 14:57:38.114326 ip-10-0-138-220 kubenswrapper[2558]: W0719 14:57:38.113408 2558 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 19 14:57:38.114326 ip-10-0-138-220 kubenswrapper[2558]: W0719 14:57:38.113410 2558 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 19 14:57:38.114326 ip-10-0-138-220 kubenswrapper[2558]: W0719 14:57:38.113413 2558 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 19 14:57:38.114326 ip-10-0-138-220 kubenswrapper[2558]: W0719 14:57:38.113416 2558 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 19 14:57:38.114326 ip-10-0-138-220 kubenswrapper[2558]: W0719 14:57:38.113419 2558 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 19 14:57:38.114326 ip-10-0-138-220 kubenswrapper[2558]: W0719 14:57:38.113422 2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 19 14:57:38.114326 ip-10-0-138-220 kubenswrapper[2558]: W0719 14:57:38.113424 2558 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 19 14:57:38.114326 ip-10-0-138-220 kubenswrapper[2558]: W0719 14:57:38.113427 2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 19 14:57:38.114326 ip-10-0-138-220 kubenswrapper[2558]: W0719 14:57:38.113429 2558 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 19 14:57:38.114326 ip-10-0-138-220 kubenswrapper[2558]: W0719 14:57:38.113432 2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 19 14:57:38.114326 ip-10-0-138-220 kubenswrapper[2558]: W0719 14:57:38.113434 2558 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 19 14:57:38.114326 ip-10-0-138-220 kubenswrapper[2558]: W0719 14:57:38.113437 2558 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 19 14:57:38.114326 ip-10-0-138-220 kubenswrapper[2558]: W0719 14:57:38.113440 2558 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 19 14:57:38.114326 ip-10-0-138-220 kubenswrapper[2558]: W0719 14:57:38.113443 2558 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 19 14:57:38.114326 ip-10-0-138-220 kubenswrapper[2558]: W0719 14:57:38.113445 2558 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 19 14:57:38.114326 ip-10-0-138-220 kubenswrapper[2558]: W0719 14:57:38.113448 2558 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 19 14:57:38.114326 ip-10-0-138-220 kubenswrapper[2558]: W0719 14:57:38.113451 2558 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 19 14:57:38.114326 ip-10-0-138-220 kubenswrapper[2558]: W0719 14:57:38.113453 2558 feature_gate.go:328] unrecognized feature gate: Example Jul 19 14:57:38.114326 ip-10-0-138-220 kubenswrapper[2558]: W0719 14:57:38.113456 2558 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 19 14:57:38.114326 ip-10-0-138-220 kubenswrapper[2558]: W0719 14:57:38.113458 2558 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 19 14:57:38.114830 ip-10-0-138-220 kubenswrapper[2558]: W0719 14:57:38.113460 2558 feature_gate.go:328] unrecognized feature gate: Example2 Jul 19 14:57:38.114830 ip-10-0-138-220 kubenswrapper[2558]: W0719 14:57:38.113463 2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 19 14:57:38.114830 ip-10-0-138-220 kubenswrapper[2558]: W0719 14:57:38.113465 2558 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 19 14:57:38.114830 ip-10-0-138-220 kubenswrapper[2558]: W0719 14:57:38.113468 2558 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 19 14:57:38.114830 ip-10-0-138-220 kubenswrapper[2558]: W0719 14:57:38.113470 2558 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 19 14:57:38.114830 ip-10-0-138-220 kubenswrapper[2558]: W0719 14:57:38.113473 2558 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 19 14:57:38.114830 ip-10-0-138-220 kubenswrapper[2558]: W0719 14:57:38.113475 2558 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 19 14:57:38.114830 ip-10-0-138-220 kubenswrapper[2558]: W0719 14:57:38.113478 2558 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 19 14:57:38.114830 ip-10-0-138-220 kubenswrapper[2558]: W0719 14:57:38.113481 2558 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 19 14:57:38.114830 ip-10-0-138-220 kubenswrapper[2558]: W0719 14:57:38.113483 2558 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 19 14:57:38.114830 ip-10-0-138-220 kubenswrapper[2558]: W0719 14:57:38.113486 2558 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 19 14:57:38.114830 ip-10-0-138-220 kubenswrapper[2558]: W0719 14:57:38.113488 2558 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 19 14:57:38.114830 ip-10-0-138-220 kubenswrapper[2558]: W0719 14:57:38.113492 2558 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 19 14:57:38.114830 ip-10-0-138-220 kubenswrapper[2558]: W0719 14:57:38.113495 2558 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 19 14:57:38.114830 ip-10-0-138-220 kubenswrapper[2558]: W0719 14:57:38.113498 2558 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 19 14:57:38.114830 ip-10-0-138-220 kubenswrapper[2558]: W0719 14:57:38.113501 2558 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 19 14:57:38.114830 ip-10-0-138-220 kubenswrapper[2558]: W0719 14:57:38.113504 2558 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 19 14:57:38.114830 ip-10-0-138-220 kubenswrapper[2558]: W0719 14:57:38.113507 2558 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 19 14:57:38.114830 ip-10-0-138-220 kubenswrapper[2558]: W0719 14:57:38.113510 2558 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 19 14:57:38.115309 ip-10-0-138-220 kubenswrapper[2558]: W0719 14:57:38.113513 2558 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 19 14:57:38.115309 ip-10-0-138-220 kubenswrapper[2558]: W0719 14:57:38.113515 2558 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 19 14:57:38.115309 ip-10-0-138-220 kubenswrapper[2558]: W0719 14:57:38.113518 2558 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 19 14:57:38.115309 ip-10-0-138-220 kubenswrapper[2558]: W0719 14:57:38.113520 2558 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 19 14:57:38.115309 ip-10-0-138-220 kubenswrapper[2558]: W0719 14:57:38.113523 2558 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 19 14:57:38.115309 ip-10-0-138-220 kubenswrapper[2558]: W0719 14:57:38.113527 2558 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 19 14:57:38.115309 ip-10-0-138-220 kubenswrapper[2558]: W0719 14:57:38.113530 2558 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 19 14:57:38.115309 ip-10-0-138-220 kubenswrapper[2558]: W0719 14:57:38.113533 2558 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 19 14:57:38.115309 ip-10-0-138-220 kubenswrapper[2558]: W0719 14:57:38.113535 2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 19 14:57:38.115309 ip-10-0-138-220 kubenswrapper[2558]: W0719 14:57:38.113538 2558 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 19 14:57:38.115309 ip-10-0-138-220 kubenswrapper[2558]: W0719 14:57:38.113540 2558 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 19 14:57:38.115309 ip-10-0-138-220 kubenswrapper[2558]: W0719 14:57:38.113543 2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 19 14:57:38.115309 ip-10-0-138-220 kubenswrapper[2558]: W0719 14:57:38.113545 2558 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 19 14:57:38.115309 ip-10-0-138-220 kubenswrapper[2558]: W0719 14:57:38.113548 2558 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 19 14:57:38.115309 ip-10-0-138-220 kubenswrapper[2558]: W0719 14:57:38.113550 2558 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 19 14:57:38.115309 ip-10-0-138-220 kubenswrapper[2558]: W0719 14:57:38.113553 2558 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 19 14:57:38.115309 ip-10-0-138-220 kubenswrapper[2558]: W0719 14:57:38.113555 2558 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 19 14:57:38.115309 ip-10-0-138-220 kubenswrapper[2558]: W0719 14:57:38.113558 2558 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 19 14:57:38.115309 ip-10-0-138-220 kubenswrapper[2558]: W0719 14:57:38.113560 2558 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 19 14:57:38.115309 ip-10-0-138-220 kubenswrapper[2558]: W0719 14:57:38.114664 2558 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 19 14:57:38.115800 ip-10-0-138-220 kubenswrapper[2558]: W0719 14:57:38.114671 2558 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 19 14:57:38.115800 ip-10-0-138-220 kubenswrapper[2558]: W0719 14:57:38.114674 2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 19 14:57:38.115800 ip-10-0-138-220 kubenswrapper[2558]: W0719 14:57:38.114677 2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 19 14:57:38.115800 ip-10-0-138-220 kubenswrapper[2558]: W0719 14:57:38.114680 2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 19 14:57:38.115800 ip-10-0-138-220 kubenswrapper[2558]: W0719 14:57:38.114683 2558 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 19 14:57:38.115800 ip-10-0-138-220 kubenswrapper[2558]: W0719 14:57:38.114686 2558 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 19 14:57:38.115800 ip-10-0-138-220 kubenswrapper[2558]: W0719 14:57:38.114689 2558 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 19 14:57:38.115800 ip-10-0-138-220 kubenswrapper[2558]: W0719 14:57:38.114692 2558 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 19 14:57:38.115800 ip-10-0-138-220 kubenswrapper[2558]: W0719 14:57:38.114694 2558 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 19 14:57:38.115800 ip-10-0-138-220 kubenswrapper[2558]: W0719 14:57:38.114699 2558 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 19 14:57:38.115800 ip-10-0-138-220 kubenswrapper[2558]: W0719 14:57:38.114701 2558 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 19 14:57:38.115800 ip-10-0-138-220 kubenswrapper[2558]: W0719 14:57:38.114704 2558 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 19 14:57:38.115800 ip-10-0-138-220 kubenswrapper[2558]: W0719 14:57:38.114707 2558 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 19 14:57:38.115800 ip-10-0-138-220 kubenswrapper[2558]: W0719 14:57:38.114709 2558 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 19 14:57:38.115800 ip-10-0-138-220 kubenswrapper[2558]: W0719 14:57:38.114712 2558 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 19 14:57:38.115800 ip-10-0-138-220 kubenswrapper[2558]: W0719 14:57:38.114715 2558 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 19 14:57:38.115800 ip-10-0-138-220 kubenswrapper[2558]: W0719 14:57:38.114717 2558 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 19 14:57:38.115800 ip-10-0-138-220 kubenswrapper[2558]: W0719 14:57:38.114720 2558 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 19 14:57:38.115800 ip-10-0-138-220 kubenswrapper[2558]: W0719 14:57:38.114724 2558 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 19 14:57:38.115800 ip-10-0-138-220 kubenswrapper[2558]: W0719 14:57:38.114727 2558 feature_gate.go:328] unrecognized feature gate: Example Jul 19 14:57:38.116308 ip-10-0-138-220 kubenswrapper[2558]: W0719 14:57:38.114729 2558 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 19 14:57:38.116308 ip-10-0-138-220 kubenswrapper[2558]: W0719 14:57:38.114732 2558 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 19 14:57:38.116308 ip-10-0-138-220 kubenswrapper[2558]: W0719 14:57:38.114734 2558 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 19 14:57:38.116308 ip-10-0-138-220 kubenswrapper[2558]: W0719 14:57:38.114737 2558 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 19 14:57:38.116308 ip-10-0-138-220 kubenswrapper[2558]: W0719 14:57:38.114740 2558 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 19 14:57:38.116308 ip-10-0-138-220 kubenswrapper[2558]: W0719 14:57:38.114742 2558 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 19 14:57:38.116308 ip-10-0-138-220 kubenswrapper[2558]: W0719 14:57:38.114745 2558 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 19 14:57:38.116308 ip-10-0-138-220 kubenswrapper[2558]: W0719 14:57:38.114747 2558 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 19 14:57:38.116308 ip-10-0-138-220 kubenswrapper[2558]: W0719 14:57:38.114750 2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 19 14:57:38.116308 ip-10-0-138-220 kubenswrapper[2558]: W0719 14:57:38.114752 2558 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 19 14:57:38.116308 ip-10-0-138-220 kubenswrapper[2558]: W0719 14:57:38.114755 2558 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 19 14:57:38.116308 ip-10-0-138-220 kubenswrapper[2558]: W0719 14:57:38.114757 2558 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 19 14:57:38.116308 ip-10-0-138-220 kubenswrapper[2558]: W0719 14:57:38.114760 2558 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 19 14:57:38.116308 ip-10-0-138-220 kubenswrapper[2558]: W0719 14:57:38.114763 2558 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 19 14:57:38.116308 ip-10-0-138-220 kubenswrapper[2558]: W0719 14:57:38.114765 2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 19 14:57:38.116308 ip-10-0-138-220 kubenswrapper[2558]: W0719 14:57:38.114768 2558 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 19 14:57:38.116308 ip-10-0-138-220 kubenswrapper[2558]: W0719 14:57:38.114771 2558 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 19 14:57:38.116308 ip-10-0-138-220 kubenswrapper[2558]: W0719 14:57:38.114773 2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 19 14:57:38.116308 ip-10-0-138-220 kubenswrapper[2558]: W0719 14:57:38.114775 2558 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 19 14:57:38.116308 ip-10-0-138-220 kubenswrapper[2558]: W0719 14:57:38.114778 2558 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 19 14:57:38.116796 ip-10-0-138-220 kubenswrapper[2558]: W0719 14:57:38.114782 2558 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 19 14:57:38.116796 ip-10-0-138-220 kubenswrapper[2558]: W0719 14:57:38.114803 2558 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 19 14:57:38.116796 ip-10-0-138-220 kubenswrapper[2558]: W0719 14:57:38.114807 2558 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 19 14:57:38.116796 ip-10-0-138-220 kubenswrapper[2558]: W0719 14:57:38.114810 2558 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 19 14:57:38.116796 ip-10-0-138-220 kubenswrapper[2558]: W0719 14:57:38.114813 2558 feature_gate.go:328] unrecognized feature gate: Example2 Jul 19 14:57:38.116796 ip-10-0-138-220 kubenswrapper[2558]: W0719 14:57:38.114816 2558 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 19 14:57:38.116796 ip-10-0-138-220 kubenswrapper[2558]: W0719 14:57:38.114819 2558 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 19 14:57:38.116796 ip-10-0-138-220 kubenswrapper[2558]: W0719 14:57:38.114821 2558 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 19 14:57:38.116796 ip-10-0-138-220 kubenswrapper[2558]: W0719 14:57:38.114824 2558 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 19 14:57:38.116796 ip-10-0-138-220 kubenswrapper[2558]: W0719 14:57:38.114826 2558 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 19 14:57:38.116796 ip-10-0-138-220 kubenswrapper[2558]: W0719 14:57:38.114829 2558 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 19 14:57:38.116796 ip-10-0-138-220 kubenswrapper[2558]: W0719 14:57:38.114832 2558 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 19 14:57:38.116796 ip-10-0-138-220 kubenswrapper[2558]: W0719 14:57:38.114835 2558 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 19 14:57:38.116796 ip-10-0-138-220 kubenswrapper[2558]: W0719 14:57:38.114837 2558 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 19 14:57:38.116796 ip-10-0-138-220 kubenswrapper[2558]: W0719 14:57:38.114840 2558 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 19 14:57:38.116796 ip-10-0-138-220 kubenswrapper[2558]: W0719 14:57:38.114843 2558 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 19 14:57:38.116796 ip-10-0-138-220 kubenswrapper[2558]: W0719 14:57:38.114845 2558 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 19 14:57:38.116796 ip-10-0-138-220 kubenswrapper[2558]: W0719 14:57:38.114848 2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 19 14:57:38.116796 ip-10-0-138-220 kubenswrapper[2558]: W0719 14:57:38.114850 2558 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 19 14:57:38.117291 ip-10-0-138-220 kubenswrapper[2558]: W0719 14:57:38.114853 2558 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 19 14:57:38.117291 ip-10-0-138-220 kubenswrapper[2558]: W0719 14:57:38.114855 2558 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 19 14:57:38.117291 ip-10-0-138-220 kubenswrapper[2558]: W0719 14:57:38.114858 2558 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 19 14:57:38.117291 ip-10-0-138-220 kubenswrapper[2558]: W0719 14:57:38.114860 2558 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 19 14:57:38.117291 ip-10-0-138-220 kubenswrapper[2558]: W0719 14:57:38.114863 2558 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 19 14:57:38.117291 ip-10-0-138-220 kubenswrapper[2558]: W0719 14:57:38.114866 2558 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 19 14:57:38.117291 ip-10-0-138-220 kubenswrapper[2558]: W0719 14:57:38.114868 2558 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 19 14:57:38.117291 ip-10-0-138-220 kubenswrapper[2558]: W0719 14:57:38.114871 2558 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 19 14:57:38.117291 ip-10-0-138-220 kubenswrapper[2558]: W0719 14:57:38.114874 2558 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 19 14:57:38.117291 ip-10-0-138-220 kubenswrapper[2558]: W0719 14:57:38.114877 2558 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 19 14:57:38.117291 ip-10-0-138-220 kubenswrapper[2558]: W0719 14:57:38.114881 2558 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 19 14:57:38.117291 ip-10-0-138-220 kubenswrapper[2558]: W0719 14:57:38.114883 2558 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 19 14:57:38.117291 ip-10-0-138-220 kubenswrapper[2558]: W0719 14:57:38.114886 2558 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 19 14:57:38.117291 ip-10-0-138-220 kubenswrapper[2558]: W0719 14:57:38.114889 2558 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 19 14:57:38.117291 ip-10-0-138-220 kubenswrapper[2558]: W0719 14:57:38.114891 2558 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 19 14:57:38.117291 ip-10-0-138-220 kubenswrapper[2558]: W0719 14:57:38.114894 2558 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 19 14:57:38.117291 ip-10-0-138-220 kubenswrapper[2558]: W0719 14:57:38.114897 2558 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 19 14:57:38.117291 ip-10-0-138-220 kubenswrapper[2558]: W0719 14:57:38.114899 2558 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 19 14:57:38.117291 ip-10-0-138-220 kubenswrapper[2558]: W0719 14:57:38.114902 2558 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 19 14:57:38.117751 ip-10-0-138-220 kubenswrapper[2558]: W0719 14:57:38.114904 2558 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 19 14:57:38.117751 ip-10-0-138-220 kubenswrapper[2558]: W0719 14:57:38.114907 2558 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 19 14:57:38.117751 ip-10-0-138-220 kubenswrapper[2558]: W0719 14:57:38.114909 2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 19 14:57:38.117751 ip-10-0-138-220 kubenswrapper[2558]: W0719 14:57:38.114911 2558 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 19 14:57:38.117751 ip-10-0-138-220 kubenswrapper[2558]: W0719 14:57:38.114914 2558 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 19 14:57:38.117751 ip-10-0-138-220 kubenswrapper[2558]: W0719 14:57:38.114917 2558 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 19 14:57:38.117751 ip-10-0-138-220 kubenswrapper[2558]: W0719 14:57:38.114920 2558 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 19 14:57:38.117751 ip-10-0-138-220 kubenswrapper[2558]: W0719 14:57:38.114923 2558 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 19 14:57:38.117751 ip-10-0-138-220 kubenswrapper[2558]: W0719 14:57:38.114925 2558 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 19 14:57:38.117751 ip-10-0-138-220 kubenswrapper[2558]: W0719 14:57:38.114928 2558 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 19 14:57:38.117751 ip-10-0-138-220 kubenswrapper[2558]: W0719 14:57:38.114930 2558 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 19 14:57:38.117751 ip-10-0-138-220 kubenswrapper[2558]: W0719 14:57:38.114933 2558 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 19 14:57:38.117751 ip-10-0-138-220 kubenswrapper[2558]: W0719 14:57:38.114935 2558 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 19 14:57:38.117751 ip-10-0-138-220 kubenswrapper[2558]: W0719 14:57:38.114938 2558 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 19 14:57:38.117751 ip-10-0-138-220 kubenswrapper[2558]: W0719 14:57:38.114940 2558 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 19 14:57:38.117751 ip-10-0-138-220 kubenswrapper[2558]: W0719 14:57:38.114943 2558 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 19 14:57:38.117751 ip-10-0-138-220 kubenswrapper[2558]: W0719 14:57:38.114945 2558 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 19 14:57:38.117751 ip-10-0-138-220 kubenswrapper[2558]: W0719 14:57:38.114947 2558 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 19 14:57:38.117751 ip-10-0-138-220 kubenswrapper[2558]: W0719 14:57:38.114950 2558 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 19 14:57:38.117751 ip-10-0-138-220 kubenswrapper[2558]: W0719 14:57:38.114952 2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 19 14:57:38.118255 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:38.115014 2558 flags.go:64] FLAG: --address="0.0.0.0" Jul 19 14:57:38.118255 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:38.115026 2558 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Jul 19 14:57:38.118255 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:38.115033 2558 flags.go:64] FLAG: --anonymous-auth="true" Jul 19 14:57:38.118255 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:38.115037 2558 flags.go:64] FLAG: --application-metrics-count-limit="100" Jul 19 14:57:38.118255 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:38.115040 2558 flags.go:64] FLAG: --authentication-token-webhook="false" Jul 19 14:57:38.118255 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:38.115044 2558 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Jul 19 14:57:38.118255 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:38.115048 2558 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Jul 19 14:57:38.118255 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:38.115053 2558 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Jul 19 14:57:38.118255 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:38.115056 2558 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Jul 19 14:57:38.118255 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:38.115059 2558 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Jul 19 14:57:38.118255 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:38.115063 2558 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Jul 19 14:57:38.118255 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:38.115066 2558 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Jul 19 14:57:38.118255 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:38.115069 2558 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Jul 19 14:57:38.118255 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:38.115072 2558 flags.go:64] FLAG: --cgroup-root="" Jul 19 14:57:38.118255 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:38.115075 2558 flags.go:64] FLAG: --cgroups-per-qos="true" Jul 19 14:57:38.118255 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:38.115078 2558 flags.go:64] FLAG: --client-ca-file="" Jul 19 14:57:38.118255 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:38.115081 2558 flags.go:64] FLAG: --cloud-provider="external" Jul 19 14:57:38.118255 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:38.115084 2558 flags.go:64] FLAG: --cluster-dns="[]" Jul 19 14:57:38.118255 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:38.115089 2558 flags.go:64] FLAG: --cluster-domain="" Jul 19 14:57:38.118255 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:38.115091 2558 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Jul 19 14:57:38.118255 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:38.115094 2558 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Jul 19 14:57:38.118255 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:38.115097 2558 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Jul 19 14:57:38.118255 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:38.115101 2558 flags.go:64] FLAG: --container-log-max-files="5" Jul 19 14:57:38.118255 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:38.115104 2558 flags.go:64] FLAG: --container-log-max-size="10Mi" Jul 19 14:57:38.118869 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:38.115107 2558 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Jul 19 14:57:38.118869 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:38.115110 2558 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Jul 19 14:57:38.118869 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:38.115113 2558 flags.go:64] FLAG: --containerd-namespace="k8s.io" Jul 19 14:57:38.118869 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:38.115116 2558 flags.go:64] FLAG: --contention-profiling="false" Jul 19 14:57:38.118869 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:38.115119 2558 flags.go:64] FLAG: --cpu-cfs-quota="true" Jul 19 14:57:38.118869 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:38.115122 2558 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Jul 19 14:57:38.118869 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:38.115125 2558 flags.go:64] FLAG: --cpu-manager-policy="none" Jul 19 14:57:38.118869 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:38.115128 2558 flags.go:64] FLAG: --cpu-manager-policy-options="" Jul 19 14:57:38.118869 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:38.115132 2558 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Jul 19 14:57:38.118869 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:38.115135 2558 flags.go:64] FLAG: --enable-controller-attach-detach="true" Jul 19 14:57:38.118869 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:38.115138 2558 flags.go:64] FLAG: --enable-debugging-handlers="true" Jul 19 14:57:38.118869 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:38.115141 2558 flags.go:64] FLAG: --enable-load-reader="false" Jul 19 14:57:38.118869 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:38.115144 2558 flags.go:64] FLAG: --enable-server="true" Jul 19 14:57:38.118869 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:38.115147 2558 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Jul 19 14:57:38.118869 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:38.115152 2558 flags.go:64] FLAG: --event-burst="100" Jul 19 14:57:38.118869 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:38.115155 2558 flags.go:64] FLAG: --event-qps="50" Jul 19 14:57:38.118869 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:38.115158 2558 flags.go:64] FLAG: --event-storage-age-limit="default=0" Jul 19 14:57:38.118869 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:38.115161 2558 flags.go:64] FLAG: --event-storage-event-limit="default=0" Jul 19 14:57:38.118869 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:38.115164 2558 flags.go:64] FLAG: --eviction-hard="" Jul 19 14:57:38.118869 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:38.115168 2558 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Jul 19 14:57:38.118869 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:38.115170 2558 flags.go:64] FLAG: --eviction-minimum-reclaim="" Jul 19 14:57:38.118869 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:38.115173 2558 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Jul 19 14:57:38.118869 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:38.115176 2558 flags.go:64] FLAG: --eviction-soft="" Jul 19 14:57:38.118869 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:38.115179 2558 flags.go:64] FLAG: --eviction-soft-grace-period="" Jul 19 14:57:38.118869 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:38.115182 2558 flags.go:64] FLAG: --exit-on-lock-contention="false" Jul 19 14:57:38.119471 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:38.115185 2558 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Jul 19 14:57:38.119471 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:38.115188 2558 flags.go:64] FLAG: --experimental-mounter-path="" Jul 19 14:57:38.119471 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:38.115191 2558 flags.go:64] FLAG: --fail-cgroupv1="false" Jul 19 14:57:38.119471 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:38.115194 2558 flags.go:64] FLAG: --fail-swap-on="true" Jul 19 14:57:38.119471 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:38.115197 2558 flags.go:64] FLAG: --feature-gates="" Jul 19 14:57:38.119471 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:38.115200 2558 flags.go:64] FLAG: --file-check-frequency="20s" Jul 19 14:57:38.119471 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:38.115203 2558 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Jul 19 14:57:38.119471 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:38.115206 2558 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Jul 19 14:57:38.119471 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:38.115209 2558 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Jul 19 14:57:38.119471 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:38.115213 2558 flags.go:64] FLAG: --healthz-port="10248" Jul 19 14:57:38.119471 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:38.115215 2558 flags.go:64] FLAG: --help="false" Jul 19 14:57:38.119471 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:38.115218 2558 flags.go:64] FLAG: --hostname-override="ip-10-0-138-220.ec2.internal" Jul 19 14:57:38.119471 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:38.115221 2558 flags.go:64] FLAG: --housekeeping-interval="10s" Jul 19 14:57:38.119471 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:38.115224 2558 flags.go:64] FLAG: --http-check-frequency="20s" Jul 19 14:57:38.119471 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:38.115227 2558 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Jul 19 14:57:38.119471 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:38.115230 2558 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Jul 19 14:57:38.119471 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:38.115233 2558 flags.go:64] FLAG: --image-gc-high-threshold="85" Jul 19 14:57:38.119471 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:38.115236 2558 flags.go:64] FLAG: --image-gc-low-threshold="80" Jul 19 14:57:38.119471 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:38.115239 2558 flags.go:64] FLAG: --image-service-endpoint="" Jul 19 14:57:38.119471 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:38.115242 2558 flags.go:64] FLAG: --kernel-memcg-notification="false" Jul 19 14:57:38.119471 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:38.115245 2558 flags.go:64] FLAG: --kube-api-burst="100" Jul 19 14:57:38.119471 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:38.115250 2558 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Jul 19 14:57:38.119471 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:38.115253 2558 flags.go:64] FLAG: --kube-api-qps="50" Jul 19 14:57:38.119471 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:38.115256 2558 flags.go:64] FLAG: --kube-reserved="" Jul 19 14:57:38.120107 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:38.115259 2558 flags.go:64] FLAG: --kube-reserved-cgroup="" Jul 19 14:57:38.120107 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:38.115262 2558 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Jul 19 14:57:38.120107 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:38.115265 2558 flags.go:64] FLAG: --kubelet-cgroups="" Jul 19 14:57:38.120107 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:38.115268 2558 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Jul 19 14:57:38.120107 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:38.115271 2558 flags.go:64] FLAG: --lock-file="" Jul 19 14:57:38.120107 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:38.115273 2558 flags.go:64] FLAG: --log-cadvisor-usage="false" Jul 19 14:57:38.120107 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:38.115276 2558 flags.go:64] FLAG: --log-flush-frequency="5s" Jul 19 14:57:38.120107 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:38.115279 2558 flags.go:64] FLAG: --log-json-info-buffer-size="0" Jul 19 14:57:38.120107 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:38.115285 2558 flags.go:64] FLAG: --log-json-split-stream="false" Jul 19 14:57:38.120107 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:38.115287 2558 flags.go:64] FLAG: --log-text-info-buffer-size="0" Jul 19 14:57:38.120107 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:38.115290 2558 flags.go:64] FLAG: --log-text-split-stream="false" Jul 19 14:57:38.120107 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:38.115294 2558 flags.go:64] FLAG: --logging-format="text" Jul 19 14:57:38.120107 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:38.115296 2558 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Jul 19 14:57:38.120107 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:38.115300 2558 flags.go:64] FLAG: --make-iptables-util-chains="true" Jul 19 14:57:38.120107 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:38.115302 2558 flags.go:64] FLAG: --manifest-url="" Jul 19 14:57:38.120107 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:38.115305 2558 flags.go:64] FLAG: --manifest-url-header="" Jul 19 14:57:38.120107 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:38.115309 2558 flags.go:64] FLAG: --max-housekeeping-interval="15s" Jul 19 14:57:38.120107 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:38.115312 2558 flags.go:64] FLAG: --max-open-files="1000000" Jul 19 14:57:38.120107 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:38.115316 2558 flags.go:64] FLAG: --max-pods="110" Jul 19 14:57:38.120107 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:38.115319 2558 flags.go:64] FLAG: --maximum-dead-containers="-1" Jul 19 14:57:38.120107 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:38.115322 2558 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Jul 19 14:57:38.120107 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:38.115325 2558 flags.go:64] FLAG: --memory-manager-policy="None" Jul 19 14:57:38.120107 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:38.115328 2558 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Jul 19 14:57:38.120107 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:38.115331 2558 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Jul 19 14:57:38.120107 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:38.115334 2558 flags.go:64] FLAG: --node-ip="0.0.0.0" Jul 19 14:57:38.120712 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:38.115337 2558 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Jul 19 14:57:38.120712 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:38.115343 2558 flags.go:64] FLAG: --node-status-max-images="50" Jul 19 14:57:38.120712 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:38.115346 2558 flags.go:64] FLAG: --node-status-update-frequency="10s" Jul 19 14:57:38.120712 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:38.115349 2558 flags.go:64] FLAG: --oom-score-adj="-999" Jul 19 14:57:38.120712 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:38.115353 2558 flags.go:64] FLAG: --pod-cidr="" Jul 19 14:57:38.120712 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:38.115356 2558 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:4017e99d76d7b91598d3a3430cd0eba6eda8b33653addd3838fdfe3cca69c5a0" Jul 19 14:57:38.120712 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:38.115361 2558 flags.go:64] FLAG: --pod-manifest-path="" Jul 19 14:57:38.120712 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:38.115364 2558 flags.go:64] FLAG: --pod-max-pids="-1" Jul 19 14:57:38.120712 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:38.115367 2558 flags.go:64] FLAG: --pods-per-core="0" Jul 19 14:57:38.120712 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:38.115370 2558 flags.go:64] FLAG: --port="10250" Jul 19 14:57:38.120712 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:38.115373 2558 flags.go:64] FLAG: --protect-kernel-defaults="false" Jul 19 14:57:38.120712 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:38.115375 2558 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0041c2f602b70365a" Jul 19 14:57:38.120712 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:38.115378 2558 flags.go:64] FLAG: --qos-reserved="" Jul 19 14:57:38.120712 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:38.115381 2558 flags.go:64] FLAG: --read-only-port="10255" Jul 19 14:57:38.120712 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:38.115384 2558 flags.go:64] FLAG: --register-node="true" Jul 19 14:57:38.120712 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:38.115387 2558 flags.go:64] FLAG: --register-with-taints="" Jul 19 14:57:38.120712 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:38.115390 2558 flags.go:64] FLAG: --registry-burst="10" Jul 19 14:57:38.120712 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:38.115393 2558 flags.go:64] FLAG: --registry-qps="5" Jul 19 14:57:38.120712 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:38.115396 2558 flags.go:64] FLAG: --reserved-cpus="" Jul 19 14:57:38.120712 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:38.115399 2558 flags.go:64] FLAG: --reserved-memory="" Jul 19 14:57:38.120712 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:38.115403 2558 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Jul 19 14:57:38.120712 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:38.115406 2558 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Jul 19 14:57:38.120712 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:38.115408 2558 flags.go:64] FLAG: --rotate-certificates="false" Jul 19 14:57:38.120712 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:38.115411 2558 flags.go:64] FLAG: --rotate-server-certificates="false" Jul 19 14:57:38.120712 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:38.115414 2558 flags.go:64] FLAG: --runonce="false" Jul 19 14:57:38.121333 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:38.115417 2558 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Jul 19 14:57:38.121333 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:38.115419 2558 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Jul 19 14:57:38.121333 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:38.115422 2558 flags.go:64] FLAG: --seccomp-default="false" Jul 19 14:57:38.121333 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:38.115425 2558 flags.go:64] FLAG: --serialize-image-pulls="true" Jul 19 14:57:38.121333 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:38.115428 2558 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Jul 19 14:57:38.121333 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:38.115430 2558 flags.go:64] FLAG: --storage-driver-db="cadvisor" Jul 19 14:57:38.121333 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:38.115435 2558 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Jul 19 14:57:38.121333 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:38.115438 2558 flags.go:64] FLAG: --storage-driver-password="root" Jul 19 14:57:38.121333 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:38.115441 2558 flags.go:64] FLAG: --storage-driver-secure="false" Jul 19 14:57:38.121333 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:38.115444 2558 flags.go:64] FLAG: --storage-driver-table="stats" Jul 19 14:57:38.121333 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:38.115447 2558 flags.go:64] FLAG: --storage-driver-user="root" Jul 19 14:57:38.121333 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:38.115451 2558 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Jul 19 14:57:38.121333 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:38.115454 2558 flags.go:64] FLAG: --sync-frequency="1m0s" Jul 19 14:57:38.121333 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:38.115457 2558 flags.go:64] FLAG: --system-cgroups="" Jul 19 14:57:38.121333 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:38.115460 2558 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Jul 19 14:57:38.121333 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:38.115465 2558 flags.go:64] FLAG: --system-reserved-cgroup="" Jul 19 14:57:38.121333 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:38.115468 2558 flags.go:64] FLAG: --tls-cert-file="" Jul 19 14:57:38.121333 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:38.115470 2558 flags.go:64] FLAG: --tls-cipher-suites="[]" Jul 19 14:57:38.121333 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:38.115474 2558 flags.go:64] FLAG: --tls-min-version="" Jul 19 14:57:38.121333 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:38.115477 2558 flags.go:64] FLAG: --tls-private-key-file="" Jul 19 14:57:38.121333 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:38.115479 2558 flags.go:64] FLAG: --topology-manager-policy="none" Jul 19 14:57:38.121333 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:38.115482 2558 flags.go:64] FLAG: --topology-manager-policy-options="" Jul 19 14:57:38.121333 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:38.115485 2558 flags.go:64] FLAG: --topology-manager-scope="container" Jul 19 14:57:38.121333 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:38.115488 2558 flags.go:64] FLAG: --v="2" Jul 19 14:57:38.121333 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:38.115492 2558 flags.go:64] FLAG: --version="false" Jul 19 14:57:38.121959 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:38.115495 2558 flags.go:64] FLAG: --vmodule="" Jul 19 14:57:38.121959 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:38.115499 2558 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Jul 19 14:57:38.121959 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:38.115502 2558 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Jul 19 14:57:38.121959 ip-10-0-138-220 kubenswrapper[2558]: W0719 14:57:38.115610 2558 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 19 14:57:38.121959 ip-10-0-138-220 kubenswrapper[2558]: W0719 14:57:38.115614 2558 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 19 14:57:38.121959 ip-10-0-138-220 kubenswrapper[2558]: W0719 14:57:38.115617 2558 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 19 14:57:38.121959 ip-10-0-138-220 kubenswrapper[2558]: W0719 14:57:38.115619 2558 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 19 14:57:38.121959 ip-10-0-138-220 kubenswrapper[2558]: W0719 14:57:38.115622 2558 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 19 14:57:38.121959 ip-10-0-138-220 kubenswrapper[2558]: W0719 14:57:38.115624 2558 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 19 14:57:38.121959 ip-10-0-138-220 kubenswrapper[2558]: W0719 14:57:38.115627 2558 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 19 14:57:38.121959 ip-10-0-138-220 kubenswrapper[2558]: W0719 14:57:38.115629 2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 19 14:57:38.121959 ip-10-0-138-220 kubenswrapper[2558]: W0719 14:57:38.115632 2558 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 19 14:57:38.121959 ip-10-0-138-220 kubenswrapper[2558]: W0719 14:57:38.115634 2558 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 19 14:57:38.121959 ip-10-0-138-220 kubenswrapper[2558]: W0719 14:57:38.115638 2558 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 19 14:57:38.121959 ip-10-0-138-220 kubenswrapper[2558]: W0719 14:57:38.115640 2558 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 19 14:57:38.121959 ip-10-0-138-220 kubenswrapper[2558]: W0719 14:57:38.115643 2558 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 19 14:57:38.121959 ip-10-0-138-220 kubenswrapper[2558]: W0719 14:57:38.115645 2558 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 19 14:57:38.121959 ip-10-0-138-220 kubenswrapper[2558]: W0719 14:57:38.115648 2558 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 19 14:57:38.121959 ip-10-0-138-220 kubenswrapper[2558]: W0719 14:57:38.115652 2558 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 19 14:57:38.121959 ip-10-0-138-220 kubenswrapper[2558]: W0719 14:57:38.115654 2558 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 19 14:57:38.122474 ip-10-0-138-220 kubenswrapper[2558]: W0719 14:57:38.115657 2558 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 19 14:57:38.122474 ip-10-0-138-220 kubenswrapper[2558]: W0719 14:57:38.115661 2558 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 19 14:57:38.122474 ip-10-0-138-220 kubenswrapper[2558]: W0719 14:57:38.115664 2558 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 19 14:57:38.122474 ip-10-0-138-220 kubenswrapper[2558]: W0719 14:57:38.115668 2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 19 14:57:38.122474 ip-10-0-138-220 kubenswrapper[2558]: W0719 14:57:38.115671 2558 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 19 14:57:38.122474 ip-10-0-138-220 kubenswrapper[2558]: W0719 14:57:38.115674 2558 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 19 14:57:38.122474 ip-10-0-138-220 kubenswrapper[2558]: W0719 14:57:38.115677 2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 19 14:57:38.122474 ip-10-0-138-220 kubenswrapper[2558]: W0719 14:57:38.115679 2558 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 19 14:57:38.122474 ip-10-0-138-220 kubenswrapper[2558]: W0719 14:57:38.115682 2558 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 19 14:57:38.122474 ip-10-0-138-220 kubenswrapper[2558]: W0719 14:57:38.115685 2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 19 14:57:38.122474 ip-10-0-138-220 kubenswrapper[2558]: W0719 14:57:38.115687 2558 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 19 14:57:38.122474 ip-10-0-138-220 kubenswrapper[2558]: W0719 14:57:38.115691 2558 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 19 14:57:38.122474 ip-10-0-138-220 kubenswrapper[2558]: W0719 14:57:38.115693 2558 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 19 14:57:38.122474 ip-10-0-138-220 kubenswrapper[2558]: W0719 14:57:38.115696 2558 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 19 14:57:38.122474 ip-10-0-138-220 kubenswrapper[2558]: W0719 14:57:38.115698 2558 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 19 14:57:38.122474 ip-10-0-138-220 kubenswrapper[2558]: W0719 14:57:38.115701 2558 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 19 14:57:38.122474 ip-10-0-138-220 kubenswrapper[2558]: W0719 14:57:38.115704 2558 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 19 14:57:38.122474 ip-10-0-138-220 kubenswrapper[2558]: W0719 14:57:38.115707 2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 19 14:57:38.122474 ip-10-0-138-220 kubenswrapper[2558]: W0719 14:57:38.115709 2558 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 19 14:57:38.122474 ip-10-0-138-220 kubenswrapper[2558]: W0719 14:57:38.115711 2558 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 19 14:57:38.123166 ip-10-0-138-220 kubenswrapper[2558]: W0719 14:57:38.115714 2558 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 19 14:57:38.123166 ip-10-0-138-220 kubenswrapper[2558]: W0719 14:57:38.115716 2558 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 19 14:57:38.123166 ip-10-0-138-220 kubenswrapper[2558]: W0719 14:57:38.115720 2558 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 19 14:57:38.123166 ip-10-0-138-220 kubenswrapper[2558]: W0719 14:57:38.115723 2558 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 19 14:57:38.123166 ip-10-0-138-220 kubenswrapper[2558]: W0719 14:57:38.115726 2558 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 19 14:57:38.123166 ip-10-0-138-220 kubenswrapper[2558]: W0719 14:57:38.115730 2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 19 14:57:38.123166 ip-10-0-138-220 kubenswrapper[2558]: W0719 14:57:38.115733 2558 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 19 14:57:38.123166 ip-10-0-138-220 kubenswrapper[2558]: W0719 14:57:38.115736 2558 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 19 14:57:38.123166 ip-10-0-138-220 kubenswrapper[2558]: W0719 14:57:38.115738 2558 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 19 14:57:38.123166 ip-10-0-138-220 kubenswrapper[2558]: W0719 14:57:38.115741 2558 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 19 14:57:38.123166 ip-10-0-138-220 kubenswrapper[2558]: W0719 14:57:38.115745 2558 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 19 14:57:38.123166 ip-10-0-138-220 kubenswrapper[2558]: W0719 14:57:38.115747 2558 feature_gate.go:328] unrecognized feature gate: Example Jul 19 14:57:38.123166 ip-10-0-138-220 kubenswrapper[2558]: W0719 14:57:38.115750 2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 19 14:57:38.123166 ip-10-0-138-220 kubenswrapper[2558]: W0719 14:57:38.115752 2558 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 19 14:57:38.123166 ip-10-0-138-220 kubenswrapper[2558]: W0719 14:57:38.115755 2558 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 19 14:57:38.123166 ip-10-0-138-220 kubenswrapper[2558]: W0719 14:57:38.115758 2558 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 19 14:57:38.123166 ip-10-0-138-220 kubenswrapper[2558]: W0719 14:57:38.115760 2558 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 19 14:57:38.123166 ip-10-0-138-220 kubenswrapper[2558]: W0719 14:57:38.115763 2558 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 19 14:57:38.123166 ip-10-0-138-220 kubenswrapper[2558]: W0719 14:57:38.115765 2558 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 19 14:57:38.123166 ip-10-0-138-220 kubenswrapper[2558]: W0719 14:57:38.115767 2558 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 19 14:57:38.123938 ip-10-0-138-220 kubenswrapper[2558]: W0719 14:57:38.115770 2558 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 19 14:57:38.123938 ip-10-0-138-220 kubenswrapper[2558]: W0719 14:57:38.115773 2558 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 19 14:57:38.123938 ip-10-0-138-220 kubenswrapper[2558]: W0719 14:57:38.115775 2558 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 19 14:57:38.123938 ip-10-0-138-220 kubenswrapper[2558]: W0719 14:57:38.115778 2558 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 19 14:57:38.123938 ip-10-0-138-220 kubenswrapper[2558]: W0719 14:57:38.115780 2558 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 19 14:57:38.123938 ip-10-0-138-220 kubenswrapper[2558]: W0719 14:57:38.115797 2558 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 19 14:57:38.123938 ip-10-0-138-220 kubenswrapper[2558]: W0719 14:57:38.115800 2558 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 19 14:57:38.123938 ip-10-0-138-220 kubenswrapper[2558]: W0719 14:57:38.115803 2558 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 19 14:57:38.123938 ip-10-0-138-220 kubenswrapper[2558]: W0719 14:57:38.115806 2558 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 19 14:57:38.123938 ip-10-0-138-220 kubenswrapper[2558]: W0719 14:57:38.115809 2558 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 19 14:57:38.123938 ip-10-0-138-220 kubenswrapper[2558]: W0719 14:57:38.115812 2558 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 19 14:57:38.123938 ip-10-0-138-220 kubenswrapper[2558]: W0719 14:57:38.115814 2558 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 19 14:57:38.123938 ip-10-0-138-220 kubenswrapper[2558]: W0719 14:57:38.115817 2558 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 19 14:57:38.123938 ip-10-0-138-220 kubenswrapper[2558]: W0719 14:57:38.115820 2558 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 19 14:57:38.123938 ip-10-0-138-220 kubenswrapper[2558]: W0719 14:57:38.115822 2558 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 19 14:57:38.123938 ip-10-0-138-220 kubenswrapper[2558]: W0719 14:57:38.115825 2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 19 14:57:38.123938 ip-10-0-138-220 kubenswrapper[2558]: W0719 14:57:38.115827 2558 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 19 14:57:38.123938 ip-10-0-138-220 kubenswrapper[2558]: W0719 14:57:38.115831 2558 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 19 14:57:38.123938 ip-10-0-138-220 kubenswrapper[2558]: W0719 14:57:38.115833 2558 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 19 14:57:38.123938 ip-10-0-138-220 kubenswrapper[2558]: W0719 14:57:38.115836 2558 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 19 14:57:38.124420 ip-10-0-138-220 kubenswrapper[2558]: W0719 14:57:38.115838 2558 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 19 14:57:38.124420 ip-10-0-138-220 kubenswrapper[2558]: W0719 14:57:38.115841 2558 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 19 14:57:38.124420 ip-10-0-138-220 kubenswrapper[2558]: W0719 14:57:38.115845 2558 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 19 14:57:38.124420 ip-10-0-138-220 kubenswrapper[2558]: W0719 14:57:38.115849 2558 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 19 14:57:38.124420 ip-10-0-138-220 kubenswrapper[2558]: W0719 14:57:38.115852 2558 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 19 14:57:38.124420 ip-10-0-138-220 kubenswrapper[2558]: W0719 14:57:38.115855 2558 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 19 14:57:38.124420 ip-10-0-138-220 kubenswrapper[2558]: W0719 14:57:38.115857 2558 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 19 14:57:38.124420 ip-10-0-138-220 kubenswrapper[2558]: W0719 14:57:38.115860 2558 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 19 14:57:38.124420 ip-10-0-138-220 kubenswrapper[2558]: W0719 14:57:38.115862 2558 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 19 14:57:38.124420 ip-10-0-138-220 kubenswrapper[2558]: W0719 14:57:38.115865 2558 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 19 14:57:38.124420 ip-10-0-138-220 kubenswrapper[2558]: W0719 14:57:38.115868 2558 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 19 14:57:38.124420 ip-10-0-138-220 kubenswrapper[2558]: W0719 14:57:38.115871 2558 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 19 14:57:38.124420 ip-10-0-138-220 kubenswrapper[2558]: W0719 14:57:38.115874 2558 feature_gate.go:328] unrecognized feature gate: Example2 Jul 19 14:57:38.124420 ip-10-0-138-220 kubenswrapper[2558]: W0719 14:57:38.115876 2558 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 19 14:57:38.124420 ip-10-0-138-220 kubenswrapper[2558]: W0719 14:57:38.115879 2558 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 19 14:57:38.124420 ip-10-0-138-220 kubenswrapper[2558]: W0719 14:57:38.115881 2558 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 19 14:57:38.124420 ip-10-0-138-220 kubenswrapper[2558]: W0719 14:57:38.115884 2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 19 14:57:38.124420 ip-10-0-138-220 kubenswrapper[2558]: W0719 14:57:38.115887 2558 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 19 14:57:38.124420 ip-10-0-138-220 kubenswrapper[2558]: W0719 14:57:38.115889 2558 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 19 14:57:38.124903 ip-10-0-138-220 kubenswrapper[2558]: W0719 14:57:38.115892 2558 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 19 14:57:38.124903 ip-10-0-138-220 kubenswrapper[2558]: W0719 14:57:38.115894 2558 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 19 14:57:38.124903 ip-10-0-138-220 kubenswrapper[2558]: W0719 14:57:38.115897 2558 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 19 14:57:38.124903 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:38.116628 2558 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 19 14:57:38.125286 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:38.125266 2558 server.go:529] "Kubelet version" kubeletVersion="v1.34.9" Jul 19 14:57:38.125317 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:38.125287 2558 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Jul 19 14:57:38.125317 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:38.125306 2558 watchdog_linux.go:95] "Systemd watchdog is not enabled" Jul 19 14:57:38.125317 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:38.125314 2558 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Jul 19 14:57:38.125390 ip-10-0-138-220 kubenswrapper[2558]: W0719 14:57:38.125374 2558 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 19 14:57:38.125390 ip-10-0-138-220 kubenswrapper[2558]: W0719 14:57:38.125379 2558 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 19 14:57:38.125390 ip-10-0-138-220 kubenswrapper[2558]: W0719 14:57:38.125383 2558 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 19 14:57:38.125390 ip-10-0-138-220 kubenswrapper[2558]: W0719 14:57:38.125386 2558 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 19 14:57:38.125390 ip-10-0-138-220 kubenswrapper[2558]: W0719 14:57:38.125389 2558 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 19 14:57:38.125390 ip-10-0-138-220 kubenswrapper[2558]: W0719 14:57:38.125392 2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 19 14:57:38.125537 ip-10-0-138-220 kubenswrapper[2558]: W0719 14:57:38.125395 2558 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 19 14:57:38.125537 ip-10-0-138-220 kubenswrapper[2558]: W0719 14:57:38.125397 2558 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 19 14:57:38.125537 ip-10-0-138-220 kubenswrapper[2558]: W0719 14:57:38.125400 2558 feature_gate.go:328] unrecognized feature gate: Example2 Jul 19 14:57:38.125537 ip-10-0-138-220 kubenswrapper[2558]: W0719 14:57:38.125402 2558 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 19 14:57:38.125537 ip-10-0-138-220 kubenswrapper[2558]: W0719 14:57:38.125405 2558 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 19 14:57:38.125537 ip-10-0-138-220 kubenswrapper[2558]: W0719 14:57:38.125407 2558 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 19 14:57:38.125537 ip-10-0-138-220 kubenswrapper[2558]: W0719 14:57:38.125410 2558 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 19 14:57:38.125537 ip-10-0-138-220 kubenswrapper[2558]: W0719 14:57:38.125412 2558 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 19 14:57:38.125537 ip-10-0-138-220 kubenswrapper[2558]: W0719 14:57:38.125415 2558 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 19 14:57:38.125537 ip-10-0-138-220 kubenswrapper[2558]: W0719 14:57:38.125419 2558 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 19 14:57:38.125537 ip-10-0-138-220 kubenswrapper[2558]: W0719 14:57:38.125422 2558 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 19 14:57:38.125537 ip-10-0-138-220 kubenswrapper[2558]: W0719 14:57:38.125426 2558 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 19 14:57:38.125537 ip-10-0-138-220 kubenswrapper[2558]: W0719 14:57:38.125430 2558 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 19 14:57:38.125537 ip-10-0-138-220 kubenswrapper[2558]: W0719 14:57:38.125433 2558 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 19 14:57:38.125537 ip-10-0-138-220 kubenswrapper[2558]: W0719 14:57:38.125436 2558 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 19 14:57:38.125537 ip-10-0-138-220 kubenswrapper[2558]: W0719 14:57:38.125439 2558 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 19 14:57:38.125537 ip-10-0-138-220 kubenswrapper[2558]: W0719 14:57:38.125442 2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 19 14:57:38.125537 ip-10-0-138-220 kubenswrapper[2558]: W0719 14:57:38.125445 2558 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 19 14:57:38.125537 ip-10-0-138-220 kubenswrapper[2558]: W0719 14:57:38.125448 2558 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 19 14:57:38.126071 ip-10-0-138-220 kubenswrapper[2558]: W0719 14:57:38.125450 2558 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 19 14:57:38.126071 ip-10-0-138-220 kubenswrapper[2558]: W0719 14:57:38.125453 2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 19 14:57:38.126071 ip-10-0-138-220 kubenswrapper[2558]: W0719 14:57:38.125455 2558 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 19 14:57:38.126071 ip-10-0-138-220 kubenswrapper[2558]: W0719 14:57:38.125458 2558 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 19 14:57:38.126071 ip-10-0-138-220 kubenswrapper[2558]: W0719 14:57:38.125461 2558 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 19 14:57:38.126071 ip-10-0-138-220 kubenswrapper[2558]: W0719 14:57:38.125463 2558 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 19 14:57:38.126071 ip-10-0-138-220 kubenswrapper[2558]: W0719 14:57:38.125466 2558 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 19 14:57:38.126071 ip-10-0-138-220 kubenswrapper[2558]: W0719 14:57:38.125469 2558 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 19 14:57:38.126071 ip-10-0-138-220 kubenswrapper[2558]: W0719 14:57:38.125472 2558 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 19 14:57:38.126071 ip-10-0-138-220 kubenswrapper[2558]: W0719 14:57:38.125474 2558 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 19 14:57:38.126071 ip-10-0-138-220 kubenswrapper[2558]: W0719 14:57:38.125477 2558 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 19 14:57:38.126071 ip-10-0-138-220 kubenswrapper[2558]: W0719 14:57:38.125479 2558 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 19 14:57:38.126071 ip-10-0-138-220 kubenswrapper[2558]: W0719 14:57:38.125482 2558 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 19 14:57:38.126071 ip-10-0-138-220 kubenswrapper[2558]: W0719 14:57:38.125484 2558 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 19 14:57:38.126071 ip-10-0-138-220 kubenswrapper[2558]: W0719 14:57:38.125488 2558 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 19 14:57:38.126071 ip-10-0-138-220 kubenswrapper[2558]: W0719 14:57:38.125491 2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 19 14:57:38.126071 ip-10-0-138-220 kubenswrapper[2558]: W0719 14:57:38.125493 2558 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 19 14:57:38.126071 ip-10-0-138-220 kubenswrapper[2558]: W0719 14:57:38.125496 2558 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 19 14:57:38.126071 ip-10-0-138-220 kubenswrapper[2558]: W0719 14:57:38.125498 2558 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 19 14:57:38.126071 ip-10-0-138-220 kubenswrapper[2558]: W0719 14:57:38.125501 2558 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 19 14:57:38.126587 ip-10-0-138-220 kubenswrapper[2558]: W0719 14:57:38.125503 2558 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 19 14:57:38.126587 ip-10-0-138-220 kubenswrapper[2558]: W0719 14:57:38.125506 2558 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 19 14:57:38.126587 ip-10-0-138-220 kubenswrapper[2558]: W0719 14:57:38.125509 2558 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 19 14:57:38.126587 ip-10-0-138-220 kubenswrapper[2558]: W0719 14:57:38.125512 2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 19 14:57:38.126587 ip-10-0-138-220 kubenswrapper[2558]: W0719 14:57:38.125514 2558 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 19 14:57:38.126587 ip-10-0-138-220 kubenswrapper[2558]: W0719 14:57:38.125517 2558 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 19 14:57:38.126587 ip-10-0-138-220 kubenswrapper[2558]: W0719 14:57:38.125519 2558 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 19 14:57:38.126587 ip-10-0-138-220 kubenswrapper[2558]: W0719 14:57:38.125522 2558 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 19 14:57:38.126587 ip-10-0-138-220 kubenswrapper[2558]: W0719 14:57:38.125524 2558 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 19 14:57:38.126587 ip-10-0-138-220 kubenswrapper[2558]: W0719 14:57:38.125527 2558 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 19 14:57:38.126587 ip-10-0-138-220 kubenswrapper[2558]: W0719 14:57:38.125530 2558 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 19 14:57:38.126587 ip-10-0-138-220 kubenswrapper[2558]: W0719 14:57:38.125532 2558 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 19 14:57:38.126587 ip-10-0-138-220 kubenswrapper[2558]: W0719 14:57:38.125535 2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 19 14:57:38.126587 ip-10-0-138-220 kubenswrapper[2558]: W0719 14:57:38.125537 2558 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 19 14:57:38.126587 ip-10-0-138-220 kubenswrapper[2558]: W0719 14:57:38.125540 2558 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 19 14:57:38.126587 ip-10-0-138-220 kubenswrapper[2558]: W0719 14:57:38.125543 2558 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 19 14:57:38.126587 ip-10-0-138-220 kubenswrapper[2558]: W0719 14:57:38.125545 2558 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 19 14:57:38.126587 ip-10-0-138-220 kubenswrapper[2558]: W0719 14:57:38.125548 2558 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 19 14:57:38.126587 ip-10-0-138-220 kubenswrapper[2558]: W0719 14:57:38.125551 2558 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 19 14:57:38.126587 ip-10-0-138-220 kubenswrapper[2558]: W0719 14:57:38.125553 2558 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 19 14:57:38.126587 ip-10-0-138-220 kubenswrapper[2558]: W0719 14:57:38.125556 2558 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 19 14:57:38.127093 ip-10-0-138-220 kubenswrapper[2558]: W0719 14:57:38.125558 2558 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 19 14:57:38.127093 ip-10-0-138-220 kubenswrapper[2558]: W0719 14:57:38.125561 2558 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 19 14:57:38.127093 ip-10-0-138-220 kubenswrapper[2558]: W0719 14:57:38.125563 2558 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 19 14:57:38.127093 ip-10-0-138-220 kubenswrapper[2558]: W0719 14:57:38.125566 2558 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 19 14:57:38.127093 ip-10-0-138-220 kubenswrapper[2558]: W0719 14:57:38.125568 2558 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 19 14:57:38.127093 ip-10-0-138-220 kubenswrapper[2558]: W0719 14:57:38.125571 2558 feature_gate.go:328] unrecognized feature gate: Example Jul 19 14:57:38.127093 ip-10-0-138-220 kubenswrapper[2558]: W0719 14:57:38.125574 2558 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 19 14:57:38.127093 ip-10-0-138-220 kubenswrapper[2558]: W0719 14:57:38.125576 2558 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 19 14:57:38.127093 ip-10-0-138-220 kubenswrapper[2558]: W0719 14:57:38.125579 2558 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 19 14:57:38.127093 ip-10-0-138-220 kubenswrapper[2558]: W0719 14:57:38.125581 2558 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 19 14:57:38.127093 ip-10-0-138-220 kubenswrapper[2558]: W0719 14:57:38.125584 2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 19 14:57:38.127093 ip-10-0-138-220 kubenswrapper[2558]: W0719 14:57:38.125586 2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 19 14:57:38.127093 ip-10-0-138-220 kubenswrapper[2558]: W0719 14:57:38.125589 2558 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 19 14:57:38.127093 ip-10-0-138-220 kubenswrapper[2558]: W0719 14:57:38.125592 2558 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 19 14:57:38.127093 ip-10-0-138-220 kubenswrapper[2558]: W0719 14:57:38.125595 2558 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 19 14:57:38.127093 ip-10-0-138-220 kubenswrapper[2558]: W0719 14:57:38.125597 2558 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 19 14:57:38.127093 ip-10-0-138-220 kubenswrapper[2558]: W0719 14:57:38.125600 2558 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 19 14:57:38.127093 ip-10-0-138-220 kubenswrapper[2558]: W0719 14:57:38.125602 2558 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 19 14:57:38.127093 ip-10-0-138-220 kubenswrapper[2558]: W0719 14:57:38.125605 2558 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 19 14:57:38.127093 ip-10-0-138-220 kubenswrapper[2558]: W0719 14:57:38.125607 2558 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 19 14:57:38.127570 ip-10-0-138-220 kubenswrapper[2558]: W0719 14:57:38.125610 2558 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 19 14:57:38.127570 ip-10-0-138-220 kubenswrapper[2558]: W0719 14:57:38.125612 2558 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 19 14:57:38.127570 ip-10-0-138-220 kubenswrapper[2558]: W0719 14:57:38.125615 2558 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 19 14:57:38.127570 ip-10-0-138-220 kubenswrapper[2558]: W0719 14:57:38.125617 2558 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 19 14:57:38.127570 ip-10-0-138-220 kubenswrapper[2558]: W0719 14:57:38.125619 2558 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 19 14:57:38.127570 ip-10-0-138-220 kubenswrapper[2558]: W0719 14:57:38.125622 2558 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 19 14:57:38.127570 ip-10-0-138-220 kubenswrapper[2558]: W0719 14:57:38.125626 2558 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 19 14:57:38.127570 ip-10-0-138-220 kubenswrapper[2558]: W0719 14:57:38.125629 2558 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 19 14:57:38.127570 ip-10-0-138-220 kubenswrapper[2558]: W0719 14:57:38.125632 2558 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 19 14:57:38.127570 ip-10-0-138-220 kubenswrapper[2558]: W0719 14:57:38.125635 2558 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 19 14:57:38.127570 ip-10-0-138-220 kubenswrapper[2558]: W0719 14:57:38.125638 2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 19 14:57:38.127570 ip-10-0-138-220 kubenswrapper[2558]: W0719 14:57:38.125641 2558 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 19 14:57:38.127570 ip-10-0-138-220 kubenswrapper[2558]: W0719 14:57:38.125644 2558 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 19 14:57:38.127570 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:38.125649 2558 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 19 14:57:38.127921 ip-10-0-138-220 kubenswrapper[2558]: W0719 14:57:38.125768 2558 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 19 14:57:38.127921 ip-10-0-138-220 kubenswrapper[2558]: W0719 14:57:38.125773 2558 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 19 14:57:38.127921 ip-10-0-138-220 kubenswrapper[2558]: W0719 14:57:38.125776 2558 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 19 14:57:38.127921 ip-10-0-138-220 kubenswrapper[2558]: W0719 14:57:38.125779 2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 19 14:57:38.127921 ip-10-0-138-220 kubenswrapper[2558]: W0719 14:57:38.125782 2558 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 19 14:57:38.127921 ip-10-0-138-220 kubenswrapper[2558]: W0719 14:57:38.125805 2558 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 19 14:57:38.127921 ip-10-0-138-220 kubenswrapper[2558]: W0719 14:57:38.125808 2558 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 19 14:57:38.127921 ip-10-0-138-220 kubenswrapper[2558]: W0719 14:57:38.125811 2558 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 19 14:57:38.127921 ip-10-0-138-220 kubenswrapper[2558]: W0719 14:57:38.125815 2558 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 19 14:57:38.127921 ip-10-0-138-220 kubenswrapper[2558]: W0719 14:57:38.125820 2558 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 19 14:57:38.127921 ip-10-0-138-220 kubenswrapper[2558]: W0719 14:57:38.125824 2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 19 14:57:38.127921 ip-10-0-138-220 kubenswrapper[2558]: W0719 14:57:38.125827 2558 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 19 14:57:38.127921 ip-10-0-138-220 kubenswrapper[2558]: W0719 14:57:38.125831 2558 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 19 14:57:38.127921 ip-10-0-138-220 kubenswrapper[2558]: W0719 14:57:38.125833 2558 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 19 14:57:38.127921 ip-10-0-138-220 kubenswrapper[2558]: W0719 14:57:38.125837 2558 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 19 14:57:38.127921 ip-10-0-138-220 kubenswrapper[2558]: W0719 14:57:38.125839 2558 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 19 14:57:38.127921 ip-10-0-138-220 kubenswrapper[2558]: W0719 14:57:38.125842 2558 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 19 14:57:38.127921 ip-10-0-138-220 kubenswrapper[2558]: W0719 14:57:38.125845 2558 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 19 14:57:38.127921 ip-10-0-138-220 kubenswrapper[2558]: W0719 14:57:38.125847 2558 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 19 14:57:38.128377 ip-10-0-138-220 kubenswrapper[2558]: W0719 14:57:38.125850 2558 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 19 14:57:38.128377 ip-10-0-138-220 kubenswrapper[2558]: W0719 14:57:38.125853 2558 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 19 14:57:38.128377 ip-10-0-138-220 kubenswrapper[2558]: W0719 14:57:38.125855 2558 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 19 14:57:38.128377 ip-10-0-138-220 kubenswrapper[2558]: W0719 14:57:38.125858 2558 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 19 14:57:38.128377 ip-10-0-138-220 kubenswrapper[2558]: W0719 14:57:38.125860 2558 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 19 14:57:38.128377 ip-10-0-138-220 kubenswrapper[2558]: W0719 14:57:38.125862 2558 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 19 14:57:38.128377 ip-10-0-138-220 kubenswrapper[2558]: W0719 14:57:38.125865 2558 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 19 14:57:38.128377 ip-10-0-138-220 kubenswrapper[2558]: W0719 14:57:38.125869 2558 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 19 14:57:38.128377 ip-10-0-138-220 kubenswrapper[2558]: W0719 14:57:38.125872 2558 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 19 14:57:38.128377 ip-10-0-138-220 kubenswrapper[2558]: W0719 14:57:38.125875 2558 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 19 14:57:38.128377 ip-10-0-138-220 kubenswrapper[2558]: W0719 14:57:38.125878 2558 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 19 14:57:38.128377 ip-10-0-138-220 kubenswrapper[2558]: W0719 14:57:38.125881 2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 19 14:57:38.128377 ip-10-0-138-220 kubenswrapper[2558]: W0719 14:57:38.125884 2558 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 19 14:57:38.128377 ip-10-0-138-220 kubenswrapper[2558]: W0719 14:57:38.125886 2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 19 14:57:38.128377 ip-10-0-138-220 kubenswrapper[2558]: W0719 14:57:38.125889 2558 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 19 14:57:38.128377 ip-10-0-138-220 kubenswrapper[2558]: W0719 14:57:38.125891 2558 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 19 14:57:38.128377 ip-10-0-138-220 kubenswrapper[2558]: W0719 14:57:38.125894 2558 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 19 14:57:38.128377 ip-10-0-138-220 kubenswrapper[2558]: W0719 14:57:38.125897 2558 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 19 14:57:38.128377 ip-10-0-138-220 kubenswrapper[2558]: W0719 14:57:38.125899 2558 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 19 14:57:38.128941 ip-10-0-138-220 kubenswrapper[2558]: W0719 14:57:38.125902 2558 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 19 14:57:38.128941 ip-10-0-138-220 kubenswrapper[2558]: W0719 14:57:38.125904 2558 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 19 14:57:38.128941 ip-10-0-138-220 kubenswrapper[2558]: W0719 14:57:38.125907 2558 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 19 14:57:38.128941 ip-10-0-138-220 kubenswrapper[2558]: W0719 14:57:38.125909 2558 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 19 14:57:38.128941 ip-10-0-138-220 kubenswrapper[2558]: W0719 14:57:38.125913 2558 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 19 14:57:38.128941 ip-10-0-138-220 kubenswrapper[2558]: W0719 14:57:38.125916 2558 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 19 14:57:38.128941 ip-10-0-138-220 kubenswrapper[2558]: W0719 14:57:38.125919 2558 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 19 14:57:38.128941 ip-10-0-138-220 kubenswrapper[2558]: W0719 14:57:38.125923 2558 feature_gate.go:328] unrecognized feature gate: Example2 Jul 19 14:57:38.128941 ip-10-0-138-220 kubenswrapper[2558]: W0719 14:57:38.125925 2558 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 19 14:57:38.128941 ip-10-0-138-220 kubenswrapper[2558]: W0719 14:57:38.125928 2558 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 19 14:57:38.128941 ip-10-0-138-220 kubenswrapper[2558]: W0719 14:57:38.125930 2558 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 19 14:57:38.128941 ip-10-0-138-220 kubenswrapper[2558]: W0719 14:57:38.125933 2558 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 19 14:57:38.128941 ip-10-0-138-220 kubenswrapper[2558]: W0719 14:57:38.125935 2558 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 19 14:57:38.128941 ip-10-0-138-220 kubenswrapper[2558]: W0719 14:57:38.125938 2558 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 19 14:57:38.128941 ip-10-0-138-220 kubenswrapper[2558]: W0719 14:57:38.125940 2558 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 19 14:57:38.128941 ip-10-0-138-220 kubenswrapper[2558]: W0719 14:57:38.125943 2558 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 19 14:57:38.128941 ip-10-0-138-220 kubenswrapper[2558]: W0719 14:57:38.125945 2558 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 19 14:57:38.128941 ip-10-0-138-220 kubenswrapper[2558]: W0719 14:57:38.125947 2558 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 19 14:57:38.128941 ip-10-0-138-220 kubenswrapper[2558]: W0719 14:57:38.125950 2558 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 19 14:57:38.128941 ip-10-0-138-220 kubenswrapper[2558]: W0719 14:57:38.125952 2558 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 19 14:57:38.129412 ip-10-0-138-220 kubenswrapper[2558]: W0719 14:57:38.125955 2558 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 19 14:57:38.129412 ip-10-0-138-220 kubenswrapper[2558]: W0719 14:57:38.125957 2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 19 14:57:38.129412 ip-10-0-138-220 kubenswrapper[2558]: W0719 14:57:38.125960 2558 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 19 14:57:38.129412 ip-10-0-138-220 kubenswrapper[2558]: W0719 14:57:38.125962 2558 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 19 14:57:38.129412 ip-10-0-138-220 kubenswrapper[2558]: W0719 14:57:38.125965 2558 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 19 14:57:38.129412 ip-10-0-138-220 kubenswrapper[2558]: W0719 14:57:38.125967 2558 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 19 14:57:38.129412 ip-10-0-138-220 kubenswrapper[2558]: W0719 14:57:38.125970 2558 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 19 14:57:38.129412 ip-10-0-138-220 kubenswrapper[2558]: W0719 14:57:38.125973 2558 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 19 14:57:38.129412 ip-10-0-138-220 kubenswrapper[2558]: W0719 14:57:38.125975 2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 19 14:57:38.129412 ip-10-0-138-220 kubenswrapper[2558]: W0719 14:57:38.125978 2558 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 19 14:57:38.129412 ip-10-0-138-220 kubenswrapper[2558]: W0719 14:57:38.125981 2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 19 14:57:38.129412 ip-10-0-138-220 kubenswrapper[2558]: W0719 14:57:38.125984 2558 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 19 14:57:38.129412 ip-10-0-138-220 kubenswrapper[2558]: W0719 14:57:38.125986 2558 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 19 14:57:38.129412 ip-10-0-138-220 kubenswrapper[2558]: W0719 14:57:38.125988 2558 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 19 14:57:38.129412 ip-10-0-138-220 kubenswrapper[2558]: W0719 14:57:38.125991 2558 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 19 14:57:38.129412 ip-10-0-138-220 kubenswrapper[2558]: W0719 14:57:38.125993 2558 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 19 14:57:38.129412 ip-10-0-138-220 kubenswrapper[2558]: W0719 14:57:38.125995 2558 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 19 14:57:38.129412 ip-10-0-138-220 kubenswrapper[2558]: W0719 14:57:38.125998 2558 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 19 14:57:38.129412 ip-10-0-138-220 kubenswrapper[2558]: W0719 14:57:38.126001 2558 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 19 14:57:38.129412 ip-10-0-138-220 kubenswrapper[2558]: W0719 14:57:38.126003 2558 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 19 14:57:38.129903 ip-10-0-138-220 kubenswrapper[2558]: W0719 14:57:38.126006 2558 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 19 14:57:38.129903 ip-10-0-138-220 kubenswrapper[2558]: W0719 14:57:38.126008 2558 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 19 14:57:38.129903 ip-10-0-138-220 kubenswrapper[2558]: W0719 14:57:38.126011 2558 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 19 14:57:38.129903 ip-10-0-138-220 kubenswrapper[2558]: W0719 14:57:38.126013 2558 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 19 14:57:38.129903 ip-10-0-138-220 kubenswrapper[2558]: W0719 14:57:38.126015 2558 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 19 14:57:38.129903 ip-10-0-138-220 kubenswrapper[2558]: W0719 14:57:38.126018 2558 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 19 14:57:38.129903 ip-10-0-138-220 kubenswrapper[2558]: W0719 14:57:38.126020 2558 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 19 14:57:38.129903 ip-10-0-138-220 kubenswrapper[2558]: W0719 14:57:38.126022 2558 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 19 14:57:38.129903 ip-10-0-138-220 kubenswrapper[2558]: W0719 14:57:38.126025 2558 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 19 14:57:38.129903 ip-10-0-138-220 kubenswrapper[2558]: W0719 14:57:38.126027 2558 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 19 14:57:38.129903 ip-10-0-138-220 kubenswrapper[2558]: W0719 14:57:38.126029 2558 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 19 14:57:38.129903 ip-10-0-138-220 kubenswrapper[2558]: W0719 14:57:38.126032 2558 feature_gate.go:328] unrecognized feature gate: Example Jul 19 14:57:38.129903 ip-10-0-138-220 kubenswrapper[2558]: W0719 14:57:38.126034 2558 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 19 14:57:38.129903 ip-10-0-138-220 kubenswrapper[2558]: W0719 14:57:38.126036 2558 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 19 14:57:38.129903 ip-10-0-138-220 kubenswrapper[2558]: W0719 14:57:38.126039 2558 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 19 14:57:38.129903 ip-10-0-138-220 kubenswrapper[2558]: W0719 14:57:38.126041 2558 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 19 14:57:38.129903 ip-10-0-138-220 kubenswrapper[2558]: W0719 14:57:38.126044 2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 19 14:57:38.129903 ip-10-0-138-220 kubenswrapper[2558]: W0719 14:57:38.126046 2558 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 19 14:57:38.129903 ip-10-0-138-220 kubenswrapper[2558]: W0719 14:57:38.126048 2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 19 14:57:38.129903 ip-10-0-138-220 kubenswrapper[2558]: W0719 14:57:38.126051 2558 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 19 14:57:38.129903 ip-10-0-138-220 kubenswrapper[2558]: W0719 14:57:38.126053 2558 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 19 14:57:38.130405 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:38.126058 2558 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 19 14:57:38.130405 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:38.126169 2558 server.go:962] "Client rotation is on, will bootstrap in background" Jul 19 14:57:38.130405 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:38.129413 2558 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Jul 19 14:57:38.131091 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:38.131078 2558 server.go:1019] "Starting client certificate rotation" Jul 19 14:57:38.131200 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:38.131183 2558 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 19 14:57:38.131233 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:38.131220 2558 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 19 14:57:38.135614 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:38.135601 2558 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 19 14:57:38.136865 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:38.136835 2558 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 19 14:57:38.149234 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:38.149216 2558 log.go:25] "Validated CRI v1 runtime API" Jul 19 14:57:38.151952 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:38.151937 2558 log.go:25] "Validated CRI v1 image API" Jul 19 14:57:38.152851 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:38.152832 2558 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Jul 19 14:57:38.157493 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:38.157472 2558 fs.go:135] Filesystem UUIDs: map[1d83e0c3-53fb-4499-a57a-8abdf922c8f1:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 9ab7e7e9-c296-4ad3-95fa-0e02c8917b7d:/dev/nvme0n1p3] Jul 19 14:57:38.157572 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:38.157491 2558 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Jul 19 14:57:38.162418 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:38.162397 2558 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 19 14:57:38.164056 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:38.163949 2558 manager.go:216] Machine: {Timestamp:2026-07-19 14:57:38.162606747 +0000 UTC m=+0.378544612 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3100050 MemoryCapacity:32812175360 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2e14f15c28d53cb6e2fde0328ad7ad SystemUUID:ec2e14f1-5c28-d53c-b6e2-fde0328ad7ad BootID:7091494a-6472-48fa-90c6-9296425aae86 Filesystems:[{Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16406089728 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16406085632 Type:vfs Inodes:4005392 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6562435072 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:75:11:80:ef:67 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:75:11:80:ef:67 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:9a:b8:b4:d9:5a:65 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812175360 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Jul 19 14:57:38.164056 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:38.164050 2558 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Jul 19 14:57:38.164207 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:38.164144 2558 manager.go:232] Version: {KernelVersion:5.14.0-570.128.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260715-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Jul 19 14:57:38.167326 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:38.167305 2558 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Jul 19 14:57:38.167454 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:38.167329 2558 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-138-220.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Jul 19 14:57:38.168862 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:38.168852 2558 topology_manager.go:138] "Creating topology manager with none policy" Jul 19 14:57:38.168895 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:38.168864 2558 container_manager_linux.go:309] "Creating device plugin manager" Jul 19 14:57:38.168895 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:38.168873 2558 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 19 14:57:38.168895 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:38.168886 2558 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 19 14:57:38.170482 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:38.170472 2558 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Jul 19 14:57:38.174763 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:38.174753 2558 state_mem.go:36] "Initialized new in-memory state store" Jul 19 14:57:38.174879 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:38.174870 2558 server.go:1240] "Using root directory" path="/var/lib/kubelet" Jul 19 14:57:38.174912 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:38.174898 2558 kubelet.go:486] "Attempting to sync node with API server" Jul 19 14:57:38.174912 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:38.174907 2558 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Jul 19 14:57:38.174966 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:38.174926 2558 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Jul 19 14:57:38.174966 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:38.174934 2558 kubelet.go:398] "Adding apiserver pod source" Jul 19 14:57:38.174966 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:38.174943 2558 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Jul 19 14:57:38.175243 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:38.175232 2558 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 19 14:57:38.175287 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:38.175250 2558 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 19 14:57:38.177715 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:38.177695 2558 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1" Jul 19 14:57:38.180014 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:38.180001 2558 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Jul 19 14:57:38.180067 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:38.180031 2558 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Jul 19 14:57:38.181457 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:38.181437 2558 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Jul 19 14:57:38.181457 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:38.181453 2558 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Jul 19 14:57:38.181526 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:38.181478 2558 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Jul 19 14:57:38.181526 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:38.181487 2558 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Jul 19 14:57:38.181526 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:38.181500 2558 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Jul 19 14:57:38.181526 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:38.181517 2558 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Jul 19 14:57:38.181526 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:38.181526 2558 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Jul 19 14:57:38.181644 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:38.181532 2558 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Jul 19 14:57:38.181644 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:38.181538 2558 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Jul 19 14:57:38.181644 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:38.181544 2558 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Jul 19 14:57:38.181644 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:38.181553 2558 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Jul 19 14:57:38.181644 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:38.181558 2558 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Jul 19 14:57:38.182412 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:38.182401 2558 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Jul 19 14:57:38.182412 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:38.182413 2558 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Jul 19 14:57:38.184965 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:38.184911 2558 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-q4bxv" Jul 19 14:57:38.185084 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:38.185070 2558 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-138-220.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Jul 19 14:57:38.185194 ip-10-0-138-220 kubenswrapper[2558]: E0719 14:57:38.185169 2558 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-138-220.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Jul 19 14:57:38.185225 ip-10-0-138-220 kubenswrapper[2558]: E0719 14:57:38.185169 2558 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Jul 19 14:57:38.186009 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:38.185996 2558 server.go:1268] "Started kubelet" Jul 19 14:57:38.186315 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:38.186279 2558 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Jul 19 14:57:38.186591 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:38.186313 2558 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Jul 19 14:57:38.186759 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:38.186733 2558 server_v1.go:49] "podresources" method="list" useActivePods=true Jul 19 14:57:38.186867 ip-10-0-138-220 systemd[1]: Started Kubernetes Kubelet. Jul 19 14:57:38.188468 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:38.188451 2558 server.go:310] "Adding debug handlers to kubelet server" Jul 19 14:57:38.188674 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:38.188648 2558 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Jul 19 14:57:38.189733 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:38.189715 2558 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-q4bxv" Jul 19 14:57:38.191713 ip-10-0-138-220 kubenswrapper[2558]: E0719 14:57:38.191679 2558 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Jul 19 14:57:38.192588 ip-10-0-138-220 kubenswrapper[2558]: E0719 14:57:38.190732 2558 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-138-220.ec2.internal.18c3b8cab86c95cf default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-138-220.ec2.internal,UID:ip-10-0-138-220.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-138-220.ec2.internal,},FirstTimestamp:2026-07-19 14:57:38.185876943 +0000 UTC m=+0.401814808,LastTimestamp:2026-07-19 14:57:38.185876943 +0000 UTC m=+0.401814808,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-138-220.ec2.internal,}" Jul 19 14:57:38.198512 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:38.198495 2558 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Jul 19 14:57:38.198596 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:38.198562 2558 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Jul 19 14:57:38.199151 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:38.199135 2558 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Jul 19 14:57:38.199151 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:38.199140 2558 volume_manager.go:311] "The desired_state_of_world populator starts" Jul 19 14:57:38.199297 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:38.199173 2558 volume_manager.go:313] "Starting Kubelet Volume Manager" Jul 19 14:57:38.199830 ip-10-0-138-220 kubenswrapper[2558]: E0719 14:57:38.198615 2558 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-220.ec2.internal\" not found" Jul 19 14:57:38.199911 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:38.199255 2558 reconstruct.go:97] "Volume reconstruction finished" Jul 19 14:57:38.199911 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:38.199848 2558 reconciler.go:29] "Reconciler: start to sync state" Jul 19 14:57:38.200426 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:38.200395 2558 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Jul 19 14:57:38.200426 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:38.200418 2558 factory.go:55] Registering systemd factory Jul 19 14:57:38.200426 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:38.200426 2558 factory.go:223] Registration of the systemd container factory successfully Jul 19 14:57:38.200714 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:38.200700 2558 factory.go:153] Registering CRI-O factory Jul 19 14:57:38.200858 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:38.200847 2558 factory.go:223] Registration of the crio container factory successfully Jul 19 14:57:38.201060 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:38.201051 2558 factory.go:103] Registering Raw factory Jul 19 14:57:38.201134 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:38.201113 2558 manager.go:1196] Started watching for new ooms in manager Jul 19 14:57:38.201903 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:38.201888 2558 manager.go:349] Starting recovery of all containers Jul 19 14:57:38.209002 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:38.208707 2558 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 19 14:57:38.211061 ip-10-0-138-220 kubenswrapper[2558]: E0719 14:57:38.211041 2558 nodelease.go:49] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"ip-10-0-138-220.ec2.internal\" not found" node="ip-10-0-138-220.ec2.internal" Jul 19 14:57:38.214196 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:38.214179 2558 manager.go:354] Recovery completed Jul 19 14:57:38.218120 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:38.218107 2558 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 19 14:57:38.220248 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:38.220232 2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-220.ec2.internal" event="NodeHasSufficientMemory" Jul 19 14:57:38.220306 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:38.220260 2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-220.ec2.internal" event="NodeHasNoDiskPressure" Jul 19 14:57:38.220306 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:38.220270 2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-220.ec2.internal" event="NodeHasSufficientPID" Jul 19 14:57:38.220833 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:38.220817 2558 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 19 14:57:38.220891 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:38.220818 2558 cpu_manager.go:222] "Starting CPU manager" policy="none" Jul 19 14:57:38.220891 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:38.220872 2558 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Jul 19 14:57:38.220891 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:38.220891 2558 state_mem.go:36] "Initialized new in-memory state store" Jul 19 14:57:38.221499 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:38.221486 2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-220.ec2.internal" event="NodeHasSufficientMemory" Jul 19 14:57:38.221564 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:38.221510 2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-220.ec2.internal" event="NodeHasNoDiskPressure" Jul 19 14:57:38.221564 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:38.221519 2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-220.ec2.internal" event="NodeHasSufficientPID" Jul 19 14:57:38.223058 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:38.222967 2558 policy_none.go:49] "None policy: Start" Jul 19 14:57:38.223107 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:38.223067 2558 memory_manager.go:187] "Starting memorymanager" policy="None" Jul 19 14:57:38.223107 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:38.223078 2558 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Jul 19 14:57:38.225018 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:38.225008 2558 policy_none.go:47] "Start" Jul 19 14:57:38.276278 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:38.263371 2558 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Jul 19 14:57:38.276278 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:38.264649 2558 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Jul 19 14:57:38.276278 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:38.264687 2558 status_manager.go:244] "Starting to sync pod status with apiserver" Jul 19 14:57:38.276278 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:38.264709 2558 kubelet.go:2448] "Starting kubelet main sync loop" Jul 19 14:57:38.276278 ip-10-0-138-220 kubenswrapper[2558]: E0719 14:57:38.264755 2558 kubelet.go:2472] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]" Jul 19 14:57:38.276278 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:38.265543 2558 manager.go:337] "Starting Device Plugin manager" Jul 19 14:57:38.276278 ip-10-0-138-220 kubenswrapper[2558]: E0719 14:57:38.265572 2558 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Jul 19 14:57:38.276278 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:38.265584 2558 server.go:87] "Starting device plugin registration server" Jul 19 14:57:38.276278 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:38.265842 2558 eviction_manager.go:189] "Eviction manager: starting control loop" Jul 19 14:57:38.276278 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:38.266409 2558 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Jul 19 14:57:38.276278 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:38.266497 2558 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Jul 19 14:57:38.276278 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:38.266582 2558 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Jul 19 14:57:38.276278 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:38.266592 2558 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Jul 19 14:57:38.276278 ip-10-0-138-220 kubenswrapper[2558]: E0719 14:57:38.267038 2558 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Jul 19 14:57:38.276278 ip-10-0-138-220 kubenswrapper[2558]: E0719 14:57:38.267083 2558 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-138-220.ec2.internal\" not found" Jul 19 14:57:38.276278 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:38.267095 2558 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 19 14:57:38.365928 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:38.365860 2558 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-220.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-138-220.ec2.internal"] Jul 19 14:57:38.365990 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:38.365943 2558 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 19 14:57:38.367319 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:38.367306 2558 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 19 14:57:38.367532 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:38.367516 2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-220.ec2.internal" event="NodeHasSufficientMemory" Jul 19 14:57:38.367609 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:38.367548 2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-220.ec2.internal" event="NodeHasNoDiskPressure" Jul 19 14:57:38.367609 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:38.367563 2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-220.ec2.internal" event="NodeHasSufficientPID" Jul 19 14:57:38.368027 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:38.368003 2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-220.ec2.internal" event="NodeHasSufficientMemory" Jul 19 14:57:38.368088 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:38.368038 2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-220.ec2.internal" event="NodeHasNoDiskPressure" Jul 19 14:57:38.368088 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:38.368048 2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-220.ec2.internal" event="NodeHasSufficientPID" Jul 19 14:57:38.368088 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:38.368069 2558 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-138-220.ec2.internal" Jul 19 14:57:38.369923 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:38.369908 2558 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 19 14:57:38.370039 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:38.370027 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-220.ec2.internal" Jul 19 14:57:38.370083 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:38.370054 2558 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 19 14:57:38.370560 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:38.370544 2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-220.ec2.internal" event="NodeHasSufficientMemory" Jul 19 14:57:38.370626 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:38.370573 2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-220.ec2.internal" event="NodeHasNoDiskPressure" Jul 19 14:57:38.370626 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:38.370548 2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-220.ec2.internal" event="NodeHasSufficientMemory" Jul 19 14:57:38.370626 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:38.370587 2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-220.ec2.internal" event="NodeHasSufficientPID" Jul 19 14:57:38.370626 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:38.370605 2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-220.ec2.internal" event="NodeHasNoDiskPressure" Jul 19 14:57:38.370626 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:38.370618 2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-220.ec2.internal" event="NodeHasSufficientPID" Jul 19 14:57:38.372938 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:38.372921 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-220.ec2.internal" Jul 19 14:57:38.373017 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:38.372952 2558 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 19 14:57:38.373577 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:38.373561 2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-220.ec2.internal" event="NodeHasSufficientMemory" Jul 19 14:57:38.373662 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:38.373583 2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-220.ec2.internal" event="NodeHasNoDiskPressure" Jul 19 14:57:38.373662 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:38.373593 2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-220.ec2.internal" event="NodeHasSufficientPID" Jul 19 14:57:38.375633 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:38.375607 2558 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-138-220.ec2.internal" Jul 19 14:57:38.375633 ip-10-0-138-220 kubenswrapper[2558]: E0719 14:57:38.375628 2558 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-138-220.ec2.internal\": node \"ip-10-0-138-220.ec2.internal\" not found" Jul 19 14:57:38.389696 ip-10-0-138-220 kubenswrapper[2558]: E0719 14:57:38.389681 2558 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-220.ec2.internal\" not found" Jul 19 14:57:38.402335 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:38.402318 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/0d2ca3cee4882a3f29ec3808e1bd3fba-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-138-220.ec2.internal\" (UID: \"0d2ca3cee4882a3f29ec3808e1bd3fba\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-220.ec2.internal" Jul 19 14:57:38.402411 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:38.402341 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0d2ca3cee4882a3f29ec3808e1bd3fba-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-138-220.ec2.internal\" (UID: \"0d2ca3cee4882a3f29ec3808e1bd3fba\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-220.ec2.internal" Jul 19 14:57:38.402411 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:38.402361 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/e02b018e3a25db1c9e92391d89932378-config\") pod \"kube-apiserver-proxy-ip-10-0-138-220.ec2.internal\" (UID: \"e02b018e3a25db1c9e92391d89932378\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-138-220.ec2.internal" Jul 19 14:57:38.410347 ip-10-0-138-220 kubenswrapper[2558]: E0719 14:57:38.410328 2558 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-138-220.ec2.internal\" not found" node="ip-10-0-138-220.ec2.internal" Jul 19 14:57:38.414470 ip-10-0-138-220 kubenswrapper[2558]: E0719 14:57:38.414456 2558 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-138-220.ec2.internal\" not found" node="ip-10-0-138-220.ec2.internal" Jul 19 14:57:38.490367 ip-10-0-138-220 kubenswrapper[2558]: E0719 14:57:38.490351 2558 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-220.ec2.internal\" not found" Jul 19 14:57:38.502931 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:38.502913 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/e02b018e3a25db1c9e92391d89932378-config\") pod \"kube-apiserver-proxy-ip-10-0-138-220.ec2.internal\" (UID: \"e02b018e3a25db1c9e92391d89932378\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-138-220.ec2.internal" Jul 19 14:57:38.503036 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:38.502943 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/0d2ca3cee4882a3f29ec3808e1bd3fba-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-138-220.ec2.internal\" (UID: \"0d2ca3cee4882a3f29ec3808e1bd3fba\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-220.ec2.internal" Jul 19 14:57:38.503036 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:38.502969 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0d2ca3cee4882a3f29ec3808e1bd3fba-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-138-220.ec2.internal\" (UID: \"0d2ca3cee4882a3f29ec3808e1bd3fba\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-220.ec2.internal" Jul 19 14:57:38.503036 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:38.503004 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0d2ca3cee4882a3f29ec3808e1bd3fba-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-138-220.ec2.internal\" (UID: \"0d2ca3cee4882a3f29ec3808e1bd3fba\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-220.ec2.internal" Jul 19 14:57:38.503036 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:38.503024 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/e02b018e3a25db1c9e92391d89932378-config\") pod \"kube-apiserver-proxy-ip-10-0-138-220.ec2.internal\" (UID: \"e02b018e3a25db1c9e92391d89932378\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-138-220.ec2.internal" Jul 19 14:57:38.503213 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:38.503027 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/0d2ca3cee4882a3f29ec3808e1bd3fba-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-138-220.ec2.internal\" (UID: \"0d2ca3cee4882a3f29ec3808e1bd3fba\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-220.ec2.internal" Jul 19 14:57:38.591292 ip-10-0-138-220 kubenswrapper[2558]: E0719 14:57:38.591257 2558 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-220.ec2.internal\" not found" Jul 19 14:57:38.692072 ip-10-0-138-220 kubenswrapper[2558]: E0719 14:57:38.692005 2558 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-220.ec2.internal\" not found" Jul 19 14:57:38.714532 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:38.714510 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-220.ec2.internal" Jul 19 14:57:38.718286 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:38.718272 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-220.ec2.internal" Jul 19 14:57:38.792708 ip-10-0-138-220 kubenswrapper[2558]: E0719 14:57:38.792684 2558 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-220.ec2.internal\" not found" Jul 19 14:57:38.893250 ip-10-0-138-220 kubenswrapper[2558]: E0719 14:57:38.893225 2558 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-220.ec2.internal\" not found" Jul 19 14:57:38.993752 ip-10-0-138-220 kubenswrapper[2558]: E0719 14:57:38.993694 2558 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-220.ec2.internal\" not found" Jul 19 14:57:39.043645 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:39.043620 2558 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Jul 19 14:57:39.053301 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:39.053278 2558 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Jul 19 14:57:39.099017 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:39.098996 2558 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-220.ec2.internal" Jul 19 14:57:39.109306 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:39.109288 2558 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 19 14:57:39.110324 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:39.110311 2558 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-220.ec2.internal" Jul 19 14:57:39.120333 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:39.120313 2558 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 19 14:57:39.130479 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:39.130461 2558 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Jul 19 14:57:39.130599 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:39.130569 2558 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 19 14:57:39.130688 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:39.130604 2558 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 19 14:57:39.130688 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:39.130604 2558 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 19 14:57:39.130688 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:39.130621 2558 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 19 14:57:39.175800 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:39.175761 2558 apiserver.go:52] "Watching apiserver" Jul 19 14:57:39.189414 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:39.189386 2558 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Jul 19 14:57:39.189759 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:39.189739 2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/global-pull-secret-syncer-kj72j","kube-system/kube-apiserver-proxy-ip-10-0-138-220.ec2.internal","openshift-dns/node-resolver-qsm4z","openshift-image-registry/node-ca-tnrrh","openshift-multus/network-metrics-daemon-g52v5","openshift-network-diagnostics/network-check-target-xpl4t","kube-system/konnectivity-agent-t8n4w","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tf7zt","openshift-cluster-node-tuning-operator/tuned-4tgkg","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-220.ec2.internal","openshift-multus/multus-additional-cni-plugins-vzds8","openshift-multus/multus-jhf4r","openshift-network-operator/iptables-alerter-q2fq8","openshift-ovn-kubernetes/ovnkube-node-5wfzs"] Jul 19 14:57:39.192242 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:39.192228 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-t8n4w" Jul 19 14:57:39.194267 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:39.194239 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Jul 19 14:57:39.194360 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:39.194292 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Jul 19 14:57:39.194360 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:39.194310 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-bq7zq\"" Jul 19 14:57:39.194360 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:39.194312 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-q2fq8" Jul 19 14:57:39.196123 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:39.196064 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Jul 19 14:57:39.196123 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:39.196077 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-59kt9\"" Jul 19 14:57:39.196258 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:39.196173 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Jul 19 14:57:39.196258 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:39.196220 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Jul 19 14:57:39.196351 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:39.196340 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-qsm4z" Jul 19 14:57:39.197569 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:39.197525 2558 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-18 14:52:39 +0000 UTC" deadline="2028-02-17 09:28:54.671861699 +0000 UTC" Jul 19 14:57:39.197569 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:39.197560 2558 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13866h31m15.474305142s" Jul 19 14:57:39.198035 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:39.197987 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Jul 19 14:57:39.198194 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:39.198181 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-2z896\"" Jul 19 14:57:39.198291 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:39.198206 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Jul 19 14:57:39.198495 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:39.198483 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-tnrrh" Jul 19 14:57:39.198623 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:39.198613 2558 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Jul 19 14:57:39.200171 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:39.200148 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Jul 19 14:57:39.200423 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:39.200405 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Jul 19 14:57:39.200503 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:39.200495 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Jul 19 14:57:39.200570 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:39.200507 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-qknzm\"" Jul 19 14:57:39.200615 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:39.200578 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-g52v5" Jul 19 14:57:39.200691 ip-10-0-138-220 kubenswrapper[2558]: E0719 14:57:39.200640 2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-g52v5" podUID="c9a1c57d-99f6-49a1-bd75-d00fcffc86f1" Jul 19 14:57:39.202707 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:39.202689 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xpl4t" Jul 19 14:57:39.202770 ip-10-0-138-220 kubenswrapper[2558]: E0719 14:57:39.202753 2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xpl4t" podUID="11a5463d-a4e8-47d2-8791-5a10ea52cfc2" Jul 19 14:57:39.204862 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:39.204849 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-4tgkg" Jul 19 14:57:39.206642 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:39.206604 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l7mlf\" (UniqueName: \"kubernetes.io/projected/8aa5107a-42c8-4d25-a245-8a151e5b2514-kube-api-access-l7mlf\") pod \"iptables-alerter-q2fq8\" (UID: \"8aa5107a-42c8-4d25-a245-8a151e5b2514\") " pod="openshift-network-operator/iptables-alerter-q2fq8" Jul 19 14:57:39.206642 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:39.206638 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Jul 19 14:57:39.206813 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:39.206657 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fph4j\" (UniqueName: \"kubernetes.io/projected/8c1b2505-369a-4989-ac89-344f9d9991f9-kube-api-access-fph4j\") pod \"node-resolver-qsm4z\" (UID: \"8c1b2505-369a-4989-ac89-344f9d9991f9\") " pod="openshift-dns/node-resolver-qsm4z" Jul 19 14:57:39.206813 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:39.206705 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c9a1c57d-99f6-49a1-bd75-d00fcffc86f1-metrics-certs\") pod \"network-metrics-daemon-g52v5\" (UID: \"c9a1c57d-99f6-49a1-bd75-d00fcffc86f1\") " pod="openshift-multus/network-metrics-daemon-g52v5" Jul 19 14:57:39.206813 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:39.206718 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Jul 19 14:57:39.206813 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:39.206732 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z7zfq\" (UniqueName: \"kubernetes.io/projected/c9a1c57d-99f6-49a1-bd75-d00fcffc86f1-kube-api-access-z7zfq\") pod \"network-metrics-daemon-g52v5\" (UID: \"c9a1c57d-99f6-49a1-bd75-d00fcffc86f1\") " pod="openshift-multus/network-metrics-daemon-g52v5" Jul 19 14:57:39.206813 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:39.206800 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/8aa5107a-42c8-4d25-a245-8a151e5b2514-iptables-alerter-script\") pod \"iptables-alerter-q2fq8\" (UID: \"8aa5107a-42c8-4d25-a245-8a151e5b2514\") " pod="openshift-network-operator/iptables-alerter-q2fq8" Jul 19 14:57:39.207040 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:39.206861 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-xvqbp\"" Jul 19 14:57:39.207040 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:39.206865 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/8aa5107a-42c8-4d25-a245-8a151e5b2514-host-slash\") pod \"iptables-alerter-q2fq8\" (UID: \"8aa5107a-42c8-4d25-a245-8a151e5b2514\") " pod="openshift-network-operator/iptables-alerter-q2fq8" Jul 19 14:57:39.207040 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:39.206896 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/8c1b2505-369a-4989-ac89-344f9d9991f9-hosts-file\") pod \"node-resolver-qsm4z\" (UID: \"8c1b2505-369a-4989-ac89-344f9d9991f9\") " pod="openshift-dns/node-resolver-qsm4z" Jul 19 14:57:39.207040 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:39.206920 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/80a8f70e-f1a5-408d-bb95-b347a0ec8715-serviceca\") pod \"node-ca-tnrrh\" (UID: \"80a8f70e-f1a5-408d-bb95-b347a0ec8715\") " pod="openshift-image-registry/node-ca-tnrrh" Jul 19 14:57:39.207040 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:39.206945 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dt474\" (UniqueName: \"kubernetes.io/projected/80a8f70e-f1a5-408d-bb95-b347a0ec8715-kube-api-access-dt474\") pod \"node-ca-tnrrh\" (UID: \"80a8f70e-f1a5-408d-bb95-b347a0ec8715\") " pod="openshift-image-registry/node-ca-tnrrh" Jul 19 14:57:39.207040 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:39.206968 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8c1b2505-369a-4989-ac89-344f9d9991f9-tmp-dir\") pod \"node-resolver-qsm4z\" (UID: \"8c1b2505-369a-4989-ac89-344f9d9991f9\") " pod="openshift-dns/node-resolver-qsm4z" Jul 19 14:57:39.207040 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:39.206990 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/80a8f70e-f1a5-408d-bb95-b347a0ec8715-host\") pod \"node-ca-tnrrh\" (UID: \"80a8f70e-f1a5-408d-bb95-b347a0ec8715\") " pod="openshift-image-registry/node-ca-tnrrh" Jul 19 14:57:39.207040 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:39.207015 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/5f0a3171-e26a-47ef-b546-078b7c259d3e-agent-certs\") pod \"konnectivity-agent-t8n4w\" (UID: \"5f0a3171-e26a-47ef-b546-078b7c259d3e\") " pod="kube-system/konnectivity-agent-t8n4w" Jul 19 14:57:39.207311 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:39.207048 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/5f0a3171-e26a-47ef-b546-078b7c259d3e-konnectivity-ca\") pod \"konnectivity-agent-t8n4w\" (UID: \"5f0a3171-e26a-47ef-b546-078b7c259d3e\") " pod="kube-system/konnectivity-agent-t8n4w" Jul 19 14:57:39.207311 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:39.207068 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kj72j" Jul 19 14:57:39.207311 ip-10-0-138-220 kubenswrapper[2558]: E0719 14:57:39.207132 2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kj72j" podUID="5994a848-5c3c-4d6c-898c-bb659e1a41b2" Jul 19 14:57:39.209344 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:39.209324 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-5wfzs" Jul 19 14:57:39.213733 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:39.213215 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tf7zt" Jul 19 14:57:39.213733 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:39.213342 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Jul 19 14:57:39.213733 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:39.213406 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Jul 19 14:57:39.213733 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:39.213472 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Jul 19 14:57:39.213963 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:39.213905 2558 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 19 14:57:39.213963 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:39.213936 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Jul 19 14:57:39.214139 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:39.214111 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Jul 19 14:57:39.214191 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:39.214122 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-hzdkc\"" Jul 19 14:57:39.214385 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:39.214369 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Jul 19 14:57:39.215094 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:39.215073 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Jul 19 14:57:39.215559 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:39.215545 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Jul 19 14:57:39.215620 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:39.215562 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-nxwfw\"" Jul 19 14:57:39.215781 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:39.215769 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Jul 19 14:57:39.216042 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:39.216025 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-vzds8" Jul 19 14:57:39.217967 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:39.217949 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Jul 19 14:57:39.217967 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:39.217959 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Jul 19 14:57:39.218597 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:39.218168 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Jul 19 14:57:39.218597 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:39.218190 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Jul 19 14:57:39.218597 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:39.218214 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-qntgh\"" Jul 19 14:57:39.218597 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:39.218222 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Jul 19 14:57:39.218597 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:39.218347 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-jhf4r" Jul 19 14:57:39.220023 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:39.220009 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-4c9qp\"" Jul 19 14:57:39.220143 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:39.220130 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Jul 19 14:57:39.238035 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:39.238014 2558 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-5v46m" Jul 19 14:57:39.245442 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:39.245402 2558 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-5v46m" Jul 19 14:57:39.278079 ip-10-0-138-220 kubenswrapper[2558]: W0719 14:57:39.278052 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0d2ca3cee4882a3f29ec3808e1bd3fba.slice/crio-8bff91165da9cb2d7a47730b9616b55b1a9d919d4b35a31d4346d84c8dba66f5 WatchSource:0}: Error finding container 8bff91165da9cb2d7a47730b9616b55b1a9d919d4b35a31d4346d84c8dba66f5: Status 404 returned error can't find the container with id 8bff91165da9cb2d7a47730b9616b55b1a9d919d4b35a31d4346d84c8dba66f5 Jul 19 14:57:39.278277 ip-10-0-138-220 kubenswrapper[2558]: W0719 14:57:39.278262 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode02b018e3a25db1c9e92391d89932378.slice/crio-b749e900b62f843321c8f11b3f690e5d9e8cb78f60bbe178144e63ecf25dddf1 WatchSource:0}: Error finding container b749e900b62f843321c8f11b3f690e5d9e8cb78f60bbe178144e63ecf25dddf1: Status 404 returned error can't find the container with id b749e900b62f843321c8f11b3f690e5d9e8cb78f60bbe178144e63ecf25dddf1 Jul 19 14:57:39.285634 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:39.285618 2558 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 19 14:57:39.299937 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:39.299923 2558 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Jul 19 14:57:39.308135 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:39.308116 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/8aa5107a-42c8-4d25-a245-8a151e5b2514-host-slash\") pod \"iptables-alerter-q2fq8\" (UID: \"8aa5107a-42c8-4d25-a245-8a151e5b2514\") " pod="openshift-network-operator/iptables-alerter-q2fq8" Jul 19 14:57:39.308225 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:39.308142 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dt474\" (UniqueName: \"kubernetes.io/projected/80a8f70e-f1a5-408d-bb95-b347a0ec8715-kube-api-access-dt474\") pod \"node-ca-tnrrh\" (UID: \"80a8f70e-f1a5-408d-bb95-b347a0ec8715\") " pod="openshift-image-registry/node-ca-tnrrh" Jul 19 14:57:39.308225 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:39.308162 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/1367ccc0-45ba-4d3f-9892-40226e1bdfec-etc-systemd\") pod \"tuned-4tgkg\" (UID: \"1367ccc0-45ba-4d3f-9892-40226e1bdfec\") " pod="openshift-cluster-node-tuning-operator/tuned-4tgkg" Jul 19 14:57:39.308225 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:39.308177 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/5b2d3a07-86b2-4ba2-8a71-c2d89ada79c6-host-cni-bin\") pod \"ovnkube-node-5wfzs\" (UID: \"5b2d3a07-86b2-4ba2-8a71-c2d89ada79c6\") " pod="openshift-ovn-kubernetes/ovnkube-node-5wfzs" Jul 19 14:57:39.308225 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:39.308194 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/75da72bf-18f9-4965-9bab-b355cb1d3263-registration-dir\") pod \"aws-ebs-csi-driver-node-tf7zt\" (UID: \"75da72bf-18f9-4965-9bab-b355cb1d3263\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tf7zt" Jul 19 14:57:39.308380 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:39.308233 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/8aa5107a-42c8-4d25-a245-8a151e5b2514-host-slash\") pod \"iptables-alerter-q2fq8\" (UID: \"8aa5107a-42c8-4d25-a245-8a151e5b2514\") " pod="openshift-network-operator/iptables-alerter-q2fq8" Jul 19 14:57:39.308380 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:39.308265 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/5994a848-5c3c-4d6c-898c-bb659e1a41b2-kubelet-config\") pod \"global-pull-secret-syncer-kj72j\" (UID: \"5994a848-5c3c-4d6c-898c-bb659e1a41b2\") " pod="kube-system/global-pull-secret-syncer-kj72j" Jul 19 14:57:39.308380 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:39.308311 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5994a848-5c3c-4d6c-898c-bb659e1a41b2-original-pull-secret\") pod \"global-pull-secret-syncer-kj72j\" (UID: \"5994a848-5c3c-4d6c-898c-bb659e1a41b2\") " pod="kube-system/global-pull-secret-syncer-kj72j" Jul 19 14:57:39.308380 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:39.308338 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1367ccc0-45ba-4d3f-9892-40226e1bdfec-etc-kubernetes\") pod \"tuned-4tgkg\" (UID: \"1367ccc0-45ba-4d3f-9892-40226e1bdfec\") " pod="openshift-cluster-node-tuning-operator/tuned-4tgkg" Jul 19 14:57:39.308380 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:39.308360 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/1367ccc0-45ba-4d3f-9892-40226e1bdfec-host\") pod \"tuned-4tgkg\" (UID: \"1367ccc0-45ba-4d3f-9892-40226e1bdfec\") " pod="openshift-cluster-node-tuning-operator/tuned-4tgkg" Jul 19 14:57:39.308546 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:39.308382 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/5b2d3a07-86b2-4ba2-8a71-c2d89ada79c6-host-kubelet\") pod \"ovnkube-node-5wfzs\" (UID: \"5b2d3a07-86b2-4ba2-8a71-c2d89ada79c6\") " pod="openshift-ovn-kubernetes/ovnkube-node-5wfzs" Jul 19 14:57:39.308546 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:39.308404 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/5b2d3a07-86b2-4ba2-8a71-c2d89ada79c6-host-slash\") pod \"ovnkube-node-5wfzs\" (UID: \"5b2d3a07-86b2-4ba2-8a71-c2d89ada79c6\") " pod="openshift-ovn-kubernetes/ovnkube-node-5wfzs" Jul 19 14:57:39.308546 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:39.308427 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/75da72bf-18f9-4965-9bab-b355cb1d3263-device-dir\") pod \"aws-ebs-csi-driver-node-tf7zt\" (UID: \"75da72bf-18f9-4965-9bab-b355cb1d3263\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tf7zt" Jul 19 14:57:39.308546 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:39.308468 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/75da72bf-18f9-4965-9bab-b355cb1d3263-sys-fs\") pod \"aws-ebs-csi-driver-node-tf7zt\" (UID: \"75da72bf-18f9-4965-9bab-b355cb1d3263\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tf7zt" Jul 19 14:57:39.308546 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:39.308489 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/5f0a3171-e26a-47ef-b546-078b7c259d3e-konnectivity-ca\") pod \"konnectivity-agent-t8n4w\" (UID: \"5f0a3171-e26a-47ef-b546-078b7c259d3e\") " pod="kube-system/konnectivity-agent-t8n4w" Jul 19 14:57:39.308546 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:39.308506 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z7zfq\" (UniqueName: \"kubernetes.io/projected/c9a1c57d-99f6-49a1-bd75-d00fcffc86f1-kube-api-access-z7zfq\") pod \"network-metrics-daemon-g52v5\" (UID: \"c9a1c57d-99f6-49a1-bd75-d00fcffc86f1\") " pod="openshift-multus/network-metrics-daemon-g52v5" Jul 19 14:57:39.308546 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:39.308527 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5a78ce34-491f-4ae5-971b-65d650736dff-system-cni-dir\") pod \"multus-jhf4r\" (UID: \"5a78ce34-491f-4ae5-971b-65d650736dff\") " pod="openshift-multus/multus-jhf4r" Jul 19 14:57:39.308850 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:39.308548 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/5a78ce34-491f-4ae5-971b-65d650736dff-cnibin\") pod \"multus-jhf4r\" (UID: \"5a78ce34-491f-4ae5-971b-65d650736dff\") " pod="openshift-multus/multus-jhf4r" Jul 19 14:57:39.308850 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:39.308569 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/5a78ce34-491f-4ae5-971b-65d650736dff-host-run-k8s-cni-cncf-io\") pod \"multus-jhf4r\" (UID: \"5a78ce34-491f-4ae5-971b-65d650736dff\") " pod="openshift-multus/multus-jhf4r" Jul 19 14:57:39.308850 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:39.308590 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/5994a848-5c3c-4d6c-898c-bb659e1a41b2-dbus\") pod \"global-pull-secret-syncer-kj72j\" (UID: \"5994a848-5c3c-4d6c-898c-bb659e1a41b2\") " pod="kube-system/global-pull-secret-syncer-kj72j" Jul 19 14:57:39.308850 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:39.308610 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/1367ccc0-45ba-4d3f-9892-40226e1bdfec-run\") pod \"tuned-4tgkg\" (UID: \"1367ccc0-45ba-4d3f-9892-40226e1bdfec\") " pod="openshift-cluster-node-tuning-operator/tuned-4tgkg" Jul 19 14:57:39.308850 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:39.308648 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/1367ccc0-45ba-4d3f-9892-40226e1bdfec-lib-modules\") pod \"tuned-4tgkg\" (UID: \"1367ccc0-45ba-4d3f-9892-40226e1bdfec\") " pod="openshift-cluster-node-tuning-operator/tuned-4tgkg" Jul 19 14:57:39.308850 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:39.308692 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/5b2d3a07-86b2-4ba2-8a71-c2d89ada79c6-systemd-units\") pod \"ovnkube-node-5wfzs\" (UID: \"5b2d3a07-86b2-4ba2-8a71-c2d89ada79c6\") " pod="openshift-ovn-kubernetes/ovnkube-node-5wfzs" Jul 19 14:57:39.308850 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:39.308719 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5b2d3a07-86b2-4ba2-8a71-c2d89ada79c6-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-5wfzs\" (UID: \"5b2d3a07-86b2-4ba2-8a71-c2d89ada79c6\") " pod="openshift-ovn-kubernetes/ovnkube-node-5wfzs" Jul 19 14:57:39.308850 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:39.308754 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/5b2d3a07-86b2-4ba2-8a71-c2d89ada79c6-env-overrides\") pod \"ovnkube-node-5wfzs\" (UID: \"5b2d3a07-86b2-4ba2-8a71-c2d89ada79c6\") " pod="openshift-ovn-kubernetes/ovnkube-node-5wfzs" Jul 19 14:57:39.308850 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:39.308776 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/5a78ce34-491f-4ae5-971b-65d650736dff-os-release\") pod \"multus-jhf4r\" (UID: \"5a78ce34-491f-4ae5-971b-65d650736dff\") " pod="openshift-multus/multus-jhf4r" Jul 19 14:57:39.308850 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:39.308810 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/80a8f70e-f1a5-408d-bb95-b347a0ec8715-serviceca\") pod \"node-ca-tnrrh\" (UID: \"80a8f70e-f1a5-408d-bb95-b347a0ec8715\") " pod="openshift-image-registry/node-ca-tnrrh" Jul 19 14:57:39.308850 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:39.308834 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/5b2d3a07-86b2-4ba2-8a71-c2d89ada79c6-ovn-node-metrics-cert\") pod \"ovnkube-node-5wfzs\" (UID: \"5b2d3a07-86b2-4ba2-8a71-c2d89ada79c6\") " pod="openshift-ovn-kubernetes/ovnkube-node-5wfzs" Jul 19 14:57:39.309184 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:39.308866 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x2cg5\" (UniqueName: \"kubernetes.io/projected/5b2d3a07-86b2-4ba2-8a71-c2d89ada79c6-kube-api-access-x2cg5\") pod \"ovnkube-node-5wfzs\" (UID: \"5b2d3a07-86b2-4ba2-8a71-c2d89ada79c6\") " pod="openshift-ovn-kubernetes/ovnkube-node-5wfzs" Jul 19 14:57:39.309184 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:39.308888 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5a78ce34-491f-4ae5-971b-65d650736dff-multus-cni-dir\") pod \"multus-jhf4r\" (UID: \"5a78ce34-491f-4ae5-971b-65d650736dff\") " pod="openshift-multus/multus-jhf4r" Jul 19 14:57:39.309184 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:39.308909 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/5a78ce34-491f-4ae5-971b-65d650736dff-host-var-lib-cni-multus\") pod \"multus-jhf4r\" (UID: \"5a78ce34-491f-4ae5-971b-65d650736dff\") " pod="openshift-multus/multus-jhf4r" Jul 19 14:57:39.309184 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:39.308937 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vglps\" (UniqueName: \"kubernetes.io/projected/5a78ce34-491f-4ae5-971b-65d650736dff-kube-api-access-vglps\") pod \"multus-jhf4r\" (UID: \"5a78ce34-491f-4ae5-971b-65d650736dff\") " pod="openshift-multus/multus-jhf4r" Jul 19 14:57:39.309184 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:39.308947 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/5f0a3171-e26a-47ef-b546-078b7c259d3e-konnectivity-ca\") pod \"konnectivity-agent-t8n4w\" (UID: \"5f0a3171-e26a-47ef-b546-078b7c259d3e\") " pod="kube-system/konnectivity-agent-t8n4w" Jul 19 14:57:39.309184 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:39.308960 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4l4fp\" (UniqueName: \"kubernetes.io/projected/2571b663-0e56-4668-9831-fcdaf3793a1f-kube-api-access-4l4fp\") pod \"multus-additional-cni-plugins-vzds8\" (UID: \"2571b663-0e56-4668-9831-fcdaf3793a1f\") " pod="openshift-multus/multus-additional-cni-plugins-vzds8" Jul 19 14:57:39.309184 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:39.309010 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/5b2d3a07-86b2-4ba2-8a71-c2d89ada79c6-ovnkube-config\") pod \"ovnkube-node-5wfzs\" (UID: \"5b2d3a07-86b2-4ba2-8a71-c2d89ada79c6\") " pod="openshift-ovn-kubernetes/ovnkube-node-5wfzs" Jul 19 14:57:39.309184 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:39.309032 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l7mlf\" (UniqueName: \"kubernetes.io/projected/8aa5107a-42c8-4d25-a245-8a151e5b2514-kube-api-access-l7mlf\") pod \"iptables-alerter-q2fq8\" (UID: \"8aa5107a-42c8-4d25-a245-8a151e5b2514\") " pod="openshift-network-operator/iptables-alerter-q2fq8" Jul 19 14:57:39.309184 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:39.309048 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c9a1c57d-99f6-49a1-bd75-d00fcffc86f1-metrics-certs\") pod \"network-metrics-daemon-g52v5\" (UID: \"c9a1c57d-99f6-49a1-bd75-d00fcffc86f1\") " pod="openshift-multus/network-metrics-daemon-g52v5" Jul 19 14:57:39.309184 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:39.309063 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/5a78ce34-491f-4ae5-971b-65d650736dff-host-run-multus-certs\") pod \"multus-jhf4r\" (UID: \"5a78ce34-491f-4ae5-971b-65d650736dff\") " pod="openshift-multus/multus-jhf4r" Jul 19 14:57:39.309184 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:39.309078 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/2571b663-0e56-4668-9831-fcdaf3793a1f-cni-binary-copy\") pod \"multus-additional-cni-plugins-vzds8\" (UID: \"2571b663-0e56-4668-9831-fcdaf3793a1f\") " pod="openshift-multus/multus-additional-cni-plugins-vzds8" Jul 19 14:57:39.309184 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:39.309113 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/5b2d3a07-86b2-4ba2-8a71-c2d89ada79c6-etc-openvswitch\") pod \"ovnkube-node-5wfzs\" (UID: \"5b2d3a07-86b2-4ba2-8a71-c2d89ada79c6\") " pod="openshift-ovn-kubernetes/ovnkube-node-5wfzs" Jul 19 14:57:39.309184 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:39.309133 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/5b2d3a07-86b2-4ba2-8a71-c2d89ada79c6-run-openvswitch\") pod \"ovnkube-node-5wfzs\" (UID: \"5b2d3a07-86b2-4ba2-8a71-c2d89ada79c6\") " pod="openshift-ovn-kubernetes/ovnkube-node-5wfzs" Jul 19 14:57:39.309184 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:39.309148 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/5b2d3a07-86b2-4ba2-8a71-c2d89ada79c6-log-socket\") pod \"ovnkube-node-5wfzs\" (UID: \"5b2d3a07-86b2-4ba2-8a71-c2d89ada79c6\") " pod="openshift-ovn-kubernetes/ovnkube-node-5wfzs" Jul 19 14:57:39.309184 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:39.309164 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/75da72bf-18f9-4965-9bab-b355cb1d3263-kubelet-dir\") pod \"aws-ebs-csi-driver-node-tf7zt\" (UID: \"75da72bf-18f9-4965-9bab-b355cb1d3263\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tf7zt" Jul 19 14:57:39.309680 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:39.309199 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/80a8f70e-f1a5-408d-bb95-b347a0ec8715-serviceca\") pod \"node-ca-tnrrh\" (UID: \"80a8f70e-f1a5-408d-bb95-b347a0ec8715\") " pod="openshift-image-registry/node-ca-tnrrh" Jul 19 14:57:39.309680 ip-10-0-138-220 kubenswrapper[2558]: E0719 14:57:39.309203 2558 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 19 14:57:39.309680 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:39.309200 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2qfp8\" (UniqueName: \"kubernetes.io/projected/75da72bf-18f9-4965-9bab-b355cb1d3263-kube-api-access-2qfp8\") pod \"aws-ebs-csi-driver-node-tf7zt\" (UID: \"75da72bf-18f9-4965-9bab-b355cb1d3263\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tf7zt" Jul 19 14:57:39.309680 ip-10-0-138-220 kubenswrapper[2558]: E0719 14:57:39.309270 2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c9a1c57d-99f6-49a1-bd75-d00fcffc86f1-metrics-certs podName:c9a1c57d-99f6-49a1-bd75-d00fcffc86f1 nodeName:}" failed. No retries permitted until 2026-07-19 14:57:39.809247054 +0000 UTC m=+2.025184907 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c9a1c57d-99f6-49a1-bd75-d00fcffc86f1-metrics-certs") pod "network-metrics-daemon-g52v5" (UID: "c9a1c57d-99f6-49a1-bd75-d00fcffc86f1") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 19 14:57:39.309680 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:39.309290 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/8aa5107a-42c8-4d25-a245-8a151e5b2514-iptables-alerter-script\") pod \"iptables-alerter-q2fq8\" (UID: \"8aa5107a-42c8-4d25-a245-8a151e5b2514\") " pod="openshift-network-operator/iptables-alerter-q2fq8" Jul 19 14:57:39.309680 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:39.309306 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/8c1b2505-369a-4989-ac89-344f9d9991f9-hosts-file\") pod \"node-resolver-qsm4z\" (UID: \"8c1b2505-369a-4989-ac89-344f9d9991f9\") " pod="openshift-dns/node-resolver-qsm4z" Jul 19 14:57:39.309680 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:39.309321 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/2571b663-0e56-4668-9831-fcdaf3793a1f-cnibin\") pod \"multus-additional-cni-plugins-vzds8\" (UID: \"2571b663-0e56-4668-9831-fcdaf3793a1f\") " pod="openshift-multus/multus-additional-cni-plugins-vzds8" Jul 19 14:57:39.309680 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:39.309338 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/2571b663-0e56-4668-9831-fcdaf3793a1f-tuning-conf-dir\") pod \"multus-additional-cni-plugins-vzds8\" (UID: \"2571b663-0e56-4668-9831-fcdaf3793a1f\") " pod="openshift-multus/multus-additional-cni-plugins-vzds8" Jul 19 14:57:39.309680 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:39.309357 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1367ccc0-45ba-4d3f-9892-40226e1bdfec-var-lib-kubelet\") pod \"tuned-4tgkg\" (UID: \"1367ccc0-45ba-4d3f-9892-40226e1bdfec\") " pod="openshift-cluster-node-tuning-operator/tuned-4tgkg" Jul 19 14:57:39.309680 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:39.309371 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/75da72bf-18f9-4965-9bab-b355cb1d3263-socket-dir\") pod \"aws-ebs-csi-driver-node-tf7zt\" (UID: \"75da72bf-18f9-4965-9bab-b355cb1d3263\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tf7zt" Jul 19 14:57:39.309680 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:39.309392 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/8c1b2505-369a-4989-ac89-344f9d9991f9-hosts-file\") pod \"node-resolver-qsm4z\" (UID: \"8c1b2505-369a-4989-ac89-344f9d9991f9\") " pod="openshift-dns/node-resolver-qsm4z" Jul 19 14:57:39.309680 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:39.309396 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5a78ce34-491f-4ae5-971b-65d650736dff-host-var-lib-kubelet\") pod \"multus-jhf4r\" (UID: \"5a78ce34-491f-4ae5-971b-65d650736dff\") " pod="openshift-multus/multus-jhf4r" Jul 19 14:57:39.309680 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:39.309431 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/1367ccc0-45ba-4d3f-9892-40226e1bdfec-etc-sysconfig\") pod \"tuned-4tgkg\" (UID: \"1367ccc0-45ba-4d3f-9892-40226e1bdfec\") " pod="openshift-cluster-node-tuning-operator/tuned-4tgkg" Jul 19 14:57:39.309680 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:39.309453 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/1367ccc0-45ba-4d3f-9892-40226e1bdfec-etc-tuned\") pod \"tuned-4tgkg\" (UID: \"1367ccc0-45ba-4d3f-9892-40226e1bdfec\") " pod="openshift-cluster-node-tuning-operator/tuned-4tgkg" Jul 19 14:57:39.309680 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:39.309521 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/5b2d3a07-86b2-4ba2-8a71-c2d89ada79c6-node-log\") pod \"ovnkube-node-5wfzs\" (UID: \"5b2d3a07-86b2-4ba2-8a71-c2d89ada79c6\") " pod="openshift-ovn-kubernetes/ovnkube-node-5wfzs" Jul 19 14:57:39.309680 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:39.309554 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fph4j\" (UniqueName: \"kubernetes.io/projected/8c1b2505-369a-4989-ac89-344f9d9991f9-kube-api-access-fph4j\") pod \"node-resolver-qsm4z\" (UID: \"8c1b2505-369a-4989-ac89-344f9d9991f9\") " pod="openshift-dns/node-resolver-qsm4z" Jul 19 14:57:39.310380 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:39.309599 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-44z2s\" (UniqueName: \"kubernetes.io/projected/1367ccc0-45ba-4d3f-9892-40226e1bdfec-kube-api-access-44z2s\") pod \"tuned-4tgkg\" (UID: \"1367ccc0-45ba-4d3f-9892-40226e1bdfec\") " pod="openshift-cluster-node-tuning-operator/tuned-4tgkg" Jul 19 14:57:39.310380 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:39.309693 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/5b2d3a07-86b2-4ba2-8a71-c2d89ada79c6-run-systemd\") pod \"ovnkube-node-5wfzs\" (UID: \"5b2d3a07-86b2-4ba2-8a71-c2d89ada79c6\") " pod="openshift-ovn-kubernetes/ovnkube-node-5wfzs" Jul 19 14:57:39.310380 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:39.309728 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/5a78ce34-491f-4ae5-971b-65d650736dff-host-run-netns\") pod \"multus-jhf4r\" (UID: \"5a78ce34-491f-4ae5-971b-65d650736dff\") " pod="openshift-multus/multus-jhf4r" Jul 19 14:57:39.310380 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:39.309752 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/1367ccc0-45ba-4d3f-9892-40226e1bdfec-etc-sysctl-d\") pod \"tuned-4tgkg\" (UID: \"1367ccc0-45ba-4d3f-9892-40226e1bdfec\") " pod="openshift-cluster-node-tuning-operator/tuned-4tgkg" Jul 19 14:57:39.310380 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:39.309773 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/8aa5107a-42c8-4d25-a245-8a151e5b2514-iptables-alerter-script\") pod \"iptables-alerter-q2fq8\" (UID: \"8aa5107a-42c8-4d25-a245-8a151e5b2514\") " pod="openshift-network-operator/iptables-alerter-q2fq8" Jul 19 14:57:39.310380 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:39.309776 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5b2d3a07-86b2-4ba2-8a71-c2d89ada79c6-host-run-ovn-kubernetes\") pod \"ovnkube-node-5wfzs\" (UID: \"5b2d3a07-86b2-4ba2-8a71-c2d89ada79c6\") " pod="openshift-ovn-kubernetes/ovnkube-node-5wfzs" Jul 19 14:57:39.310380 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:39.309847 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/5b2d3a07-86b2-4ba2-8a71-c2d89ada79c6-ovnkube-script-lib\") pod \"ovnkube-node-5wfzs\" (UID: \"5b2d3a07-86b2-4ba2-8a71-c2d89ada79c6\") " pod="openshift-ovn-kubernetes/ovnkube-node-5wfzs" Jul 19 14:57:39.310380 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:39.309871 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/5a78ce34-491f-4ae5-971b-65d650736dff-cni-binary-copy\") pod \"multus-jhf4r\" (UID: \"5a78ce34-491f-4ae5-971b-65d650736dff\") " pod="openshift-multus/multus-jhf4r" Jul 19 14:57:39.310380 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:39.309914 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/5a78ce34-491f-4ae5-971b-65d650736dff-host-var-lib-cni-bin\") pod \"multus-jhf4r\" (UID: \"5a78ce34-491f-4ae5-971b-65d650736dff\") " pod="openshift-multus/multus-jhf4r" Jul 19 14:57:39.310380 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:39.309940 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/5a78ce34-491f-4ae5-971b-65d650736dff-multus-conf-dir\") pod \"multus-jhf4r\" (UID: \"5a78ce34-491f-4ae5-971b-65d650736dff\") " pod="openshift-multus/multus-jhf4r" Jul 19 14:57:39.310380 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:39.309966 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/2571b663-0e56-4668-9831-fcdaf3793a1f-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-vzds8\" (UID: \"2571b663-0e56-4668-9831-fcdaf3793a1f\") " pod="openshift-multus/multus-additional-cni-plugins-vzds8" Jul 19 14:57:39.310380 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:39.309988 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/1367ccc0-45ba-4d3f-9892-40226e1bdfec-etc-sysctl-conf\") pod \"tuned-4tgkg\" (UID: \"1367ccc0-45ba-4d3f-9892-40226e1bdfec\") " pod="openshift-cluster-node-tuning-operator/tuned-4tgkg" Jul 19 14:57:39.310380 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:39.310010 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/75da72bf-18f9-4965-9bab-b355cb1d3263-etc-selinux\") pod \"aws-ebs-csi-driver-node-tf7zt\" (UID: \"75da72bf-18f9-4965-9bab-b355cb1d3263\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tf7zt" Jul 19 14:57:39.310380 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:39.310037 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/5a78ce34-491f-4ae5-971b-65d650736dff-hostroot\") pod \"multus-jhf4r\" (UID: \"5a78ce34-491f-4ae5-971b-65d650736dff\") " pod="openshift-multus/multus-jhf4r" Jul 19 14:57:39.310380 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:39.310072 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/5a78ce34-491f-4ae5-971b-65d650736dff-multus-daemon-config\") pod \"multus-jhf4r\" (UID: \"5a78ce34-491f-4ae5-971b-65d650736dff\") " pod="openshift-multus/multus-jhf4r" Jul 19 14:57:39.310380 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:39.310095 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8c1b2505-369a-4989-ac89-344f9d9991f9-tmp-dir\") pod \"node-resolver-qsm4z\" (UID: \"8c1b2505-369a-4989-ac89-344f9d9991f9\") " pod="openshift-dns/node-resolver-qsm4z" Jul 19 14:57:39.310380 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:39.310113 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/80a8f70e-f1a5-408d-bb95-b347a0ec8715-host\") pod \"node-ca-tnrrh\" (UID: \"80a8f70e-f1a5-408d-bb95-b347a0ec8715\") " pod="openshift-image-registry/node-ca-tnrrh" Jul 19 14:57:39.310955 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:39.310161 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2571b663-0e56-4668-9831-fcdaf3793a1f-system-cni-dir\") pod \"multus-additional-cni-plugins-vzds8\" (UID: \"2571b663-0e56-4668-9831-fcdaf3793a1f\") " pod="openshift-multus/multus-additional-cni-plugins-vzds8" Jul 19 14:57:39.310955 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:39.310182 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/1367ccc0-45ba-4d3f-9892-40226e1bdfec-etc-modprobe-d\") pod \"tuned-4tgkg\" (UID: \"1367ccc0-45ba-4d3f-9892-40226e1bdfec\") " pod="openshift-cluster-node-tuning-operator/tuned-4tgkg" Jul 19 14:57:39.310955 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:39.310204 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1367ccc0-45ba-4d3f-9892-40226e1bdfec-sys\") pod \"tuned-4tgkg\" (UID: \"1367ccc0-45ba-4d3f-9892-40226e1bdfec\") " pod="openshift-cluster-node-tuning-operator/tuned-4tgkg" Jul 19 14:57:39.310955 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:39.310220 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/80a8f70e-f1a5-408d-bb95-b347a0ec8715-host\") pod \"node-ca-tnrrh\" (UID: \"80a8f70e-f1a5-408d-bb95-b347a0ec8715\") " pod="openshift-image-registry/node-ca-tnrrh" Jul 19 14:57:39.310955 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:39.310222 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/5b2d3a07-86b2-4ba2-8a71-c2d89ada79c6-var-lib-openvswitch\") pod \"ovnkube-node-5wfzs\" (UID: \"5b2d3a07-86b2-4ba2-8a71-c2d89ada79c6\") " pod="openshift-ovn-kubernetes/ovnkube-node-5wfzs" Jul 19 14:57:39.310955 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:39.310256 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/5b2d3a07-86b2-4ba2-8a71-c2d89ada79c6-host-cni-netd\") pod \"ovnkube-node-5wfzs\" (UID: \"5b2d3a07-86b2-4ba2-8a71-c2d89ada79c6\") " pod="openshift-ovn-kubernetes/ovnkube-node-5wfzs" Jul 19 14:57:39.310955 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:39.310281 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/5f0a3171-e26a-47ef-b546-078b7c259d3e-agent-certs\") pod \"konnectivity-agent-t8n4w\" (UID: \"5f0a3171-e26a-47ef-b546-078b7c259d3e\") " pod="kube-system/konnectivity-agent-t8n4w" Jul 19 14:57:39.310955 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:39.310297 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/2571b663-0e56-4668-9831-fcdaf3793a1f-os-release\") pod \"multus-additional-cni-plugins-vzds8\" (UID: \"2571b663-0e56-4668-9831-fcdaf3793a1f\") " pod="openshift-multus/multus-additional-cni-plugins-vzds8" Jul 19 14:57:39.310955 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:39.310311 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/5b2d3a07-86b2-4ba2-8a71-c2d89ada79c6-host-run-netns\") pod \"ovnkube-node-5wfzs\" (UID: \"5b2d3a07-86b2-4ba2-8a71-c2d89ada79c6\") " pod="openshift-ovn-kubernetes/ovnkube-node-5wfzs" Jul 19 14:57:39.310955 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:39.310331 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/5b2d3a07-86b2-4ba2-8a71-c2d89ada79c6-run-ovn\") pod \"ovnkube-node-5wfzs\" (UID: \"5b2d3a07-86b2-4ba2-8a71-c2d89ada79c6\") " pod="openshift-ovn-kubernetes/ovnkube-node-5wfzs" Jul 19 14:57:39.310955 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:39.310363 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8c1b2505-369a-4989-ac89-344f9d9991f9-tmp-dir\") pod \"node-resolver-qsm4z\" (UID: \"8c1b2505-369a-4989-ac89-344f9d9991f9\") " pod="openshift-dns/node-resolver-qsm4z" Jul 19 14:57:39.310955 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:39.310366 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/5a78ce34-491f-4ae5-971b-65d650736dff-multus-socket-dir-parent\") pod \"multus-jhf4r\" (UID: \"5a78ce34-491f-4ae5-971b-65d650736dff\") " pod="openshift-multus/multus-jhf4r" Jul 19 14:57:39.310955 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:39.310428 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/2571b663-0e56-4668-9831-fcdaf3793a1f-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-vzds8\" (UID: \"2571b663-0e56-4668-9831-fcdaf3793a1f\") " pod="openshift-multus/multus-additional-cni-plugins-vzds8" Jul 19 14:57:39.310955 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:39.310455 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1367ccc0-45ba-4d3f-9892-40226e1bdfec-tmp\") pod \"tuned-4tgkg\" (UID: \"1367ccc0-45ba-4d3f-9892-40226e1bdfec\") " pod="openshift-cluster-node-tuning-operator/tuned-4tgkg" Jul 19 14:57:39.310955 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:39.310479 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5a78ce34-491f-4ae5-971b-65d650736dff-etc-kubernetes\") pod \"multus-jhf4r\" (UID: \"5a78ce34-491f-4ae5-971b-65d650736dff\") " pod="openshift-multus/multus-jhf4r" Jul 19 14:57:39.310955 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:39.310505 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dnc4v\" (UniqueName: \"kubernetes.io/projected/11a5463d-a4e8-47d2-8791-5a10ea52cfc2-kube-api-access-dnc4v\") pod \"network-check-target-xpl4t\" (UID: \"11a5463d-a4e8-47d2-8791-5a10ea52cfc2\") " pod="openshift-network-diagnostics/network-check-target-xpl4t" Jul 19 14:57:39.310955 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:39.310612 2558 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Jul 19 14:57:39.313723 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:39.313703 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/5f0a3171-e26a-47ef-b546-078b7c259d3e-agent-certs\") pod \"konnectivity-agent-t8n4w\" (UID: \"5f0a3171-e26a-47ef-b546-078b7c259d3e\") " pod="kube-system/konnectivity-agent-t8n4w" Jul 19 14:57:39.315467 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:39.315442 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dt474\" (UniqueName: \"kubernetes.io/projected/80a8f70e-f1a5-408d-bb95-b347a0ec8715-kube-api-access-dt474\") pod \"node-ca-tnrrh\" (UID: \"80a8f70e-f1a5-408d-bb95-b347a0ec8715\") " pod="openshift-image-registry/node-ca-tnrrh" Jul 19 14:57:39.316974 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:39.316953 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l7mlf\" (UniqueName: \"kubernetes.io/projected/8aa5107a-42c8-4d25-a245-8a151e5b2514-kube-api-access-l7mlf\") pod \"iptables-alerter-q2fq8\" (UID: \"8aa5107a-42c8-4d25-a245-8a151e5b2514\") " pod="openshift-network-operator/iptables-alerter-q2fq8" Jul 19 14:57:39.317402 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:39.317376 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z7zfq\" (UniqueName: \"kubernetes.io/projected/c9a1c57d-99f6-49a1-bd75-d00fcffc86f1-kube-api-access-z7zfq\") pod \"network-metrics-daemon-g52v5\" (UID: \"c9a1c57d-99f6-49a1-bd75-d00fcffc86f1\") " pod="openshift-multus/network-metrics-daemon-g52v5" Jul 19 14:57:39.317505 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:39.317489 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fph4j\" (UniqueName: \"kubernetes.io/projected/8c1b2505-369a-4989-ac89-344f9d9991f9-kube-api-access-fph4j\") pod \"node-resolver-qsm4z\" (UID: \"8c1b2505-369a-4989-ac89-344f9d9991f9\") " pod="openshift-dns/node-resolver-qsm4z" Jul 19 14:57:39.411273 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:39.411249 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/2571b663-0e56-4668-9831-fcdaf3793a1f-cnibin\") pod \"multus-additional-cni-plugins-vzds8\" (UID: \"2571b663-0e56-4668-9831-fcdaf3793a1f\") " pod="openshift-multus/multus-additional-cni-plugins-vzds8" Jul 19 14:57:39.411384 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:39.411278 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/2571b663-0e56-4668-9831-fcdaf3793a1f-tuning-conf-dir\") pod \"multus-additional-cni-plugins-vzds8\" (UID: \"2571b663-0e56-4668-9831-fcdaf3793a1f\") " pod="openshift-multus/multus-additional-cni-plugins-vzds8" Jul 19 14:57:39.411384 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:39.411296 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1367ccc0-45ba-4d3f-9892-40226e1bdfec-var-lib-kubelet\") pod \"tuned-4tgkg\" (UID: \"1367ccc0-45ba-4d3f-9892-40226e1bdfec\") " pod="openshift-cluster-node-tuning-operator/tuned-4tgkg" Jul 19 14:57:39.411384 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:39.411310 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/75da72bf-18f9-4965-9bab-b355cb1d3263-socket-dir\") pod \"aws-ebs-csi-driver-node-tf7zt\" (UID: \"75da72bf-18f9-4965-9bab-b355cb1d3263\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tf7zt" Jul 19 14:57:39.411384 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:39.411326 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5a78ce34-491f-4ae5-971b-65d650736dff-host-var-lib-kubelet\") pod \"multus-jhf4r\" (UID: \"5a78ce34-491f-4ae5-971b-65d650736dff\") " pod="openshift-multus/multus-jhf4r" Jul 19 14:57:39.411384 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:39.411365 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/2571b663-0e56-4668-9831-fcdaf3793a1f-cnibin\") pod \"multus-additional-cni-plugins-vzds8\" (UID: \"2571b663-0e56-4668-9831-fcdaf3793a1f\") " pod="openshift-multus/multus-additional-cni-plugins-vzds8" Jul 19 14:57:39.411384 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:39.411364 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/1367ccc0-45ba-4d3f-9892-40226e1bdfec-etc-sysconfig\") pod \"tuned-4tgkg\" (UID: \"1367ccc0-45ba-4d3f-9892-40226e1bdfec\") " pod="openshift-cluster-node-tuning-operator/tuned-4tgkg" Jul 19 14:57:39.411583 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:39.411387 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1367ccc0-45ba-4d3f-9892-40226e1bdfec-var-lib-kubelet\") pod \"tuned-4tgkg\" (UID: \"1367ccc0-45ba-4d3f-9892-40226e1bdfec\") " pod="openshift-cluster-node-tuning-operator/tuned-4tgkg" Jul 19 14:57:39.411583 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:39.411404 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/1367ccc0-45ba-4d3f-9892-40226e1bdfec-etc-tuned\") pod \"tuned-4tgkg\" (UID: \"1367ccc0-45ba-4d3f-9892-40226e1bdfec\") " pod="openshift-cluster-node-tuning-operator/tuned-4tgkg" Jul 19 14:57:39.411583 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:39.411406 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/1367ccc0-45ba-4d3f-9892-40226e1bdfec-etc-sysconfig\") pod \"tuned-4tgkg\" (UID: \"1367ccc0-45ba-4d3f-9892-40226e1bdfec\") " pod="openshift-cluster-node-tuning-operator/tuned-4tgkg" Jul 19 14:57:39.411583 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:39.411420 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/5b2d3a07-86b2-4ba2-8a71-c2d89ada79c6-node-log\") pod \"ovnkube-node-5wfzs\" (UID: \"5b2d3a07-86b2-4ba2-8a71-c2d89ada79c6\") " pod="openshift-ovn-kubernetes/ovnkube-node-5wfzs" Jul 19 14:57:39.411583 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:39.411426 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5a78ce34-491f-4ae5-971b-65d650736dff-host-var-lib-kubelet\") pod \"multus-jhf4r\" (UID: \"5a78ce34-491f-4ae5-971b-65d650736dff\") " pod="openshift-multus/multus-jhf4r" Jul 19 14:57:39.411583 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:39.411445 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-44z2s\" (UniqueName: \"kubernetes.io/projected/1367ccc0-45ba-4d3f-9892-40226e1bdfec-kube-api-access-44z2s\") pod \"tuned-4tgkg\" (UID: \"1367ccc0-45ba-4d3f-9892-40226e1bdfec\") " pod="openshift-cluster-node-tuning-operator/tuned-4tgkg" Jul 19 14:57:39.411583 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:39.411446 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/2571b663-0e56-4668-9831-fcdaf3793a1f-tuning-conf-dir\") pod \"multus-additional-cni-plugins-vzds8\" (UID: \"2571b663-0e56-4668-9831-fcdaf3793a1f\") " pod="openshift-multus/multus-additional-cni-plugins-vzds8" Jul 19 14:57:39.411583 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:39.411462 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/5b2d3a07-86b2-4ba2-8a71-c2d89ada79c6-run-systemd\") pod \"ovnkube-node-5wfzs\" (UID: \"5b2d3a07-86b2-4ba2-8a71-c2d89ada79c6\") " pod="openshift-ovn-kubernetes/ovnkube-node-5wfzs" Jul 19 14:57:39.411583 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:39.411464 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/5b2d3a07-86b2-4ba2-8a71-c2d89ada79c6-node-log\") pod \"ovnkube-node-5wfzs\" (UID: \"5b2d3a07-86b2-4ba2-8a71-c2d89ada79c6\") " pod="openshift-ovn-kubernetes/ovnkube-node-5wfzs" Jul 19 14:57:39.411583 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:39.411476 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/5a78ce34-491f-4ae5-971b-65d650736dff-host-run-netns\") pod \"multus-jhf4r\" (UID: \"5a78ce34-491f-4ae5-971b-65d650736dff\") " pod="openshift-multus/multus-jhf4r" Jul 19 14:57:39.411583 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:39.411497 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/1367ccc0-45ba-4d3f-9892-40226e1bdfec-etc-sysctl-d\") pod \"tuned-4tgkg\" (UID: \"1367ccc0-45ba-4d3f-9892-40226e1bdfec\") " pod="openshift-cluster-node-tuning-operator/tuned-4tgkg" Jul 19 14:57:39.411583 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:39.411524 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5b2d3a07-86b2-4ba2-8a71-c2d89ada79c6-host-run-ovn-kubernetes\") pod \"ovnkube-node-5wfzs\" (UID: \"5b2d3a07-86b2-4ba2-8a71-c2d89ada79c6\") " pod="openshift-ovn-kubernetes/ovnkube-node-5wfzs" Jul 19 14:57:39.411583 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:39.411498 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/75da72bf-18f9-4965-9bab-b355cb1d3263-socket-dir\") pod \"aws-ebs-csi-driver-node-tf7zt\" (UID: \"75da72bf-18f9-4965-9bab-b355cb1d3263\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tf7zt" Jul 19 14:57:39.411583 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:39.411533 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/5b2d3a07-86b2-4ba2-8a71-c2d89ada79c6-run-systemd\") pod \"ovnkube-node-5wfzs\" (UID: \"5b2d3a07-86b2-4ba2-8a71-c2d89ada79c6\") " pod="openshift-ovn-kubernetes/ovnkube-node-5wfzs" Jul 19 14:57:39.411583 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:39.411550 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/5b2d3a07-86b2-4ba2-8a71-c2d89ada79c6-ovnkube-script-lib\") pod \"ovnkube-node-5wfzs\" (UID: \"5b2d3a07-86b2-4ba2-8a71-c2d89ada79c6\") " pod="openshift-ovn-kubernetes/ovnkube-node-5wfzs" Jul 19 14:57:39.412282 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:39.411590 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/5a78ce34-491f-4ae5-971b-65d650736dff-host-run-netns\") pod \"multus-jhf4r\" (UID: \"5a78ce34-491f-4ae5-971b-65d650736dff\") " pod="openshift-multus/multus-jhf4r" Jul 19 14:57:39.412282 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:39.411594 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5b2d3a07-86b2-4ba2-8a71-c2d89ada79c6-host-run-ovn-kubernetes\") pod \"ovnkube-node-5wfzs\" (UID: \"5b2d3a07-86b2-4ba2-8a71-c2d89ada79c6\") " pod="openshift-ovn-kubernetes/ovnkube-node-5wfzs" Jul 19 14:57:39.412282 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:39.411646 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/1367ccc0-45ba-4d3f-9892-40226e1bdfec-etc-sysctl-d\") pod \"tuned-4tgkg\" (UID: \"1367ccc0-45ba-4d3f-9892-40226e1bdfec\") " pod="openshift-cluster-node-tuning-operator/tuned-4tgkg" Jul 19 14:57:39.412282 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:39.411651 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/5a78ce34-491f-4ae5-971b-65d650736dff-cni-binary-copy\") pod \"multus-jhf4r\" (UID: \"5a78ce34-491f-4ae5-971b-65d650736dff\") " pod="openshift-multus/multus-jhf4r" Jul 19 14:57:39.412282 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:39.411707 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/5a78ce34-491f-4ae5-971b-65d650736dff-host-var-lib-cni-bin\") pod \"multus-jhf4r\" (UID: \"5a78ce34-491f-4ae5-971b-65d650736dff\") " pod="openshift-multus/multus-jhf4r" Jul 19 14:57:39.412282 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:39.411734 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/5a78ce34-491f-4ae5-971b-65d650736dff-multus-conf-dir\") pod \"multus-jhf4r\" (UID: \"5a78ce34-491f-4ae5-971b-65d650736dff\") " pod="openshift-multus/multus-jhf4r" Jul 19 14:57:39.412282 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:39.411762 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/2571b663-0e56-4668-9831-fcdaf3793a1f-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-vzds8\" (UID: \"2571b663-0e56-4668-9831-fcdaf3793a1f\") " pod="openshift-multus/multus-additional-cni-plugins-vzds8" Jul 19 14:57:39.412282 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:39.411805 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/1367ccc0-45ba-4d3f-9892-40226e1bdfec-etc-sysctl-conf\") pod \"tuned-4tgkg\" (UID: \"1367ccc0-45ba-4d3f-9892-40226e1bdfec\") " pod="openshift-cluster-node-tuning-operator/tuned-4tgkg" Jul 19 14:57:39.412282 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:39.411808 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/5a78ce34-491f-4ae5-971b-65d650736dff-host-var-lib-cni-bin\") pod \"multus-jhf4r\" (UID: \"5a78ce34-491f-4ae5-971b-65d650736dff\") " pod="openshift-multus/multus-jhf4r" Jul 19 14:57:39.412282 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:39.411842 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/75da72bf-18f9-4965-9bab-b355cb1d3263-etc-selinux\") pod \"aws-ebs-csi-driver-node-tf7zt\" (UID: \"75da72bf-18f9-4965-9bab-b355cb1d3263\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tf7zt" Jul 19 14:57:39.412282 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:39.411856 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/5a78ce34-491f-4ae5-971b-65d650736dff-multus-conf-dir\") pod \"multus-jhf4r\" (UID: \"5a78ce34-491f-4ae5-971b-65d650736dff\") " pod="openshift-multus/multus-jhf4r" Jul 19 14:57:39.412282 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:39.411870 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/5a78ce34-491f-4ae5-971b-65d650736dff-hostroot\") pod \"multus-jhf4r\" (UID: \"5a78ce34-491f-4ae5-971b-65d650736dff\") " pod="openshift-multus/multus-jhf4r" Jul 19 14:57:39.412282 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:39.411899 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/5a78ce34-491f-4ae5-971b-65d650736dff-multus-daemon-config\") pod \"multus-jhf4r\" (UID: \"5a78ce34-491f-4ae5-971b-65d650736dff\") " pod="openshift-multus/multus-jhf4r" Jul 19 14:57:39.412282 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:39.411918 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/1367ccc0-45ba-4d3f-9892-40226e1bdfec-etc-sysctl-conf\") pod \"tuned-4tgkg\" (UID: \"1367ccc0-45ba-4d3f-9892-40226e1bdfec\") " pod="openshift-cluster-node-tuning-operator/tuned-4tgkg" Jul 19 14:57:39.412282 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:39.411933 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2571b663-0e56-4668-9831-fcdaf3793a1f-system-cni-dir\") pod \"multus-additional-cni-plugins-vzds8\" (UID: \"2571b663-0e56-4668-9831-fcdaf3793a1f\") " pod="openshift-multus/multus-additional-cni-plugins-vzds8" Jul 19 14:57:39.412282 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:39.411955 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/1367ccc0-45ba-4d3f-9892-40226e1bdfec-etc-modprobe-d\") pod \"tuned-4tgkg\" (UID: \"1367ccc0-45ba-4d3f-9892-40226e1bdfec\") " pod="openshift-cluster-node-tuning-operator/tuned-4tgkg" Jul 19 14:57:39.412282 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:39.411966 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/5a78ce34-491f-4ae5-971b-65d650736dff-hostroot\") pod \"multus-jhf4r\" (UID: \"5a78ce34-491f-4ae5-971b-65d650736dff\") " pod="openshift-multus/multus-jhf4r" Jul 19 14:57:39.412282 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:39.412022 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1367ccc0-45ba-4d3f-9892-40226e1bdfec-sys\") pod \"tuned-4tgkg\" (UID: \"1367ccc0-45ba-4d3f-9892-40226e1bdfec\") " pod="openshift-cluster-node-tuning-operator/tuned-4tgkg" Jul 19 14:57:39.413117 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:39.411976 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1367ccc0-45ba-4d3f-9892-40226e1bdfec-sys\") pod \"tuned-4tgkg\" (UID: \"1367ccc0-45ba-4d3f-9892-40226e1bdfec\") " pod="openshift-cluster-node-tuning-operator/tuned-4tgkg" Jul 19 14:57:39.413117 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:39.412047 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2571b663-0e56-4668-9831-fcdaf3793a1f-system-cni-dir\") pod \"multus-additional-cni-plugins-vzds8\" (UID: \"2571b663-0e56-4668-9831-fcdaf3793a1f\") " pod="openshift-multus/multus-additional-cni-plugins-vzds8" Jul 19 14:57:39.413117 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:39.412061 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/5b2d3a07-86b2-4ba2-8a71-c2d89ada79c6-var-lib-openvswitch\") pod \"ovnkube-node-5wfzs\" (UID: \"5b2d3a07-86b2-4ba2-8a71-c2d89ada79c6\") " pod="openshift-ovn-kubernetes/ovnkube-node-5wfzs" Jul 19 14:57:39.413117 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:39.412088 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/5b2d3a07-86b2-4ba2-8a71-c2d89ada79c6-var-lib-openvswitch\") pod \"ovnkube-node-5wfzs\" (UID: \"5b2d3a07-86b2-4ba2-8a71-c2d89ada79c6\") " pod="openshift-ovn-kubernetes/ovnkube-node-5wfzs" Jul 19 14:57:39.413117 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:39.412115 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/5b2d3a07-86b2-4ba2-8a71-c2d89ada79c6-host-cni-netd\") pod \"ovnkube-node-5wfzs\" (UID: \"5b2d3a07-86b2-4ba2-8a71-c2d89ada79c6\") " pod="openshift-ovn-kubernetes/ovnkube-node-5wfzs" Jul 19 14:57:39.413117 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:39.412161 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/2571b663-0e56-4668-9831-fcdaf3793a1f-os-release\") pod \"multus-additional-cni-plugins-vzds8\" (UID: \"2571b663-0e56-4668-9831-fcdaf3793a1f\") " pod="openshift-multus/multus-additional-cni-plugins-vzds8" Jul 19 14:57:39.413117 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:39.412170 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/5b2d3a07-86b2-4ba2-8a71-c2d89ada79c6-ovnkube-script-lib\") pod \"ovnkube-node-5wfzs\" (UID: \"5b2d3a07-86b2-4ba2-8a71-c2d89ada79c6\") " pod="openshift-ovn-kubernetes/ovnkube-node-5wfzs" Jul 19 14:57:39.413117 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:39.412186 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/5b2d3a07-86b2-4ba2-8a71-c2d89ada79c6-host-run-netns\") pod \"ovnkube-node-5wfzs\" (UID: \"5b2d3a07-86b2-4ba2-8a71-c2d89ada79c6\") " pod="openshift-ovn-kubernetes/ovnkube-node-5wfzs" Jul 19 14:57:39.413117 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:39.412200 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/1367ccc0-45ba-4d3f-9892-40226e1bdfec-etc-modprobe-d\") pod \"tuned-4tgkg\" (UID: \"1367ccc0-45ba-4d3f-9892-40226e1bdfec\") " pod="openshift-cluster-node-tuning-operator/tuned-4tgkg" Jul 19 14:57:39.413117 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:39.412205 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/75da72bf-18f9-4965-9bab-b355cb1d3263-etc-selinux\") pod \"aws-ebs-csi-driver-node-tf7zt\" (UID: \"75da72bf-18f9-4965-9bab-b355cb1d3263\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tf7zt" Jul 19 14:57:39.413117 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:39.412212 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/5b2d3a07-86b2-4ba2-8a71-c2d89ada79c6-run-ovn\") pod \"ovnkube-node-5wfzs\" (UID: \"5b2d3a07-86b2-4ba2-8a71-c2d89ada79c6\") " pod="openshift-ovn-kubernetes/ovnkube-node-5wfzs" Jul 19 14:57:39.413117 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:39.412220 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/5b2d3a07-86b2-4ba2-8a71-c2d89ada79c6-host-cni-netd\") pod \"ovnkube-node-5wfzs\" (UID: \"5b2d3a07-86b2-4ba2-8a71-c2d89ada79c6\") " pod="openshift-ovn-kubernetes/ovnkube-node-5wfzs" Jul 19 14:57:39.413117 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:39.412261 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/5a78ce34-491f-4ae5-971b-65d650736dff-multus-socket-dir-parent\") pod \"multus-jhf4r\" (UID: \"5a78ce34-491f-4ae5-971b-65d650736dff\") " pod="openshift-multus/multus-jhf4r" Jul 19 14:57:39.413117 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:39.412266 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/5a78ce34-491f-4ae5-971b-65d650736dff-cni-binary-copy\") pod \"multus-jhf4r\" (UID: \"5a78ce34-491f-4ae5-971b-65d650736dff\") " pod="openshift-multus/multus-jhf4r" Jul 19 14:57:39.413117 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:39.412269 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/2571b663-0e56-4668-9831-fcdaf3793a1f-os-release\") pod \"multus-additional-cni-plugins-vzds8\" (UID: \"2571b663-0e56-4668-9831-fcdaf3793a1f\") " pod="openshift-multus/multus-additional-cni-plugins-vzds8" Jul 19 14:57:39.413117 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:39.412301 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/5b2d3a07-86b2-4ba2-8a71-c2d89ada79c6-host-run-netns\") pod \"ovnkube-node-5wfzs\" (UID: \"5b2d3a07-86b2-4ba2-8a71-c2d89ada79c6\") " pod="openshift-ovn-kubernetes/ovnkube-node-5wfzs" Jul 19 14:57:39.413117 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:39.412314 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/5b2d3a07-86b2-4ba2-8a71-c2d89ada79c6-run-ovn\") pod \"ovnkube-node-5wfzs\" (UID: \"5b2d3a07-86b2-4ba2-8a71-c2d89ada79c6\") " pod="openshift-ovn-kubernetes/ovnkube-node-5wfzs" Jul 19 14:57:39.413961 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:39.412329 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/5a78ce34-491f-4ae5-971b-65d650736dff-multus-socket-dir-parent\") pod \"multus-jhf4r\" (UID: \"5a78ce34-491f-4ae5-971b-65d650736dff\") " pod="openshift-multus/multus-jhf4r" Jul 19 14:57:39.413961 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:39.412347 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/2571b663-0e56-4668-9831-fcdaf3793a1f-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-vzds8\" (UID: \"2571b663-0e56-4668-9831-fcdaf3793a1f\") " pod="openshift-multus/multus-additional-cni-plugins-vzds8" Jul 19 14:57:39.413961 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:39.412370 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1367ccc0-45ba-4d3f-9892-40226e1bdfec-tmp\") pod \"tuned-4tgkg\" (UID: \"1367ccc0-45ba-4d3f-9892-40226e1bdfec\") " pod="openshift-cluster-node-tuning-operator/tuned-4tgkg" Jul 19 14:57:39.413961 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:39.412387 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5a78ce34-491f-4ae5-971b-65d650736dff-etc-kubernetes\") pod \"multus-jhf4r\" (UID: \"5a78ce34-491f-4ae5-971b-65d650736dff\") " pod="openshift-multus/multus-jhf4r" Jul 19 14:57:39.413961 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:39.412412 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/2571b663-0e56-4668-9831-fcdaf3793a1f-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-vzds8\" (UID: \"2571b663-0e56-4668-9831-fcdaf3793a1f\") " pod="openshift-multus/multus-additional-cni-plugins-vzds8" Jul 19 14:57:39.413961 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:39.412421 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5a78ce34-491f-4ae5-971b-65d650736dff-etc-kubernetes\") pod \"multus-jhf4r\" (UID: \"5a78ce34-491f-4ae5-971b-65d650736dff\") " pod="openshift-multus/multus-jhf4r" Jul 19 14:57:39.413961 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:39.412470 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dnc4v\" (UniqueName: \"kubernetes.io/projected/11a5463d-a4e8-47d2-8791-5a10ea52cfc2-kube-api-access-dnc4v\") pod \"network-check-target-xpl4t\" (UID: \"11a5463d-a4e8-47d2-8791-5a10ea52cfc2\") " pod="openshift-network-diagnostics/network-check-target-xpl4t" Jul 19 14:57:39.413961 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:39.412501 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/1367ccc0-45ba-4d3f-9892-40226e1bdfec-etc-systemd\") pod \"tuned-4tgkg\" (UID: \"1367ccc0-45ba-4d3f-9892-40226e1bdfec\") " pod="openshift-cluster-node-tuning-operator/tuned-4tgkg" Jul 19 14:57:39.413961 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:39.412527 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/5b2d3a07-86b2-4ba2-8a71-c2d89ada79c6-host-cni-bin\") pod \"ovnkube-node-5wfzs\" (UID: \"5b2d3a07-86b2-4ba2-8a71-c2d89ada79c6\") " pod="openshift-ovn-kubernetes/ovnkube-node-5wfzs" Jul 19 14:57:39.413961 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:39.412586 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/1367ccc0-45ba-4d3f-9892-40226e1bdfec-etc-systemd\") pod \"tuned-4tgkg\" (UID: \"1367ccc0-45ba-4d3f-9892-40226e1bdfec\") " pod="openshift-cluster-node-tuning-operator/tuned-4tgkg" Jul 19 14:57:39.413961 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:39.412591 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/5b2d3a07-86b2-4ba2-8a71-c2d89ada79c6-host-cni-bin\") pod \"ovnkube-node-5wfzs\" (UID: \"5b2d3a07-86b2-4ba2-8a71-c2d89ada79c6\") " pod="openshift-ovn-kubernetes/ovnkube-node-5wfzs" Jul 19 14:57:39.413961 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:39.412623 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/75da72bf-18f9-4965-9bab-b355cb1d3263-registration-dir\") pod \"aws-ebs-csi-driver-node-tf7zt\" (UID: \"75da72bf-18f9-4965-9bab-b355cb1d3263\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tf7zt" Jul 19 14:57:39.413961 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:39.412651 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/5a78ce34-491f-4ae5-971b-65d650736dff-multus-daemon-config\") pod \"multus-jhf4r\" (UID: \"5a78ce34-491f-4ae5-971b-65d650736dff\") " pod="openshift-multus/multus-jhf4r" Jul 19 14:57:39.413961 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:39.412676 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/5994a848-5c3c-4d6c-898c-bb659e1a41b2-kubelet-config\") pod \"global-pull-secret-syncer-kj72j\" (UID: \"5994a848-5c3c-4d6c-898c-bb659e1a41b2\") " pod="kube-system/global-pull-secret-syncer-kj72j" Jul 19 14:57:39.413961 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:39.412703 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5994a848-5c3c-4d6c-898c-bb659e1a41b2-original-pull-secret\") pod \"global-pull-secret-syncer-kj72j\" (UID: \"5994a848-5c3c-4d6c-898c-bb659e1a41b2\") " pod="kube-system/global-pull-secret-syncer-kj72j" Jul 19 14:57:39.413961 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:39.412722 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/75da72bf-18f9-4965-9bab-b355cb1d3263-registration-dir\") pod \"aws-ebs-csi-driver-node-tf7zt\" (UID: \"75da72bf-18f9-4965-9bab-b355cb1d3263\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tf7zt" Jul 19 14:57:39.413961 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:39.412745 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1367ccc0-45ba-4d3f-9892-40226e1bdfec-etc-kubernetes\") pod \"tuned-4tgkg\" (UID: \"1367ccc0-45ba-4d3f-9892-40226e1bdfec\") " pod="openshift-cluster-node-tuning-operator/tuned-4tgkg" Jul 19 14:57:39.414507 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:39.412773 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/1367ccc0-45ba-4d3f-9892-40226e1bdfec-host\") pod \"tuned-4tgkg\" (UID: \"1367ccc0-45ba-4d3f-9892-40226e1bdfec\") " pod="openshift-cluster-node-tuning-operator/tuned-4tgkg" Jul 19 14:57:39.414507 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:39.412816 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/5b2d3a07-86b2-4ba2-8a71-c2d89ada79c6-host-kubelet\") pod \"ovnkube-node-5wfzs\" (UID: \"5b2d3a07-86b2-4ba2-8a71-c2d89ada79c6\") " pod="openshift-ovn-kubernetes/ovnkube-node-5wfzs" Jul 19 14:57:39.414507 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:39.412772 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/5994a848-5c3c-4d6c-898c-bb659e1a41b2-kubelet-config\") pod \"global-pull-secret-syncer-kj72j\" (UID: \"5994a848-5c3c-4d6c-898c-bb659e1a41b2\") " pod="kube-system/global-pull-secret-syncer-kj72j" Jul 19 14:57:39.414507 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:39.412828 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1367ccc0-45ba-4d3f-9892-40226e1bdfec-etc-kubernetes\") pod \"tuned-4tgkg\" (UID: \"1367ccc0-45ba-4d3f-9892-40226e1bdfec\") " pod="openshift-cluster-node-tuning-operator/tuned-4tgkg" Jul 19 14:57:39.414507 ip-10-0-138-220 kubenswrapper[2558]: E0719 14:57:39.412835 2558 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 19 14:57:39.414507 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:39.412851 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/2571b663-0e56-4668-9831-fcdaf3793a1f-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-vzds8\" (UID: \"2571b663-0e56-4668-9831-fcdaf3793a1f\") " pod="openshift-multus/multus-additional-cni-plugins-vzds8" Jul 19 14:57:39.414507 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:39.412869 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/5b2d3a07-86b2-4ba2-8a71-c2d89ada79c6-host-kubelet\") pod \"ovnkube-node-5wfzs\" (UID: \"5b2d3a07-86b2-4ba2-8a71-c2d89ada79c6\") " pod="openshift-ovn-kubernetes/ovnkube-node-5wfzs" Jul 19 14:57:39.414507 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:39.412878 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/1367ccc0-45ba-4d3f-9892-40226e1bdfec-host\") pod \"tuned-4tgkg\" (UID: \"1367ccc0-45ba-4d3f-9892-40226e1bdfec\") " pod="openshift-cluster-node-tuning-operator/tuned-4tgkg" Jul 19 14:57:39.414507 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:39.412892 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/5b2d3a07-86b2-4ba2-8a71-c2d89ada79c6-host-slash\") pod \"ovnkube-node-5wfzs\" (UID: \"5b2d3a07-86b2-4ba2-8a71-c2d89ada79c6\") " pod="openshift-ovn-kubernetes/ovnkube-node-5wfzs" Jul 19 14:57:39.414507 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:39.412856 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/5b2d3a07-86b2-4ba2-8a71-c2d89ada79c6-host-slash\") pod \"ovnkube-node-5wfzs\" (UID: \"5b2d3a07-86b2-4ba2-8a71-c2d89ada79c6\") " pod="openshift-ovn-kubernetes/ovnkube-node-5wfzs" Jul 19 14:57:39.414507 ip-10-0-138-220 kubenswrapper[2558]: E0719 14:57:39.412913 2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5994a848-5c3c-4d6c-898c-bb659e1a41b2-original-pull-secret podName:5994a848-5c3c-4d6c-898c-bb659e1a41b2 nodeName:}" failed. No retries permitted until 2026-07-19 14:57:39.912893664 +0000 UTC m=+2.128831517 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/5994a848-5c3c-4d6c-898c-bb659e1a41b2-original-pull-secret") pod "global-pull-secret-syncer-kj72j" (UID: "5994a848-5c3c-4d6c-898c-bb659e1a41b2") : object "kube-system"/"original-pull-secret" not registered Jul 19 14:57:39.414507 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:39.412980 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/75da72bf-18f9-4965-9bab-b355cb1d3263-device-dir\") pod \"aws-ebs-csi-driver-node-tf7zt\" (UID: \"75da72bf-18f9-4965-9bab-b355cb1d3263\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tf7zt" Jul 19 14:57:39.414507 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:39.413006 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/75da72bf-18f9-4965-9bab-b355cb1d3263-sys-fs\") pod \"aws-ebs-csi-driver-node-tf7zt\" (UID: \"75da72bf-18f9-4965-9bab-b355cb1d3263\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tf7zt" Jul 19 14:57:39.414507 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:39.413035 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5a78ce34-491f-4ae5-971b-65d650736dff-system-cni-dir\") pod \"multus-jhf4r\" (UID: \"5a78ce34-491f-4ae5-971b-65d650736dff\") " pod="openshift-multus/multus-jhf4r" Jul 19 14:57:39.414507 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:39.413025 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/75da72bf-18f9-4965-9bab-b355cb1d3263-device-dir\") pod \"aws-ebs-csi-driver-node-tf7zt\" (UID: \"75da72bf-18f9-4965-9bab-b355cb1d3263\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tf7zt" Jul 19 14:57:39.414507 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:39.413061 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/5a78ce34-491f-4ae5-971b-65d650736dff-cnibin\") pod \"multus-jhf4r\" (UID: \"5a78ce34-491f-4ae5-971b-65d650736dff\") " pod="openshift-multus/multus-jhf4r" Jul 19 14:57:39.414507 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:39.413079 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/75da72bf-18f9-4965-9bab-b355cb1d3263-sys-fs\") pod \"aws-ebs-csi-driver-node-tf7zt\" (UID: \"75da72bf-18f9-4965-9bab-b355cb1d3263\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tf7zt" Jul 19 14:57:39.415031 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:39.413098 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5a78ce34-491f-4ae5-971b-65d650736dff-system-cni-dir\") pod \"multus-jhf4r\" (UID: \"5a78ce34-491f-4ae5-971b-65d650736dff\") " pod="openshift-multus/multus-jhf4r" Jul 19 14:57:39.415031 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:39.413122 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/5a78ce34-491f-4ae5-971b-65d650736dff-host-run-k8s-cni-cncf-io\") pod \"multus-jhf4r\" (UID: \"5a78ce34-491f-4ae5-971b-65d650736dff\") " pod="openshift-multus/multus-jhf4r" Jul 19 14:57:39.415031 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:39.413130 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/5a78ce34-491f-4ae5-971b-65d650736dff-cnibin\") pod \"multus-jhf4r\" (UID: \"5a78ce34-491f-4ae5-971b-65d650736dff\") " pod="openshift-multus/multus-jhf4r" Jul 19 14:57:39.415031 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:39.413144 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/5994a848-5c3c-4d6c-898c-bb659e1a41b2-dbus\") pod \"global-pull-secret-syncer-kj72j\" (UID: \"5994a848-5c3c-4d6c-898c-bb659e1a41b2\") " pod="kube-system/global-pull-secret-syncer-kj72j" Jul 19 14:57:39.415031 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:39.413167 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/1367ccc0-45ba-4d3f-9892-40226e1bdfec-run\") pod \"tuned-4tgkg\" (UID: \"1367ccc0-45ba-4d3f-9892-40226e1bdfec\") " pod="openshift-cluster-node-tuning-operator/tuned-4tgkg" Jul 19 14:57:39.415031 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:39.413165 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/5a78ce34-491f-4ae5-971b-65d650736dff-host-run-k8s-cni-cncf-io\") pod \"multus-jhf4r\" (UID: \"5a78ce34-491f-4ae5-971b-65d650736dff\") " pod="openshift-multus/multus-jhf4r" Jul 19 14:57:39.415031 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:39.413191 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/1367ccc0-45ba-4d3f-9892-40226e1bdfec-lib-modules\") pod \"tuned-4tgkg\" (UID: \"1367ccc0-45ba-4d3f-9892-40226e1bdfec\") " pod="openshift-cluster-node-tuning-operator/tuned-4tgkg" Jul 19 14:57:39.415031 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:39.413216 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/5b2d3a07-86b2-4ba2-8a71-c2d89ada79c6-systemd-units\") pod \"ovnkube-node-5wfzs\" (UID: \"5b2d3a07-86b2-4ba2-8a71-c2d89ada79c6\") " pod="openshift-ovn-kubernetes/ovnkube-node-5wfzs" Jul 19 14:57:39.415031 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:39.413226 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/5994a848-5c3c-4d6c-898c-bb659e1a41b2-dbus\") pod \"global-pull-secret-syncer-kj72j\" (UID: \"5994a848-5c3c-4d6c-898c-bb659e1a41b2\") " pod="kube-system/global-pull-secret-syncer-kj72j" Jul 19 14:57:39.415031 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:39.413228 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/1367ccc0-45ba-4d3f-9892-40226e1bdfec-run\") pod \"tuned-4tgkg\" (UID: \"1367ccc0-45ba-4d3f-9892-40226e1bdfec\") " pod="openshift-cluster-node-tuning-operator/tuned-4tgkg" Jul 19 14:57:39.415031 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:39.413241 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5b2d3a07-86b2-4ba2-8a71-c2d89ada79c6-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-5wfzs\" (UID: \"5b2d3a07-86b2-4ba2-8a71-c2d89ada79c6\") " pod="openshift-ovn-kubernetes/ovnkube-node-5wfzs" Jul 19 14:57:39.415031 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:39.413262 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/5b2d3a07-86b2-4ba2-8a71-c2d89ada79c6-systemd-units\") pod \"ovnkube-node-5wfzs\" (UID: \"5b2d3a07-86b2-4ba2-8a71-c2d89ada79c6\") " pod="openshift-ovn-kubernetes/ovnkube-node-5wfzs" Jul 19 14:57:39.415031 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:39.413272 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/5b2d3a07-86b2-4ba2-8a71-c2d89ada79c6-env-overrides\") pod \"ovnkube-node-5wfzs\" (UID: \"5b2d3a07-86b2-4ba2-8a71-c2d89ada79c6\") " pod="openshift-ovn-kubernetes/ovnkube-node-5wfzs" Jul 19 14:57:39.415031 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:39.413288 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/1367ccc0-45ba-4d3f-9892-40226e1bdfec-lib-modules\") pod \"tuned-4tgkg\" (UID: \"1367ccc0-45ba-4d3f-9892-40226e1bdfec\") " pod="openshift-cluster-node-tuning-operator/tuned-4tgkg" Jul 19 14:57:39.415031 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:39.413299 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/5a78ce34-491f-4ae5-971b-65d650736dff-os-release\") pod \"multus-jhf4r\" (UID: \"5a78ce34-491f-4ae5-971b-65d650736dff\") " pod="openshift-multus/multus-jhf4r" Jul 19 14:57:39.415031 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:39.413321 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/5b2d3a07-86b2-4ba2-8a71-c2d89ada79c6-ovn-node-metrics-cert\") pod \"ovnkube-node-5wfzs\" (UID: \"5b2d3a07-86b2-4ba2-8a71-c2d89ada79c6\") " pod="openshift-ovn-kubernetes/ovnkube-node-5wfzs" Jul 19 14:57:39.415031 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:39.413336 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x2cg5\" (UniqueName: \"kubernetes.io/projected/5b2d3a07-86b2-4ba2-8a71-c2d89ada79c6-kube-api-access-x2cg5\") pod \"ovnkube-node-5wfzs\" (UID: \"5b2d3a07-86b2-4ba2-8a71-c2d89ada79c6\") " pod="openshift-ovn-kubernetes/ovnkube-node-5wfzs" Jul 19 14:57:39.415031 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:39.413289 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5b2d3a07-86b2-4ba2-8a71-c2d89ada79c6-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-5wfzs\" (UID: \"5b2d3a07-86b2-4ba2-8a71-c2d89ada79c6\") " pod="openshift-ovn-kubernetes/ovnkube-node-5wfzs" Jul 19 14:57:39.415502 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:39.413349 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5a78ce34-491f-4ae5-971b-65d650736dff-multus-cni-dir\") pod \"multus-jhf4r\" (UID: \"5a78ce34-491f-4ae5-971b-65d650736dff\") " pod="openshift-multus/multus-jhf4r" Jul 19 14:57:39.415502 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:39.413406 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/5a78ce34-491f-4ae5-971b-65d650736dff-os-release\") pod \"multus-jhf4r\" (UID: \"5a78ce34-491f-4ae5-971b-65d650736dff\") " pod="openshift-multus/multus-jhf4r" Jul 19 14:57:39.415502 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:39.413407 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/5a78ce34-491f-4ae5-971b-65d650736dff-host-var-lib-cni-multus\") pod \"multus-jhf4r\" (UID: \"5a78ce34-491f-4ae5-971b-65d650736dff\") " pod="openshift-multus/multus-jhf4r" Jul 19 14:57:39.415502 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:39.413453 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vglps\" (UniqueName: \"kubernetes.io/projected/5a78ce34-491f-4ae5-971b-65d650736dff-kube-api-access-vglps\") pod \"multus-jhf4r\" (UID: \"5a78ce34-491f-4ae5-971b-65d650736dff\") " pod="openshift-multus/multus-jhf4r" Jul 19 14:57:39.415502 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:39.413478 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5a78ce34-491f-4ae5-971b-65d650736dff-multus-cni-dir\") pod \"multus-jhf4r\" (UID: \"5a78ce34-491f-4ae5-971b-65d650736dff\") " pod="openshift-multus/multus-jhf4r" Jul 19 14:57:39.415502 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:39.413455 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/5a78ce34-491f-4ae5-971b-65d650736dff-host-var-lib-cni-multus\") pod \"multus-jhf4r\" (UID: \"5a78ce34-491f-4ae5-971b-65d650736dff\") " pod="openshift-multus/multus-jhf4r" Jul 19 14:57:39.415502 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:39.413479 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4l4fp\" (UniqueName: \"kubernetes.io/projected/2571b663-0e56-4668-9831-fcdaf3793a1f-kube-api-access-4l4fp\") pod \"multus-additional-cni-plugins-vzds8\" (UID: \"2571b663-0e56-4668-9831-fcdaf3793a1f\") " pod="openshift-multus/multus-additional-cni-plugins-vzds8" Jul 19 14:57:39.415502 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:39.413532 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/5b2d3a07-86b2-4ba2-8a71-c2d89ada79c6-ovnkube-config\") pod \"ovnkube-node-5wfzs\" (UID: \"5b2d3a07-86b2-4ba2-8a71-c2d89ada79c6\") " pod="openshift-ovn-kubernetes/ovnkube-node-5wfzs" Jul 19 14:57:39.415502 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:39.413567 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/5a78ce34-491f-4ae5-971b-65d650736dff-host-run-multus-certs\") pod \"multus-jhf4r\" (UID: \"5a78ce34-491f-4ae5-971b-65d650736dff\") " pod="openshift-multus/multus-jhf4r" Jul 19 14:57:39.415502 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:39.413593 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/2571b663-0e56-4668-9831-fcdaf3793a1f-cni-binary-copy\") pod \"multus-additional-cni-plugins-vzds8\" (UID: \"2571b663-0e56-4668-9831-fcdaf3793a1f\") " pod="openshift-multus/multus-additional-cni-plugins-vzds8" Jul 19 14:57:39.415502 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:39.413615 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/5b2d3a07-86b2-4ba2-8a71-c2d89ada79c6-etc-openvswitch\") pod \"ovnkube-node-5wfzs\" (UID: \"5b2d3a07-86b2-4ba2-8a71-c2d89ada79c6\") " pod="openshift-ovn-kubernetes/ovnkube-node-5wfzs" Jul 19 14:57:39.415502 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:39.413628 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/5b2d3a07-86b2-4ba2-8a71-c2d89ada79c6-env-overrides\") pod \"ovnkube-node-5wfzs\" (UID: \"5b2d3a07-86b2-4ba2-8a71-c2d89ada79c6\") " pod="openshift-ovn-kubernetes/ovnkube-node-5wfzs" Jul 19 14:57:39.415502 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:39.413638 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/5b2d3a07-86b2-4ba2-8a71-c2d89ada79c6-run-openvswitch\") pod \"ovnkube-node-5wfzs\" (UID: \"5b2d3a07-86b2-4ba2-8a71-c2d89ada79c6\") " pod="openshift-ovn-kubernetes/ovnkube-node-5wfzs" Jul 19 14:57:39.415502 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:39.413662 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/5b2d3a07-86b2-4ba2-8a71-c2d89ada79c6-log-socket\") pod \"ovnkube-node-5wfzs\" (UID: \"5b2d3a07-86b2-4ba2-8a71-c2d89ada79c6\") " pod="openshift-ovn-kubernetes/ovnkube-node-5wfzs" Jul 19 14:57:39.415502 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:39.413685 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/75da72bf-18f9-4965-9bab-b355cb1d3263-kubelet-dir\") pod \"aws-ebs-csi-driver-node-tf7zt\" (UID: \"75da72bf-18f9-4965-9bab-b355cb1d3263\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tf7zt" Jul 19 14:57:39.415502 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:39.413710 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2qfp8\" (UniqueName: \"kubernetes.io/projected/75da72bf-18f9-4965-9bab-b355cb1d3263-kube-api-access-2qfp8\") pod \"aws-ebs-csi-driver-node-tf7zt\" (UID: \"75da72bf-18f9-4965-9bab-b355cb1d3263\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tf7zt" Jul 19 14:57:39.415502 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:39.413772 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/5b2d3a07-86b2-4ba2-8a71-c2d89ada79c6-run-openvswitch\") pod \"ovnkube-node-5wfzs\" (UID: \"5b2d3a07-86b2-4ba2-8a71-c2d89ada79c6\") " pod="openshift-ovn-kubernetes/ovnkube-node-5wfzs" Jul 19 14:57:39.416002 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:39.413778 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/5b2d3a07-86b2-4ba2-8a71-c2d89ada79c6-log-socket\") pod \"ovnkube-node-5wfzs\" (UID: \"5b2d3a07-86b2-4ba2-8a71-c2d89ada79c6\") " pod="openshift-ovn-kubernetes/ovnkube-node-5wfzs" Jul 19 14:57:39.416002 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:39.413832 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/5a78ce34-491f-4ae5-971b-65d650736dff-host-run-multus-certs\") pod \"multus-jhf4r\" (UID: \"5a78ce34-491f-4ae5-971b-65d650736dff\") " pod="openshift-multus/multus-jhf4r" Jul 19 14:57:39.416002 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:39.413848 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/5b2d3a07-86b2-4ba2-8a71-c2d89ada79c6-etc-openvswitch\") pod \"ovnkube-node-5wfzs\" (UID: \"5b2d3a07-86b2-4ba2-8a71-c2d89ada79c6\") " pod="openshift-ovn-kubernetes/ovnkube-node-5wfzs" Jul 19 14:57:39.416002 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:39.413858 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/75da72bf-18f9-4965-9bab-b355cb1d3263-kubelet-dir\") pod \"aws-ebs-csi-driver-node-tf7zt\" (UID: \"75da72bf-18f9-4965-9bab-b355cb1d3263\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tf7zt" Jul 19 14:57:39.416002 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:39.414064 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/5b2d3a07-86b2-4ba2-8a71-c2d89ada79c6-ovnkube-config\") pod \"ovnkube-node-5wfzs\" (UID: \"5b2d3a07-86b2-4ba2-8a71-c2d89ada79c6\") " pod="openshift-ovn-kubernetes/ovnkube-node-5wfzs" Jul 19 14:57:39.416002 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:39.414124 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/1367ccc0-45ba-4d3f-9892-40226e1bdfec-etc-tuned\") pod \"tuned-4tgkg\" (UID: \"1367ccc0-45ba-4d3f-9892-40226e1bdfec\") " pod="openshift-cluster-node-tuning-operator/tuned-4tgkg" Jul 19 14:57:39.416002 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:39.414131 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/2571b663-0e56-4668-9831-fcdaf3793a1f-cni-binary-copy\") pod \"multus-additional-cni-plugins-vzds8\" (UID: \"2571b663-0e56-4668-9831-fcdaf3793a1f\") " pod="openshift-multus/multus-additional-cni-plugins-vzds8" Jul 19 14:57:39.416002 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:39.414676 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1367ccc0-45ba-4d3f-9892-40226e1bdfec-tmp\") pod \"tuned-4tgkg\" (UID: \"1367ccc0-45ba-4d3f-9892-40226e1bdfec\") " pod="openshift-cluster-node-tuning-operator/tuned-4tgkg" Jul 19 14:57:39.416002 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:39.415524 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/5b2d3a07-86b2-4ba2-8a71-c2d89ada79c6-ovn-node-metrics-cert\") pod \"ovnkube-node-5wfzs\" (UID: \"5b2d3a07-86b2-4ba2-8a71-c2d89ada79c6\") " pod="openshift-ovn-kubernetes/ovnkube-node-5wfzs" Jul 19 14:57:39.418655 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:39.418639 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-44z2s\" (UniqueName: \"kubernetes.io/projected/1367ccc0-45ba-4d3f-9892-40226e1bdfec-kube-api-access-44z2s\") pod \"tuned-4tgkg\" (UID: \"1367ccc0-45ba-4d3f-9892-40226e1bdfec\") " pod="openshift-cluster-node-tuning-operator/tuned-4tgkg" Jul 19 14:57:39.420345 ip-10-0-138-220 kubenswrapper[2558]: E0719 14:57:39.420331 2558 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 19 14:57:39.420386 ip-10-0-138-220 kubenswrapper[2558]: E0719 14:57:39.420350 2558 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 19 14:57:39.420386 ip-10-0-138-220 kubenswrapper[2558]: E0719 14:57:39.420363 2558 projected.go:196] Error preparing data for projected volume kube-api-access-dnc4v for pod openshift-network-diagnostics/network-check-target-xpl4t: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 19 14:57:39.420452 ip-10-0-138-220 kubenswrapper[2558]: E0719 14:57:39.420441 2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/11a5463d-a4e8-47d2-8791-5a10ea52cfc2-kube-api-access-dnc4v podName:11a5463d-a4e8-47d2-8791-5a10ea52cfc2 nodeName:}" failed. No retries permitted until 2026-07-19 14:57:39.920429334 +0000 UTC m=+2.136367187 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-dnc4v" (UniqueName: "kubernetes.io/projected/11a5463d-a4e8-47d2-8791-5a10ea52cfc2-kube-api-access-dnc4v") pod "network-check-target-xpl4t" (UID: "11a5463d-a4e8-47d2-8791-5a10ea52cfc2") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 19 14:57:39.423340 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:39.423304 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2qfp8\" (UniqueName: \"kubernetes.io/projected/75da72bf-18f9-4965-9bab-b355cb1d3263-kube-api-access-2qfp8\") pod \"aws-ebs-csi-driver-node-tf7zt\" (UID: \"75da72bf-18f9-4965-9bab-b355cb1d3263\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tf7zt" Jul 19 14:57:39.423451 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:39.423430 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4l4fp\" (UniqueName: \"kubernetes.io/projected/2571b663-0e56-4668-9831-fcdaf3793a1f-kube-api-access-4l4fp\") pod \"multus-additional-cni-plugins-vzds8\" (UID: \"2571b663-0e56-4668-9831-fcdaf3793a1f\") " pod="openshift-multus/multus-additional-cni-plugins-vzds8" Jul 19 14:57:39.423553 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:39.423527 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vglps\" (UniqueName: \"kubernetes.io/projected/5a78ce34-491f-4ae5-971b-65d650736dff-kube-api-access-vglps\") pod \"multus-jhf4r\" (UID: \"5a78ce34-491f-4ae5-971b-65d650736dff\") " pod="openshift-multus/multus-jhf4r" Jul 19 14:57:39.423656 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:39.423640 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x2cg5\" (UniqueName: \"kubernetes.io/projected/5b2d3a07-86b2-4ba2-8a71-c2d89ada79c6-kube-api-access-x2cg5\") pod \"ovnkube-node-5wfzs\" (UID: \"5b2d3a07-86b2-4ba2-8a71-c2d89ada79c6\") " pod="openshift-ovn-kubernetes/ovnkube-node-5wfzs" Jul 19 14:57:39.517274 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:39.517219 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-t8n4w" Jul 19 14:57:39.526852 ip-10-0-138-220 kubenswrapper[2558]: W0719 14:57:39.526833 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5f0a3171_e26a_47ef_b546_078b7c259d3e.slice/crio-35837b26cb62298496090b662a91a652faecff17bfc5dfa549a8c0eb5233b2ca WatchSource:0}: Error finding container 35837b26cb62298496090b662a91a652faecff17bfc5dfa549a8c0eb5233b2ca: Status 404 returned error can't find the container with id 35837b26cb62298496090b662a91a652faecff17bfc5dfa549a8c0eb5233b2ca Jul 19 14:57:39.531710 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:39.531694 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-q2fq8" Jul 19 14:57:39.538514 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:39.538495 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-qsm4z" Jul 19 14:57:39.539563 ip-10-0-138-220 kubenswrapper[2558]: W0719 14:57:39.539545 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8aa5107a_42c8_4d25_a245_8a151e5b2514.slice/crio-24a0f8f82527aeba832328ed2ac79ba3b85467e554a899f9310bde515dab2a7a WatchSource:0}: Error finding container 24a0f8f82527aeba832328ed2ac79ba3b85467e554a899f9310bde515dab2a7a: Status 404 returned error can't find the container with id 24a0f8f82527aeba832328ed2ac79ba3b85467e554a899f9310bde515dab2a7a Jul 19 14:57:39.540925 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:39.540870 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-tnrrh" Jul 19 14:57:39.548075 ip-10-0-138-220 kubenswrapper[2558]: W0719 14:57:39.548054 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8c1b2505_369a_4989_ac89_344f9d9991f9.slice/crio-5c2b3d0cde37c24ed2efbf26daf61b4e46c0e7bac94bf13fa2efbf325f5112e2 WatchSource:0}: Error finding container 5c2b3d0cde37c24ed2efbf26daf61b4e46c0e7bac94bf13fa2efbf325f5112e2: Status 404 returned error can't find the container with id 5c2b3d0cde37c24ed2efbf26daf61b4e46c0e7bac94bf13fa2efbf325f5112e2 Jul 19 14:57:39.549045 ip-10-0-138-220 kubenswrapper[2558]: W0719 14:57:39.549016 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod80a8f70e_f1a5_408d_bb95_b347a0ec8715.slice/crio-97711b85e31f46a60c04413724b595b96a97dc93f51174d9406afc3e4bb6ef37 WatchSource:0}: Error finding container 97711b85e31f46a60c04413724b595b96a97dc93f51174d9406afc3e4bb6ef37: Status 404 returned error can't find the container with id 97711b85e31f46a60c04413724b595b96a97dc93f51174d9406afc3e4bb6ef37 Jul 19 14:57:39.554118 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:39.554104 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-4tgkg" Jul 19 14:57:39.563260 ip-10-0-138-220 kubenswrapper[2558]: W0719 14:57:39.563242 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1367ccc0_45ba_4d3f_9892_40226e1bdfec.slice/crio-3c0a5c39ca824fd0fa471241a3c3eeb4efecc164fac5812de0d6d8321cc956f0 WatchSource:0}: Error finding container 3c0a5c39ca824fd0fa471241a3c3eeb4efecc164fac5812de0d6d8321cc956f0: Status 404 returned error can't find the container with id 3c0a5c39ca824fd0fa471241a3c3eeb4efecc164fac5812de0d6d8321cc956f0 Jul 19 14:57:39.576898 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:39.576880 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-5wfzs" Jul 19 14:57:39.584278 ip-10-0-138-220 kubenswrapper[2558]: W0719 14:57:39.584259 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5b2d3a07_86b2_4ba2_8a71_c2d89ada79c6.slice/crio-95f26d08d2986b71bd8bb5b58b007a7c19e769379eb2a56681ee5f911d0f1c47 WatchSource:0}: Error finding container 95f26d08d2986b71bd8bb5b58b007a7c19e769379eb2a56681ee5f911d0f1c47: Status 404 returned error can't find the container with id 95f26d08d2986b71bd8bb5b58b007a7c19e769379eb2a56681ee5f911d0f1c47 Jul 19 14:57:39.589288 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:39.589272 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tf7zt" Jul 19 14:57:39.595009 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:39.594990 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-vzds8" Jul 19 14:57:39.597025 ip-10-0-138-220 kubenswrapper[2558]: W0719 14:57:39.597006 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod75da72bf_18f9_4965_9bab_b355cb1d3263.slice/crio-776c68f9f38d610366628c1ad798a3e2a6e72c60a78b69e8bee90f3c592fe936 WatchSource:0}: Error finding container 776c68f9f38d610366628c1ad798a3e2a6e72c60a78b69e8bee90f3c592fe936: Status 404 returned error can't find the container with id 776c68f9f38d610366628c1ad798a3e2a6e72c60a78b69e8bee90f3c592fe936 Jul 19 14:57:39.600003 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:39.599979 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-jhf4r" Jul 19 14:57:39.603691 ip-10-0-138-220 kubenswrapper[2558]: W0719 14:57:39.603674 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2571b663_0e56_4668_9831_fcdaf3793a1f.slice/crio-f466383d23f76bae3c9d6093cc270635608042a40e5192ad0bd0e682f7acf7ad WatchSource:0}: Error finding container f466383d23f76bae3c9d6093cc270635608042a40e5192ad0bd0e682f7acf7ad: Status 404 returned error can't find the container with id f466383d23f76bae3c9d6093cc270635608042a40e5192ad0bd0e682f7acf7ad Jul 19 14:57:39.608710 ip-10-0-138-220 kubenswrapper[2558]: W0719 14:57:39.608668 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5a78ce34_491f_4ae5_971b_65d650736dff.slice/crio-885e2a6ef5e7ea451f281573bae56eacc10da8e9889d70412eff781c5a85839f WatchSource:0}: Error finding container 885e2a6ef5e7ea451f281573bae56eacc10da8e9889d70412eff781c5a85839f: Status 404 returned error can't find the container with id 885e2a6ef5e7ea451f281573bae56eacc10da8e9889d70412eff781c5a85839f Jul 19 14:57:39.817330 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:39.817261 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c9a1c57d-99f6-49a1-bd75-d00fcffc86f1-metrics-certs\") pod \"network-metrics-daemon-g52v5\" (UID: \"c9a1c57d-99f6-49a1-bd75-d00fcffc86f1\") " pod="openshift-multus/network-metrics-daemon-g52v5" Jul 19 14:57:39.817476 ip-10-0-138-220 kubenswrapper[2558]: E0719 14:57:39.817460 2558 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 19 14:57:39.817532 ip-10-0-138-220 kubenswrapper[2558]: E0719 14:57:39.817523 2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c9a1c57d-99f6-49a1-bd75-d00fcffc86f1-metrics-certs podName:c9a1c57d-99f6-49a1-bd75-d00fcffc86f1 nodeName:}" failed. No retries permitted until 2026-07-19 14:57:40.817504698 +0000 UTC m=+3.033442556 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c9a1c57d-99f6-49a1-bd75-d00fcffc86f1-metrics-certs") pod "network-metrics-daemon-g52v5" (UID: "c9a1c57d-99f6-49a1-bd75-d00fcffc86f1") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 19 14:57:39.917675 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:39.917645 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5994a848-5c3c-4d6c-898c-bb659e1a41b2-original-pull-secret\") pod \"global-pull-secret-syncer-kj72j\" (UID: \"5994a848-5c3c-4d6c-898c-bb659e1a41b2\") " pod="kube-system/global-pull-secret-syncer-kj72j" Jul 19 14:57:39.917865 ip-10-0-138-220 kubenswrapper[2558]: E0719 14:57:39.917819 2558 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 19 14:57:39.917928 ip-10-0-138-220 kubenswrapper[2558]: E0719 14:57:39.917890 2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5994a848-5c3c-4d6c-898c-bb659e1a41b2-original-pull-secret podName:5994a848-5c3c-4d6c-898c-bb659e1a41b2 nodeName:}" failed. No retries permitted until 2026-07-19 14:57:40.917869223 +0000 UTC m=+3.133807095 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/5994a848-5c3c-4d6c-898c-bb659e1a41b2-original-pull-secret") pod "global-pull-secret-syncer-kj72j" (UID: "5994a848-5c3c-4d6c-898c-bb659e1a41b2") : object "kube-system"/"original-pull-secret" not registered Jul 19 14:57:40.018447 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:40.018413 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dnc4v\" (UniqueName: \"kubernetes.io/projected/11a5463d-a4e8-47d2-8791-5a10ea52cfc2-kube-api-access-dnc4v\") pod \"network-check-target-xpl4t\" (UID: \"11a5463d-a4e8-47d2-8791-5a10ea52cfc2\") " pod="openshift-network-diagnostics/network-check-target-xpl4t" Jul 19 14:57:40.018627 ip-10-0-138-220 kubenswrapper[2558]: E0719 14:57:40.018584 2558 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 19 14:57:40.018627 ip-10-0-138-220 kubenswrapper[2558]: E0719 14:57:40.018606 2558 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 19 14:57:40.018627 ip-10-0-138-220 kubenswrapper[2558]: E0719 14:57:40.018617 2558 projected.go:196] Error preparing data for projected volume kube-api-access-dnc4v for pod openshift-network-diagnostics/network-check-target-xpl4t: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 19 14:57:40.018956 ip-10-0-138-220 kubenswrapper[2558]: E0719 14:57:40.018671 2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/11a5463d-a4e8-47d2-8791-5a10ea52cfc2-kube-api-access-dnc4v podName:11a5463d-a4e8-47d2-8791-5a10ea52cfc2 nodeName:}" failed. No retries permitted until 2026-07-19 14:57:41.01865355 +0000 UTC m=+3.234591423 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-dnc4v" (UniqueName: "kubernetes.io/projected/11a5463d-a4e8-47d2-8791-5a10ea52cfc2-kube-api-access-dnc4v") pod "network-check-target-xpl4t" (UID: "11a5463d-a4e8-47d2-8791-5a10ea52cfc2") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 19 14:57:40.023994 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:40.023969 2558 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 19 14:57:41.542373 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:41.542263 2558 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-18 14:52:40 +0000 UTC" deadline="2028-03-06 11:12:08.811766907 +0000 UTC" Jul 19 14:57:41.542373 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:41.542326 2558 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14300h14m27.269445349s" Jul 19 14:57:41.560178 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:41.560154 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xpl4t" Jul 19 14:57:41.560323 ip-10-0-138-220 kubenswrapper[2558]: E0719 14:57:41.560267 2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xpl4t" podUID="11a5463d-a4e8-47d2-8791-5a10ea52cfc2" Jul 19 14:57:41.585833 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:41.585771 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-jhf4r" event={"ID":"5a78ce34-491f-4ae5-971b-65d650736dff","Type":"ContainerStarted","Data":"885e2a6ef5e7ea451f281573bae56eacc10da8e9889d70412eff781c5a85839f"} Jul 19 14:57:41.590757 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:41.590717 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-5wfzs" event={"ID":"5b2d3a07-86b2-4ba2-8a71-c2d89ada79c6","Type":"ContainerStarted","Data":"95f26d08d2986b71bd8bb5b58b007a7c19e769379eb2a56681ee5f911d0f1c47"} Jul 19 14:57:41.607296 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:41.607121 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-tnrrh" event={"ID":"80a8f70e-f1a5-408d-bb95-b347a0ec8715","Type":"ContainerStarted","Data":"97711b85e31f46a60c04413724b595b96a97dc93f51174d9406afc3e4bb6ef37"} Jul 19 14:57:41.625574 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:41.625543 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-qsm4z" event={"ID":"8c1b2505-369a-4989-ac89-344f9d9991f9","Type":"ContainerStarted","Data":"5c2b3d0cde37c24ed2efbf26daf61b4e46c0e7bac94bf13fa2efbf325f5112e2"} Jul 19 14:57:41.640492 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:41.640464 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-q2fq8" event={"ID":"8aa5107a-42c8-4d25-a245-8a151e5b2514","Type":"ContainerStarted","Data":"24a0f8f82527aeba832328ed2ac79ba3b85467e554a899f9310bde515dab2a7a"} Jul 19 14:57:41.645842 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:41.645818 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-t8n4w" event={"ID":"5f0a3171-e26a-47ef-b546-078b7c259d3e","Type":"ContainerStarted","Data":"35837b26cb62298496090b662a91a652faecff17bfc5dfa549a8c0eb5233b2ca"} Jul 19 14:57:41.663938 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:41.663907 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-220.ec2.internal" event={"ID":"e02b018e3a25db1c9e92391d89932378","Type":"ContainerStarted","Data":"b749e900b62f843321c8f11b3f690e5d9e8cb78f60bbe178144e63ecf25dddf1"} Jul 19 14:57:41.684453 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:41.684420 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-vzds8" event={"ID":"2571b663-0e56-4668-9831-fcdaf3793a1f","Type":"ContainerStarted","Data":"f466383d23f76bae3c9d6093cc270635608042a40e5192ad0bd0e682f7acf7ad"} Jul 19 14:57:41.690436 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:41.690411 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tf7zt" event={"ID":"75da72bf-18f9-4965-9bab-b355cb1d3263","Type":"ContainerStarted","Data":"776c68f9f38d610366628c1ad798a3e2a6e72c60a78b69e8bee90f3c592fe936"} Jul 19 14:57:41.700846 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:41.700814 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-4tgkg" event={"ID":"1367ccc0-45ba-4d3f-9892-40226e1bdfec","Type":"ContainerStarted","Data":"3c0a5c39ca824fd0fa471241a3c3eeb4efecc164fac5812de0d6d8321cc956f0"} Jul 19 14:57:41.721757 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:41.721727 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-220.ec2.internal" event={"ID":"0d2ca3cee4882a3f29ec3808e1bd3fba","Type":"ContainerStarted","Data":"8bff91165da9cb2d7a47730b9616b55b1a9d919d4b35a31d4346d84c8dba66f5"} Jul 19 14:57:41.780326 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:41.780299 2558 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 19 14:57:42.120117 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:42.120071 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c9a1c57d-99f6-49a1-bd75-d00fcffc86f1-metrics-certs\") pod \"network-metrics-daemon-g52v5\" (UID: \"c9a1c57d-99f6-49a1-bd75-d00fcffc86f1\") " pod="openshift-multus/network-metrics-daemon-g52v5" Jul 19 14:57:42.120308 ip-10-0-138-220 kubenswrapper[2558]: E0719 14:57:42.120230 2558 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 19 14:57:42.120308 ip-10-0-138-220 kubenswrapper[2558]: E0719 14:57:42.120296 2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c9a1c57d-99f6-49a1-bd75-d00fcffc86f1-metrics-certs podName:c9a1c57d-99f6-49a1-bd75-d00fcffc86f1 nodeName:}" failed. No retries permitted until 2026-07-19 14:57:44.120274698 +0000 UTC m=+5.041977415 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c9a1c57d-99f6-49a1-bd75-d00fcffc86f1-metrics-certs") pod "network-metrics-daemon-g52v5" (UID: "c9a1c57d-99f6-49a1-bd75-d00fcffc86f1") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 19 14:57:42.221334 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:42.221297 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5994a848-5c3c-4d6c-898c-bb659e1a41b2-original-pull-secret\") pod \"global-pull-secret-syncer-kj72j\" (UID: \"5994a848-5c3c-4d6c-898c-bb659e1a41b2\") " pod="kube-system/global-pull-secret-syncer-kj72j" Jul 19 14:57:42.221509 ip-10-0-138-220 kubenswrapper[2558]: E0719 14:57:42.221464 2558 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 19 14:57:42.221565 ip-10-0-138-220 kubenswrapper[2558]: E0719 14:57:42.221527 2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5994a848-5c3c-4d6c-898c-bb659e1a41b2-original-pull-secret podName:5994a848-5c3c-4d6c-898c-bb659e1a41b2 nodeName:}" failed. No retries permitted until 2026-07-19 14:57:44.221506233 +0000 UTC m=+5.143208943 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/5994a848-5c3c-4d6c-898c-bb659e1a41b2-original-pull-secret") pod "global-pull-secret-syncer-kj72j" (UID: "5994a848-5c3c-4d6c-898c-bb659e1a41b2") : object "kube-system"/"original-pull-secret" not registered Jul 19 14:57:42.322610 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:42.322571 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dnc4v\" (UniqueName: \"kubernetes.io/projected/11a5463d-a4e8-47d2-8791-5a10ea52cfc2-kube-api-access-dnc4v\") pod \"network-check-target-xpl4t\" (UID: \"11a5463d-a4e8-47d2-8791-5a10ea52cfc2\") " pod="openshift-network-diagnostics/network-check-target-xpl4t" Jul 19 14:57:42.322830 ip-10-0-138-220 kubenswrapper[2558]: E0719 14:57:42.322778 2558 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 19 14:57:42.322830 ip-10-0-138-220 kubenswrapper[2558]: E0719 14:57:42.322805 2558 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 19 14:57:42.322830 ip-10-0-138-220 kubenswrapper[2558]: E0719 14:57:42.322819 2558 projected.go:196] Error preparing data for projected volume kube-api-access-dnc4v for pod openshift-network-diagnostics/network-check-target-xpl4t: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 19 14:57:42.323039 ip-10-0-138-220 kubenswrapper[2558]: E0719 14:57:42.322873 2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/11a5463d-a4e8-47d2-8791-5a10ea52cfc2-kube-api-access-dnc4v podName:11a5463d-a4e8-47d2-8791-5a10ea52cfc2 nodeName:}" failed. No retries permitted until 2026-07-19 14:57:44.32285495 +0000 UTC m=+5.244557646 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-dnc4v" (UniqueName: "kubernetes.io/projected/11a5463d-a4e8-47d2-8791-5a10ea52cfc2-kube-api-access-dnc4v") pod "network-check-target-xpl4t" (UID: "11a5463d-a4e8-47d2-8791-5a10ea52cfc2") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 19 14:57:42.450359 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:42.450280 2558 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Jul 19 14:57:42.542486 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:42.542442 2558 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-18 14:52:40 +0000 UTC" deadline="2027-12-14 06:27:03.623527134 +0000 UTC" Jul 19 14:57:42.542486 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:42.542484 2558 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12303h29m21.081048274s" Jul 19 14:57:42.560112 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:42.559879 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kj72j" Jul 19 14:57:42.560112 ip-10-0-138-220 kubenswrapper[2558]: E0719 14:57:42.560039 2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kj72j" podUID="5994a848-5c3c-4d6c-898c-bb659e1a41b2" Jul 19 14:57:42.560342 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:42.560126 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-g52v5" Jul 19 14:57:42.560342 ip-10-0-138-220 kubenswrapper[2558]: E0719 14:57:42.560215 2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-g52v5" podUID="c9a1c57d-99f6-49a1-bd75-d00fcffc86f1" Jul 19 14:57:42.719433 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:42.718238 2558 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Jul 19 14:57:43.560155 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:43.560123 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xpl4t" Jul 19 14:57:43.560505 ip-10-0-138-220 kubenswrapper[2558]: E0719 14:57:43.560254 2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xpl4t" podUID="11a5463d-a4e8-47d2-8791-5a10ea52cfc2" Jul 19 14:57:44.141758 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:44.141726 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c9a1c57d-99f6-49a1-bd75-d00fcffc86f1-metrics-certs\") pod \"network-metrics-daemon-g52v5\" (UID: \"c9a1c57d-99f6-49a1-bd75-d00fcffc86f1\") " pod="openshift-multus/network-metrics-daemon-g52v5" Jul 19 14:57:44.141965 ip-10-0-138-220 kubenswrapper[2558]: E0719 14:57:44.141866 2558 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 19 14:57:44.141965 ip-10-0-138-220 kubenswrapper[2558]: E0719 14:57:44.141928 2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c9a1c57d-99f6-49a1-bd75-d00fcffc86f1-metrics-certs podName:c9a1c57d-99f6-49a1-bd75-d00fcffc86f1 nodeName:}" failed. No retries permitted until 2026-07-19 14:57:48.141910227 +0000 UTC m=+9.063612923 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c9a1c57d-99f6-49a1-bd75-d00fcffc86f1-metrics-certs") pod "network-metrics-daemon-g52v5" (UID: "c9a1c57d-99f6-49a1-bd75-d00fcffc86f1") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 19 14:57:44.242820 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:44.242782 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5994a848-5c3c-4d6c-898c-bb659e1a41b2-original-pull-secret\") pod \"global-pull-secret-syncer-kj72j\" (UID: \"5994a848-5c3c-4d6c-898c-bb659e1a41b2\") " pod="kube-system/global-pull-secret-syncer-kj72j" Jul 19 14:57:44.243083 ip-10-0-138-220 kubenswrapper[2558]: E0719 14:57:44.242951 2558 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 19 14:57:44.243083 ip-10-0-138-220 kubenswrapper[2558]: E0719 14:57:44.243017 2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5994a848-5c3c-4d6c-898c-bb659e1a41b2-original-pull-secret podName:5994a848-5c3c-4d6c-898c-bb659e1a41b2 nodeName:}" failed. No retries permitted until 2026-07-19 14:57:48.24299795 +0000 UTC m=+9.164700646 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/5994a848-5c3c-4d6c-898c-bb659e1a41b2-original-pull-secret") pod "global-pull-secret-syncer-kj72j" (UID: "5994a848-5c3c-4d6c-898c-bb659e1a41b2") : object "kube-system"/"original-pull-secret" not registered Jul 19 14:57:44.344761 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:44.344099 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dnc4v\" (UniqueName: \"kubernetes.io/projected/11a5463d-a4e8-47d2-8791-5a10ea52cfc2-kube-api-access-dnc4v\") pod \"network-check-target-xpl4t\" (UID: \"11a5463d-a4e8-47d2-8791-5a10ea52cfc2\") " pod="openshift-network-diagnostics/network-check-target-xpl4t" Jul 19 14:57:44.344761 ip-10-0-138-220 kubenswrapper[2558]: E0719 14:57:44.344316 2558 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 19 14:57:44.344761 ip-10-0-138-220 kubenswrapper[2558]: E0719 14:57:44.344336 2558 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 19 14:57:44.344761 ip-10-0-138-220 kubenswrapper[2558]: E0719 14:57:44.344349 2558 projected.go:196] Error preparing data for projected volume kube-api-access-dnc4v for pod openshift-network-diagnostics/network-check-target-xpl4t: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 19 14:57:44.344761 ip-10-0-138-220 kubenswrapper[2558]: E0719 14:57:44.344408 2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/11a5463d-a4e8-47d2-8791-5a10ea52cfc2-kube-api-access-dnc4v podName:11a5463d-a4e8-47d2-8791-5a10ea52cfc2 nodeName:}" failed. No retries permitted until 2026-07-19 14:57:48.344390578 +0000 UTC m=+9.266093274 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-dnc4v" (UniqueName: "kubernetes.io/projected/11a5463d-a4e8-47d2-8791-5a10ea52cfc2-kube-api-access-dnc4v") pod "network-check-target-xpl4t" (UID: "11a5463d-a4e8-47d2-8791-5a10ea52cfc2") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 19 14:57:44.560568 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:44.559851 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kj72j" Jul 19 14:57:44.560568 ip-10-0-138-220 kubenswrapper[2558]: E0719 14:57:44.559975 2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kj72j" podUID="5994a848-5c3c-4d6c-898c-bb659e1a41b2" Jul 19 14:57:44.560568 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:44.560361 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-g52v5" Jul 19 14:57:44.560568 ip-10-0-138-220 kubenswrapper[2558]: E0719 14:57:44.560470 2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-g52v5" podUID="c9a1c57d-99f6-49a1-bd75-d00fcffc86f1" Jul 19 14:57:45.562572 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:45.562093 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xpl4t" Jul 19 14:57:45.562572 ip-10-0-138-220 kubenswrapper[2558]: E0719 14:57:45.562215 2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xpl4t" podUID="11a5463d-a4e8-47d2-8791-5a10ea52cfc2" Jul 19 14:57:46.560082 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:46.560043 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kj72j" Jul 19 14:57:46.560275 ip-10-0-138-220 kubenswrapper[2558]: E0719 14:57:46.560167 2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kj72j" podUID="5994a848-5c3c-4d6c-898c-bb659e1a41b2" Jul 19 14:57:46.560275 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:46.560234 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-g52v5" Jul 19 14:57:46.560380 ip-10-0-138-220 kubenswrapper[2558]: E0719 14:57:46.560358 2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-g52v5" podUID="c9a1c57d-99f6-49a1-bd75-d00fcffc86f1" Jul 19 14:57:47.559293 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:47.559253 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xpl4t" Jul 19 14:57:47.559753 ip-10-0-138-220 kubenswrapper[2558]: E0719 14:57:47.559390 2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xpl4t" podUID="11a5463d-a4e8-47d2-8791-5a10ea52cfc2" Jul 19 14:57:48.177081 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:48.177046 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c9a1c57d-99f6-49a1-bd75-d00fcffc86f1-metrics-certs\") pod \"network-metrics-daemon-g52v5\" (UID: \"c9a1c57d-99f6-49a1-bd75-d00fcffc86f1\") " pod="openshift-multus/network-metrics-daemon-g52v5" Jul 19 14:57:48.177249 ip-10-0-138-220 kubenswrapper[2558]: E0719 14:57:48.177190 2558 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 19 14:57:48.177316 ip-10-0-138-220 kubenswrapper[2558]: E0719 14:57:48.177283 2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c9a1c57d-99f6-49a1-bd75-d00fcffc86f1-metrics-certs podName:c9a1c57d-99f6-49a1-bd75-d00fcffc86f1 nodeName:}" failed. No retries permitted until 2026-07-19 14:57:56.177259132 +0000 UTC m=+17.098961838 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c9a1c57d-99f6-49a1-bd75-d00fcffc86f1-metrics-certs") pod "network-metrics-daemon-g52v5" (UID: "c9a1c57d-99f6-49a1-bd75-d00fcffc86f1") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 19 14:57:48.277746 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:48.277710 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5994a848-5c3c-4d6c-898c-bb659e1a41b2-original-pull-secret\") pod \"global-pull-secret-syncer-kj72j\" (UID: \"5994a848-5c3c-4d6c-898c-bb659e1a41b2\") " pod="kube-system/global-pull-secret-syncer-kj72j" Jul 19 14:57:48.277880 ip-10-0-138-220 kubenswrapper[2558]: E0719 14:57:48.277846 2558 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 19 14:57:48.277915 ip-10-0-138-220 kubenswrapper[2558]: E0719 14:57:48.277901 2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5994a848-5c3c-4d6c-898c-bb659e1a41b2-original-pull-secret podName:5994a848-5c3c-4d6c-898c-bb659e1a41b2 nodeName:}" failed. No retries permitted until 2026-07-19 14:57:56.27788557 +0000 UTC m=+17.199588262 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/5994a848-5c3c-4d6c-898c-bb659e1a41b2-original-pull-secret") pod "global-pull-secret-syncer-kj72j" (UID: "5994a848-5c3c-4d6c-898c-bb659e1a41b2") : object "kube-system"/"original-pull-secret" not registered Jul 19 14:57:48.379090 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:48.379050 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dnc4v\" (UniqueName: \"kubernetes.io/projected/11a5463d-a4e8-47d2-8791-5a10ea52cfc2-kube-api-access-dnc4v\") pod \"network-check-target-xpl4t\" (UID: \"11a5463d-a4e8-47d2-8791-5a10ea52cfc2\") " pod="openshift-network-diagnostics/network-check-target-xpl4t" Jul 19 14:57:48.379257 ip-10-0-138-220 kubenswrapper[2558]: E0719 14:57:48.379194 2558 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 19 14:57:48.379257 ip-10-0-138-220 kubenswrapper[2558]: E0719 14:57:48.379208 2558 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 19 14:57:48.379257 ip-10-0-138-220 kubenswrapper[2558]: E0719 14:57:48.379217 2558 projected.go:196] Error preparing data for projected volume kube-api-access-dnc4v for pod openshift-network-diagnostics/network-check-target-xpl4t: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 19 14:57:48.379257 ip-10-0-138-220 kubenswrapper[2558]: E0719 14:57:48.379260 2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/11a5463d-a4e8-47d2-8791-5a10ea52cfc2-kube-api-access-dnc4v podName:11a5463d-a4e8-47d2-8791-5a10ea52cfc2 nodeName:}" failed. No retries permitted until 2026-07-19 14:57:56.379247096 +0000 UTC m=+17.300949788 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-dnc4v" (UniqueName: "kubernetes.io/projected/11a5463d-a4e8-47d2-8791-5a10ea52cfc2-kube-api-access-dnc4v") pod "network-check-target-xpl4t" (UID: "11a5463d-a4e8-47d2-8791-5a10ea52cfc2") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 19 14:57:48.560827 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:48.560137 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-g52v5" Jul 19 14:57:48.560827 ip-10-0-138-220 kubenswrapper[2558]: E0719 14:57:48.560279 2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-g52v5" podUID="c9a1c57d-99f6-49a1-bd75-d00fcffc86f1" Jul 19 14:57:48.560827 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:48.560337 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kj72j" Jul 19 14:57:48.560827 ip-10-0-138-220 kubenswrapper[2558]: E0719 14:57:48.560412 2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kj72j" podUID="5994a848-5c3c-4d6c-898c-bb659e1a41b2" Jul 19 14:57:49.560326 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:49.560289 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xpl4t" Jul 19 14:57:49.560502 ip-10-0-138-220 kubenswrapper[2558]: E0719 14:57:49.560397 2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xpl4t" podUID="11a5463d-a4e8-47d2-8791-5a10ea52cfc2" Jul 19 14:57:50.560198 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:50.560157 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-g52v5" Jul 19 14:57:50.560677 ip-10-0-138-220 kubenswrapper[2558]: E0719 14:57:50.560309 2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-g52v5" podUID="c9a1c57d-99f6-49a1-bd75-d00fcffc86f1" Jul 19 14:57:50.560677 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:50.560157 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kj72j" Jul 19 14:57:50.560793 ip-10-0-138-220 kubenswrapper[2558]: E0719 14:57:50.560710 2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kj72j" podUID="5994a848-5c3c-4d6c-898c-bb659e1a41b2" Jul 19 14:57:51.559537 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:51.559507 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xpl4t" Jul 19 14:57:51.559697 ip-10-0-138-220 kubenswrapper[2558]: E0719 14:57:51.559625 2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xpl4t" podUID="11a5463d-a4e8-47d2-8791-5a10ea52cfc2" Jul 19 14:57:52.559886 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:52.559858 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kj72j" Jul 19 14:57:52.559886 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:52.559894 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-g52v5" Jul 19 14:57:52.560279 ip-10-0-138-220 kubenswrapper[2558]: E0719 14:57:52.559983 2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kj72j" podUID="5994a848-5c3c-4d6c-898c-bb659e1a41b2" Jul 19 14:57:52.560279 ip-10-0-138-220 kubenswrapper[2558]: E0719 14:57:52.560015 2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-g52v5" podUID="c9a1c57d-99f6-49a1-bd75-d00fcffc86f1" Jul 19 14:57:53.559996 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:53.559965 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xpl4t" Jul 19 14:57:53.560393 ip-10-0-138-220 kubenswrapper[2558]: E0719 14:57:53.560106 2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xpl4t" podUID="11a5463d-a4e8-47d2-8791-5a10ea52cfc2" Jul 19 14:57:54.559649 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:54.559605 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-g52v5" Jul 19 14:57:54.559825 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:54.559605 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kj72j" Jul 19 14:57:54.559825 ip-10-0-138-220 kubenswrapper[2558]: E0719 14:57:54.559755 2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-g52v5" podUID="c9a1c57d-99f6-49a1-bd75-d00fcffc86f1" Jul 19 14:57:54.559825 ip-10-0-138-220 kubenswrapper[2558]: E0719 14:57:54.559804 2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kj72j" podUID="5994a848-5c3c-4d6c-898c-bb659e1a41b2" Jul 19 14:57:55.559723 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:55.559684 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xpl4t" Jul 19 14:57:55.560151 ip-10-0-138-220 kubenswrapper[2558]: E0719 14:57:55.559807 2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xpl4t" podUID="11a5463d-a4e8-47d2-8791-5a10ea52cfc2" Jul 19 14:57:56.235164 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:56.235125 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c9a1c57d-99f6-49a1-bd75-d00fcffc86f1-metrics-certs\") pod \"network-metrics-daemon-g52v5\" (UID: \"c9a1c57d-99f6-49a1-bd75-d00fcffc86f1\") " pod="openshift-multus/network-metrics-daemon-g52v5" Jul 19 14:57:56.235381 ip-10-0-138-220 kubenswrapper[2558]: E0719 14:57:56.235252 2558 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 19 14:57:56.235381 ip-10-0-138-220 kubenswrapper[2558]: E0719 14:57:56.235320 2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c9a1c57d-99f6-49a1-bd75-d00fcffc86f1-metrics-certs podName:c9a1c57d-99f6-49a1-bd75-d00fcffc86f1 nodeName:}" failed. No retries permitted until 2026-07-19 14:58:12.235300177 +0000 UTC m=+33.157002873 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c9a1c57d-99f6-49a1-bd75-d00fcffc86f1-metrics-certs") pod "network-metrics-daemon-g52v5" (UID: "c9a1c57d-99f6-49a1-bd75-d00fcffc86f1") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 19 14:57:56.336149 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:56.335942 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5994a848-5c3c-4d6c-898c-bb659e1a41b2-original-pull-secret\") pod \"global-pull-secret-syncer-kj72j\" (UID: \"5994a848-5c3c-4d6c-898c-bb659e1a41b2\") " pod="kube-system/global-pull-secret-syncer-kj72j" Jul 19 14:57:56.336149 ip-10-0-138-220 kubenswrapper[2558]: E0719 14:57:56.336133 2558 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 19 14:57:56.336379 ip-10-0-138-220 kubenswrapper[2558]: E0719 14:57:56.336212 2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5994a848-5c3c-4d6c-898c-bb659e1a41b2-original-pull-secret podName:5994a848-5c3c-4d6c-898c-bb659e1a41b2 nodeName:}" failed. No retries permitted until 2026-07-19 14:58:12.336189701 +0000 UTC m=+33.257892407 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/5994a848-5c3c-4d6c-898c-bb659e1a41b2-original-pull-secret") pod "global-pull-secret-syncer-kj72j" (UID: "5994a848-5c3c-4d6c-898c-bb659e1a41b2") : object "kube-system"/"original-pull-secret" not registered Jul 19 14:57:56.436879 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:56.436844 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dnc4v\" (UniqueName: \"kubernetes.io/projected/11a5463d-a4e8-47d2-8791-5a10ea52cfc2-kube-api-access-dnc4v\") pod \"network-check-target-xpl4t\" (UID: \"11a5463d-a4e8-47d2-8791-5a10ea52cfc2\") " pod="openshift-network-diagnostics/network-check-target-xpl4t" Jul 19 14:57:56.437082 ip-10-0-138-220 kubenswrapper[2558]: E0719 14:57:56.437002 2558 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 19 14:57:56.437082 ip-10-0-138-220 kubenswrapper[2558]: E0719 14:57:56.437046 2558 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 19 14:57:56.437082 ip-10-0-138-220 kubenswrapper[2558]: E0719 14:57:56.437060 2558 projected.go:196] Error preparing data for projected volume kube-api-access-dnc4v for pod openshift-network-diagnostics/network-check-target-xpl4t: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 19 14:57:56.437217 ip-10-0-138-220 kubenswrapper[2558]: E0719 14:57:56.437122 2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/11a5463d-a4e8-47d2-8791-5a10ea52cfc2-kube-api-access-dnc4v podName:11a5463d-a4e8-47d2-8791-5a10ea52cfc2 nodeName:}" failed. No retries permitted until 2026-07-19 14:58:12.437106317 +0000 UTC m=+33.358809032 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-dnc4v" (UniqueName: "kubernetes.io/projected/11a5463d-a4e8-47d2-8791-5a10ea52cfc2-kube-api-access-dnc4v") pod "network-check-target-xpl4t" (UID: "11a5463d-a4e8-47d2-8791-5a10ea52cfc2") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 19 14:57:56.559992 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:56.559901 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kj72j" Jul 19 14:57:56.560419 ip-10-0-138-220 kubenswrapper[2558]: E0719 14:57:56.560039 2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kj72j" podUID="5994a848-5c3c-4d6c-898c-bb659e1a41b2" Jul 19 14:57:56.560419 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:56.560080 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-g52v5" Jul 19 14:57:56.560419 ip-10-0-138-220 kubenswrapper[2558]: E0719 14:57:56.560197 2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-g52v5" podUID="c9a1c57d-99f6-49a1-bd75-d00fcffc86f1" Jul 19 14:57:57.559597 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:57.559519 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xpl4t" Jul 19 14:57:57.559759 ip-10-0-138-220 kubenswrapper[2558]: E0719 14:57:57.559647 2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xpl4t" podUID="11a5463d-a4e8-47d2-8791-5a10ea52cfc2" Jul 19 14:57:58.560254 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:58.560215 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kj72j" Jul 19 14:57:58.560666 ip-10-0-138-220 kubenswrapper[2558]: E0719 14:57:58.560350 2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kj72j" podUID="5994a848-5c3c-4d6c-898c-bb659e1a41b2" Jul 19 14:57:58.560666 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:58.560415 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-g52v5" Jul 19 14:57:58.560666 ip-10-0-138-220 kubenswrapper[2558]: E0719 14:57:58.560542 2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-g52v5" podUID="c9a1c57d-99f6-49a1-bd75-d00fcffc86f1" Jul 19 14:57:59.559990 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:57:59.559960 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xpl4t" Jul 19 14:57:59.560165 ip-10-0-138-220 kubenswrapper[2558]: E0719 14:57:59.560077 2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xpl4t" podUID="11a5463d-a4e8-47d2-8791-5a10ea52cfc2" Jul 19 14:58:00.559171 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:58:00.559144 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kj72j" Jul 19 14:58:00.559627 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:58:00.559153 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-g52v5" Jul 19 14:58:00.559627 ip-10-0-138-220 kubenswrapper[2558]: E0719 14:58:00.559247 2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kj72j" podUID="5994a848-5c3c-4d6c-898c-bb659e1a41b2" Jul 19 14:58:00.559627 ip-10-0-138-220 kubenswrapper[2558]: E0719 14:58:00.559375 2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-g52v5" podUID="c9a1c57d-99f6-49a1-bd75-d00fcffc86f1" Jul 19 14:58:01.559740 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:58:01.559571 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xpl4t" Jul 19 14:58:01.560184 ip-10-0-138-220 kubenswrapper[2558]: E0719 14:58:01.559783 2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xpl4t" podUID="11a5463d-a4e8-47d2-8791-5a10ea52cfc2" Jul 19 14:58:01.769165 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:58:01.769136 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-220.ec2.internal" event={"ID":"e02b018e3a25db1c9e92391d89932378","Type":"ContainerStarted","Data":"322ddc85a1092eb69f64276a44a87c4948be7ce054062ea9b98af35940c827c7"} Jul 19 14:58:01.770701 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:58:01.770662 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-4tgkg" event={"ID":"1367ccc0-45ba-4d3f-9892-40226e1bdfec","Type":"ContainerStarted","Data":"95ee2de743d908edd13d6f90ff0699c668bad6afb7ce12cc1966c63098ffface"} Jul 19 14:58:01.772216 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:58:01.772190 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-jhf4r" event={"ID":"5a78ce34-491f-4ae5-971b-65d650736dff","Type":"ContainerStarted","Data":"a3711d56069cc4d241abfd4bf260110854c6df537b98a05e6455511bab028a2c"} Jul 19 14:58:01.774532 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:58:01.774510 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-5wfzs_5b2d3a07-86b2-4ba2-8a71-c2d89ada79c6/ovn-acl-logging/0.log" Jul 19 14:58:01.774950 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:58:01.774930 2558 generic.go:356] "Generic (PLEG): container finished" podID="5b2d3a07-86b2-4ba2-8a71-c2d89ada79c6" containerID="ee11e2f40b6afaee3b83eca8a4196d6a739ff704e5eb9905a0706b5c7f143939" exitCode=1 Jul 19 14:58:01.775005 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:58:01.774962 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-5wfzs" event={"ID":"5b2d3a07-86b2-4ba2-8a71-c2d89ada79c6","Type":"ContainerStarted","Data":"c35af3db3edb5a2fa64e0bf13fbba5a100fdf25aaadff312006d3fa27e4a4a0b"} Jul 19 14:58:01.775054 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:58:01.775008 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-5wfzs" event={"ID":"5b2d3a07-86b2-4ba2-8a71-c2d89ada79c6","Type":"ContainerStarted","Data":"eefbb4c0ef8ac06125aea32b2c847c2ab070852a94b3c971d2ff6becaff45a06"} Jul 19 14:58:01.775085 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:58:01.775069 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-5wfzs" event={"ID":"5b2d3a07-86b2-4ba2-8a71-c2d89ada79c6","Type":"ContainerDied","Data":"ee11e2f40b6afaee3b83eca8a4196d6a739ff704e5eb9905a0706b5c7f143939"} Jul 19 14:58:01.775121 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:58:01.775100 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-5wfzs" event={"ID":"5b2d3a07-86b2-4ba2-8a71-c2d89ada79c6","Type":"ContainerStarted","Data":"6e966ccd4460ea4d94539945a8182b901ba4e79e0060d6914161253dbccb991c"} Jul 19 14:58:01.781305 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:58:01.781268 2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-220.ec2.internal" podStartSLOduration=21.781256684 podStartE2EDuration="21.781256684s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-19 14:57:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-19 14:58:01.780953145 +0000 UTC m=+22.702655859" watchObservedRunningTime="2026-07-19 14:58:01.781256684 +0000 UTC m=+22.702959397" Jul 19 14:58:01.794442 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:58:01.794408 2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-jhf4r" podStartSLOduration=2.362890758 podStartE2EDuration="22.794399369s" totalImagesPullingTime="20.431508611s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-19 14:57:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-19 14:58:01.794262949 +0000 UTC m=+22.715965662" watchObservedRunningTime="2026-07-19 14:58:01.794399369 +0000 UTC m=+22.716102073" Jul 19 14:58:01.808678 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:58:01.808642 2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-4tgkg" podStartSLOduration=2.496989759 podStartE2EDuration="22.808631846s" totalImagesPullingTime="20.311642087s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-19 14:57:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-19 14:58:01.808114596 +0000 UTC m=+22.729817310" watchObservedRunningTime="2026-07-19 14:58:01.808631846 +0000 UTC m=+22.730334560" Jul 19 14:58:02.559322 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:58:02.559291 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kj72j" Jul 19 14:58:02.559322 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:58:02.559313 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-g52v5" Jul 19 14:58:02.559626 ip-10-0-138-220 kubenswrapper[2558]: E0719 14:58:02.559415 2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kj72j" podUID="5994a848-5c3c-4d6c-898c-bb659e1a41b2" Jul 19 14:58:02.559626 ip-10-0-138-220 kubenswrapper[2558]: E0719 14:58:02.559555 2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-g52v5" podUID="c9a1c57d-99f6-49a1-bd75-d00fcffc86f1" Jul 19 14:58:02.779240 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:58:02.778949 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-t8n4w" event={"ID":"5f0a3171-e26a-47ef-b546-078b7c259d3e","Type":"ContainerStarted","Data":"d681e4c1f334aec13e05cf3304231ca10cbc462d56a9ce25166f8cb1ab8c7d13"} Jul 19 14:58:02.780800 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:58:02.780771 2558 generic.go:356] "Generic (PLEG): container finished" podID="2571b663-0e56-4668-9831-fcdaf3793a1f" containerID="2929b2cc6fbbe6baf4aa3c689574b17b3a8c4f1ac119d41c0592d6a8c04abc39" exitCode=0 Jul 19 14:58:02.780921 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:58:02.780822 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-vzds8" event={"ID":"2571b663-0e56-4668-9831-fcdaf3793a1f","Type":"ContainerDied","Data":"2929b2cc6fbbe6baf4aa3c689574b17b3a8c4f1ac119d41c0592d6a8c04abc39"} Jul 19 14:58:02.782638 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:58:02.782617 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tf7zt" event={"ID":"75da72bf-18f9-4965-9bab-b355cb1d3263","Type":"ContainerStarted","Data":"edb2d5ed5e92bc194a9a1df48a43635854680a70ed1828e3c7929a86cdee0f00"} Jul 19 14:58:02.784263 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:58:02.784238 2558 generic.go:356] "Generic (PLEG): container finished" podID="0d2ca3cee4882a3f29ec3808e1bd3fba" containerID="7222c83b285d9fee58264414a9d2bf6b416f10c2f836ae643208b1d2dd1bf778" exitCode=0 Jul 19 14:58:02.784405 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:58:02.784312 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-220.ec2.internal" event={"ID":"0d2ca3cee4882a3f29ec3808e1bd3fba","Type":"ContainerDied","Data":"7222c83b285d9fee58264414a9d2bf6b416f10c2f836ae643208b1d2dd1bf778"} Jul 19 14:58:02.787868 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:58:02.787839 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-5wfzs_5b2d3a07-86b2-4ba2-8a71-c2d89ada79c6/ovn-acl-logging/0.log" Jul 19 14:58:02.788311 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:58:02.788282 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-5wfzs" event={"ID":"5b2d3a07-86b2-4ba2-8a71-c2d89ada79c6","Type":"ContainerStarted","Data":"59eaa5c964334abd449bdb9bc73340be8f64d797d062c6414ef6bfa440dce4bc"} Jul 19 14:58:02.788401 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:58:02.788359 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-5wfzs" event={"ID":"5b2d3a07-86b2-4ba2-8a71-c2d89ada79c6","Type":"ContainerStarted","Data":"f98ab0ab878ab66749dd677d1d10d2ff2d21c59d7cc7bd8b860ea5fbad353ac1"} Jul 19 14:58:02.789881 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:58:02.789861 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-tnrrh" event={"ID":"80a8f70e-f1a5-408d-bb95-b347a0ec8715","Type":"ContainerStarted","Data":"60749a4ba626bab1598b2dfa2b3a093ed9ec72480ae38f3302e5c97be35c020d"} Jul 19 14:58:02.791777 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:58:02.791751 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-qsm4z" event={"ID":"8c1b2505-369a-4989-ac89-344f9d9991f9","Type":"ContainerStarted","Data":"4d729901e2dd936200a9663f8e5e5216112c40f023e5bed7b05713348c2d8c2c"} Jul 19 14:58:02.792296 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:58:02.792251 2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-t8n4w" podStartSLOduration=3.474439192 podStartE2EDuration="23.792236672s" totalImagesPullingTime="20.31779748s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-19 14:57:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-19 14:58:02.792166367 +0000 UTC m=+23.713869080" watchObservedRunningTime="2026-07-19 14:58:02.792236672 +0000 UTC m=+23.713939386" Jul 19 14:58:02.793628 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:58:02.793568 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-q2fq8" event={"ID":"8aa5107a-42c8-4d25-a245-8a151e5b2514","Type":"ContainerStarted","Data":"9197cfd3616cf4a0726099c1285163d8996f74b836bc6aaf1f3c11ca765411da"} Jul 19 14:58:02.836500 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:58:02.836455 2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-qsm4z" podStartSLOduration=3.540147764 podStartE2EDuration="23.836442422s" totalImagesPullingTime="20.296294658s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-19 14:57:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-19 14:58:02.835931255 +0000 UTC m=+23.757633969" watchObservedRunningTime="2026-07-19 14:58:02.836442422 +0000 UTC m=+23.758145133" Jul 19 14:58:02.848340 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:58:02.848301 2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-tnrrh" podStartSLOduration=3.540375115 podStartE2EDuration="23.848289475s" totalImagesPullingTime="20.30791436s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-19 14:57:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-19 14:58:02.847995986 +0000 UTC m=+23.769698699" watchObservedRunningTime="2026-07-19 14:58:02.848289475 +0000 UTC m=+23.769992188" Jul 19 14:58:02.860496 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:58:02.860462 2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-q2fq8" podStartSLOduration=7.630274532 podStartE2EDuration="23.86045176s" totalImagesPullingTime="16.230177228s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-19 14:57:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-19 14:58:02.860434517 +0000 UTC m=+23.782137230" watchObservedRunningTime="2026-07-19 14:58:02.86045176 +0000 UTC m=+23.782154474" Jul 19 14:58:03.269586 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:58:03.269569 2558 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Jul 19 14:58:03.559833 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:58:03.559755 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xpl4t" Jul 19 14:58:03.560037 ip-10-0-138-220 kubenswrapper[2558]: E0719 14:58:03.559885 2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xpl4t" podUID="11a5463d-a4e8-47d2-8791-5a10ea52cfc2" Jul 19 14:58:03.573470 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:58:03.573371 2558 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-19T14:58:03.269583043Z","UUID":"f5cb3d27-c0c0-4221-b16d-f257eef7007c","Handler":null,"Name":"","Endpoint":""} Jul 19 14:58:03.575076 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:58:03.575054 2558 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Jul 19 14:58:03.575198 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:58:03.575129 2558 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Jul 19 14:58:03.798289 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:58:03.798257 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tf7zt" event={"ID":"75da72bf-18f9-4965-9bab-b355cb1d3263","Type":"ContainerStarted","Data":"14f4b10df76251f1775f3b826a2b435140d8f3faba3b86f38a35eb757504edf4"} Jul 19 14:58:03.800453 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:58:03.800422 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-220.ec2.internal" event={"ID":"0d2ca3cee4882a3f29ec3808e1bd3fba","Type":"ContainerStarted","Data":"f72deadc84e34bc2de8eec0a90f918d0dd17f017026056e95c5032a388ad98f0"} Jul 19 14:58:03.815373 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:58:03.815301 2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-220.ec2.internal" podStartSLOduration=23.81528788 podStartE2EDuration="23.81528788s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-19 14:57:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-19 14:58:03.814936772 +0000 UTC m=+24.736639492" watchObservedRunningTime="2026-07-19 14:58:03.81528788 +0000 UTC m=+24.736990593" Jul 19 14:58:04.559676 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:58:04.559651 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kj72j" Jul 19 14:58:04.559816 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:58:04.559678 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-g52v5" Jul 19 14:58:04.559816 ip-10-0-138-220 kubenswrapper[2558]: E0719 14:58:04.559758 2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kj72j" podUID="5994a848-5c3c-4d6c-898c-bb659e1a41b2" Jul 19 14:58:04.559939 ip-10-0-138-220 kubenswrapper[2558]: E0719 14:58:04.559914 2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-g52v5" podUID="c9a1c57d-99f6-49a1-bd75-d00fcffc86f1" Jul 19 14:58:04.805630 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:58:04.805551 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tf7zt" event={"ID":"75da72bf-18f9-4965-9bab-b355cb1d3263","Type":"ContainerStarted","Data":"58c6fa575eae2fca60686ae1d6c885227aabd8a9914d51e77949bba2c3302ad1"} Jul 19 14:58:04.810316 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:58:04.810287 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-5wfzs_5b2d3a07-86b2-4ba2-8a71-c2d89ada79c6/ovn-acl-logging/0.log" Jul 19 14:58:04.810719 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:58:04.810689 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-5wfzs" event={"ID":"5b2d3a07-86b2-4ba2-8a71-c2d89ada79c6","Type":"ContainerStarted","Data":"193b611fc47de9c4748884e94433857b3505eae5109c9748e26a9007fb60cea0"} Jul 19 14:58:04.821768 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:58:04.821720 2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tf7zt" podStartSLOduration=3.180293319 podStartE2EDuration="25.821706979s" totalImagesPullingTime="22.64141366s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-19 14:57:39 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-19 14:58:04.821626993 +0000 UTC m=+25.743329708" watchObservedRunningTime="2026-07-19 14:58:04.821706979 +0000 UTC m=+25.743409692" Jul 19 14:58:05.559329 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:58:05.559298 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xpl4t" Jul 19 14:58:05.559490 ip-10-0-138-220 kubenswrapper[2558]: E0719 14:58:05.559404 2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xpl4t" podUID="11a5463d-a4e8-47d2-8791-5a10ea52cfc2" Jul 19 14:58:06.559193 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:58:06.559115 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kj72j" Jul 19 14:58:06.559587 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:58:06.559120 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-g52v5" Jul 19 14:58:06.559587 ip-10-0-138-220 kubenswrapper[2558]: E0719 14:58:06.559220 2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kj72j" podUID="5994a848-5c3c-4d6c-898c-bb659e1a41b2" Jul 19 14:58:06.559587 ip-10-0-138-220 kubenswrapper[2558]: E0719 14:58:06.559330 2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-g52v5" podUID="c9a1c57d-99f6-49a1-bd75-d00fcffc86f1" Jul 19 14:58:06.873875 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:58:06.873799 2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-t8n4w" Jul 19 14:58:06.874431 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:58:06.874407 2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-t8n4w" Jul 19 14:58:07.559721 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:58:07.559552 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xpl4t" Jul 19 14:58:07.560197 ip-10-0-138-220 kubenswrapper[2558]: E0719 14:58:07.559801 2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xpl4t" podUID="11a5463d-a4e8-47d2-8791-5a10ea52cfc2" Jul 19 14:58:07.818493 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:58:07.818423 2558 generic.go:356] "Generic (PLEG): container finished" podID="2571b663-0e56-4668-9831-fcdaf3793a1f" containerID="5d6cd19c649ddb66b9a5164d6e7f0fd4e227d50e26856b32e57311284408115d" exitCode=0 Jul 19 14:58:07.818643 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:58:07.818491 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-vzds8" event={"ID":"2571b663-0e56-4668-9831-fcdaf3793a1f","Type":"ContainerDied","Data":"5d6cd19c649ddb66b9a5164d6e7f0fd4e227d50e26856b32e57311284408115d"} Jul 19 14:58:07.822075 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:58:07.822057 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-5wfzs_5b2d3a07-86b2-4ba2-8a71-c2d89ada79c6/ovn-acl-logging/0.log" Jul 19 14:58:07.822450 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:58:07.822428 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-5wfzs" event={"ID":"5b2d3a07-86b2-4ba2-8a71-c2d89ada79c6","Type":"ContainerStarted","Data":"8ebfa2bf1a259ebf08e03236f0ba6a73abb601b6c491db70a4779967af499716"} Jul 19 14:58:07.823162 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:58:07.822709 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-t8n4w" Jul 19 14:58:07.823162 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:58:07.822749 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-5wfzs" Jul 19 14:58:07.823162 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:58:07.822788 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-5wfzs" Jul 19 14:58:07.823162 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:58:07.822824 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-5wfzs" Jul 19 14:58:07.823162 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:58:07.822852 2558 scope.go:117] "RemoveContainer" containerID="ee11e2f40b6afaee3b83eca8a4196d6a739ff704e5eb9905a0706b5c7f143939" Jul 19 14:58:07.823162 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:58:07.823119 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-t8n4w" Jul 19 14:58:07.839130 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:58:07.839108 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-5wfzs" Jul 19 14:58:07.839448 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:58:07.839433 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-5wfzs" Jul 19 14:58:08.559352 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:58:08.559321 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kj72j" Jul 19 14:58:08.559569 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:58:08.559327 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-g52v5" Jul 19 14:58:08.559569 ip-10-0-138-220 kubenswrapper[2558]: E0719 14:58:08.559419 2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kj72j" podUID="5994a848-5c3c-4d6c-898c-bb659e1a41b2" Jul 19 14:58:08.559569 ip-10-0-138-220 kubenswrapper[2558]: E0719 14:58:08.559498 2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-g52v5" podUID="c9a1c57d-99f6-49a1-bd75-d00fcffc86f1" Jul 19 14:58:08.829921 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:58:08.829855 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-5wfzs_5b2d3a07-86b2-4ba2-8a71-c2d89ada79c6/ovn-acl-logging/0.log" Jul 19 14:58:08.830827 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:58:08.830799 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-5wfzs" event={"ID":"5b2d3a07-86b2-4ba2-8a71-c2d89ada79c6","Type":"ContainerStarted","Data":"a8d5b5a81ac41552cd7f82a3c03c0545b8cacc5241a52d80eee32095883d6bd5"} Jul 19 14:58:08.860140 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:58:08.859902 2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-5wfzs" podStartSLOduration=9.542335713 podStartE2EDuration="29.859881437s" totalImagesPullingTime="20.317545724s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-19 14:57:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-19 14:58:08.85847614 +0000 UTC m=+29.780178868" watchObservedRunningTime="2026-07-19 14:58:08.859881437 +0000 UTC m=+29.781584213" Jul 19 14:58:09.160060 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:58:09.159936 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-kj72j"] Jul 19 14:58:09.160179 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:58:09.160071 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kj72j" Jul 19 14:58:09.160221 ip-10-0-138-220 kubenswrapper[2558]: E0719 14:58:09.160169 2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kj72j" podUID="5994a848-5c3c-4d6c-898c-bb659e1a41b2" Jul 19 14:58:09.162942 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:58:09.162918 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-xpl4t"] Jul 19 14:58:09.163109 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:58:09.163039 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xpl4t" Jul 19 14:58:09.163171 ip-10-0-138-220 kubenswrapper[2558]: E0719 14:58:09.163120 2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xpl4t" podUID="11a5463d-a4e8-47d2-8791-5a10ea52cfc2" Jul 19 14:58:09.163566 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:58:09.163546 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-g52v5"] Jul 19 14:58:09.163663 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:58:09.163654 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-g52v5" Jul 19 14:58:09.163767 ip-10-0-138-220 kubenswrapper[2558]: E0719 14:58:09.163746 2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-g52v5" podUID="c9a1c57d-99f6-49a1-bd75-d00fcffc86f1" Jul 19 14:58:09.833949 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:58:09.833912 2558 generic.go:356] "Generic (PLEG): container finished" podID="2571b663-0e56-4668-9831-fcdaf3793a1f" containerID="75d134505d18a5e6eb5f5ad64ac54c9b2203719f5985aaebe44aa881a2f1a64d" exitCode=0 Jul 19 14:58:09.834433 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:58:09.833994 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-vzds8" event={"ID":"2571b663-0e56-4668-9831-fcdaf3793a1f","Type":"ContainerDied","Data":"75d134505d18a5e6eb5f5ad64ac54c9b2203719f5985aaebe44aa881a2f1a64d"} Jul 19 14:58:10.559352 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:58:10.559306 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kj72j" Jul 19 14:58:10.559542 ip-10-0-138-220 kubenswrapper[2558]: E0719 14:58:10.559423 2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kj72j" podUID="5994a848-5c3c-4d6c-898c-bb659e1a41b2" Jul 19 14:58:10.559542 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:58:10.559454 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xpl4t" Jul 19 14:58:10.560390 ip-10-0-138-220 kubenswrapper[2558]: E0719 14:58:10.559563 2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xpl4t" podUID="11a5463d-a4e8-47d2-8791-5a10ea52cfc2" Jul 19 14:58:10.560390 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:58:10.559631 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-g52v5" Jul 19 14:58:10.560390 ip-10-0-138-220 kubenswrapper[2558]: E0719 14:58:10.559721 2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-g52v5" podUID="c9a1c57d-99f6-49a1-bd75-d00fcffc86f1" Jul 19 14:58:11.840276 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:58:11.840241 2558 generic.go:356] "Generic (PLEG): container finished" podID="2571b663-0e56-4668-9831-fcdaf3793a1f" containerID="63dd8ca027bc8f91a11f62c51d9f00716be78622f161376dc1ad0a5995520acb" exitCode=0 Jul 19 14:58:11.840713 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:58:11.840296 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-vzds8" event={"ID":"2571b663-0e56-4668-9831-fcdaf3793a1f","Type":"ContainerDied","Data":"63dd8ca027bc8f91a11f62c51d9f00716be78622f161376dc1ad0a5995520acb"} Jul 19 14:58:12.259036 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:58:12.258992 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c9a1c57d-99f6-49a1-bd75-d00fcffc86f1-metrics-certs\") pod \"network-metrics-daemon-g52v5\" (UID: \"c9a1c57d-99f6-49a1-bd75-d00fcffc86f1\") " pod="openshift-multus/network-metrics-daemon-g52v5" Jul 19 14:58:12.259182 ip-10-0-138-220 kubenswrapper[2558]: E0719 14:58:12.259121 2558 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 19 14:58:12.259182 ip-10-0-138-220 kubenswrapper[2558]: E0719 14:58:12.259177 2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c9a1c57d-99f6-49a1-bd75-d00fcffc86f1-metrics-certs podName:c9a1c57d-99f6-49a1-bd75-d00fcffc86f1 nodeName:}" failed. No retries permitted until 2026-07-19 14:58:44.259162766 +0000 UTC m=+65.180865457 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c9a1c57d-99f6-49a1-bd75-d00fcffc86f1-metrics-certs") pod "network-metrics-daemon-g52v5" (UID: "c9a1c57d-99f6-49a1-bd75-d00fcffc86f1") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 19 14:58:12.359884 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:58:12.359846 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5994a848-5c3c-4d6c-898c-bb659e1a41b2-original-pull-secret\") pod \"global-pull-secret-syncer-kj72j\" (UID: \"5994a848-5c3c-4d6c-898c-bb659e1a41b2\") " pod="kube-system/global-pull-secret-syncer-kj72j" Jul 19 14:58:12.360052 ip-10-0-138-220 kubenswrapper[2558]: E0719 14:58:12.359994 2558 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 19 14:58:12.360130 ip-10-0-138-220 kubenswrapper[2558]: E0719 14:58:12.360072 2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5994a848-5c3c-4d6c-898c-bb659e1a41b2-original-pull-secret podName:5994a848-5c3c-4d6c-898c-bb659e1a41b2 nodeName:}" failed. No retries permitted until 2026-07-19 14:58:44.360056522 +0000 UTC m=+65.281759218 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/5994a848-5c3c-4d6c-898c-bb659e1a41b2-original-pull-secret") pod "global-pull-secret-syncer-kj72j" (UID: "5994a848-5c3c-4d6c-898c-bb659e1a41b2") : object "kube-system"/"original-pull-secret" not registered Jul 19 14:58:12.461011 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:58:12.460972 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dnc4v\" (UniqueName: \"kubernetes.io/projected/11a5463d-a4e8-47d2-8791-5a10ea52cfc2-kube-api-access-dnc4v\") pod \"network-check-target-xpl4t\" (UID: \"11a5463d-a4e8-47d2-8791-5a10ea52cfc2\") " pod="openshift-network-diagnostics/network-check-target-xpl4t" Jul 19 14:58:12.461179 ip-10-0-138-220 kubenswrapper[2558]: E0719 14:58:12.461151 2558 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 19 14:58:12.461179 ip-10-0-138-220 kubenswrapper[2558]: E0719 14:58:12.461175 2558 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 19 14:58:12.461302 ip-10-0-138-220 kubenswrapper[2558]: E0719 14:58:12.461188 2558 projected.go:196] Error preparing data for projected volume kube-api-access-dnc4v for pod openshift-network-diagnostics/network-check-target-xpl4t: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 19 14:58:12.461302 ip-10-0-138-220 kubenswrapper[2558]: E0719 14:58:12.461241 2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/11a5463d-a4e8-47d2-8791-5a10ea52cfc2-kube-api-access-dnc4v podName:11a5463d-a4e8-47d2-8791-5a10ea52cfc2 nodeName:}" failed. No retries permitted until 2026-07-19 14:58:44.461227259 +0000 UTC m=+65.382929954 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-dnc4v" (UniqueName: "kubernetes.io/projected/11a5463d-a4e8-47d2-8791-5a10ea52cfc2-kube-api-access-dnc4v") pod "network-check-target-xpl4t" (UID: "11a5463d-a4e8-47d2-8791-5a10ea52cfc2") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 19 14:58:12.559690 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:58:12.559612 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kj72j" Jul 19 14:58:12.559690 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:58:12.559659 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-g52v5" Jul 19 14:58:12.559903 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:58:12.559659 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xpl4t" Jul 19 14:58:12.559903 ip-10-0-138-220 kubenswrapper[2558]: E0719 14:58:12.559741 2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-kj72j" podUID="5994a848-5c3c-4d6c-898c-bb659e1a41b2" Jul 19 14:58:12.559903 ip-10-0-138-220 kubenswrapper[2558]: E0719 14:58:12.559827 2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-g52v5" podUID="c9a1c57d-99f6-49a1-bd75-d00fcffc86f1" Jul 19 14:58:12.560038 ip-10-0-138-220 kubenswrapper[2558]: E0719 14:58:12.559911 2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xpl4t" podUID="11a5463d-a4e8-47d2-8791-5a10ea52cfc2" Jul 19 14:58:13.854052 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:58:13.853818 2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-220.ec2.internal" event="NodeReady" Jul 19 14:58:13.854398 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:58:13.854192 2558 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Jul 19 14:58:13.888006 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:58:13.887851 2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-575ddf7c75-4blsl"] Jul 19 14:58:13.891774 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:58:13.891747 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-575ddf7c75-4blsl" Jul 19 14:58:13.893897 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:58:13.893864 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-znqlg\"" Jul 19 14:58:13.894048 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:58:13.893900 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Jul 19 14:58:13.894048 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:58:13.894011 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Jul 19 14:58:13.894148 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:58:13.894099 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Jul 19 14:58:13.898381 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:58:13.898360 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Jul 19 14:58:13.902952 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:58:13.902916 2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-kxhq6"] Jul 19 14:58:13.907433 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:58:13.907411 2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-7prf4"] Jul 19 14:58:13.908406 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:58:13.907943 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-kxhq6" Jul 19 14:58:13.910214 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:58:13.910197 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Jul 19 14:58:13.910328 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:58:13.910315 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Jul 19 14:58:13.910401 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:58:13.910342 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-tvk2r\"" Jul 19 14:58:13.911650 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:58:13.911633 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-575ddf7c75-4blsl"] Jul 19 14:58:13.911734 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:58:13.911723 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-7prf4" Jul 19 14:58:13.913135 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:58:13.913016 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-7prf4"] Jul 19 14:58:13.914131 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:58:13.914112 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-g45s6\"" Jul 19 14:58:13.914224 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:58:13.914199 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Jul 19 14:58:13.914224 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:58:13.914211 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Jul 19 14:58:13.914409 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:58:13.914386 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Jul 19 14:58:13.918639 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:58:13.918619 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-kxhq6"] Jul 19 14:58:14.073367 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:58:14.073274 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6nzqw\" (UniqueName: \"kubernetes.io/projected/b50d3eba-b458-44a1-9949-0ee8c16e1e3c-kube-api-access-6nzqw\") pod \"image-registry-575ddf7c75-4blsl\" (UID: \"b50d3eba-b458-44a1-9949-0ee8c16e1e3c\") " pod="openshift-image-registry/image-registry-575ddf7c75-4blsl" Jul 19 14:58:14.073367 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:58:14.073326 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8vj7n\" (UniqueName: \"kubernetes.io/projected/47e880d5-eff9-4f7b-9c47-b7af7f74dc41-kube-api-access-8vj7n\") pod \"ingress-canary-7prf4\" (UID: \"47e880d5-eff9-4f7b-9c47-b7af7f74dc41\") " pod="openshift-ingress-canary/ingress-canary-7prf4" Jul 19 14:58:14.073367 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:58:14.073366 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/021eeff4-26d3-4e17-890e-19e1f81c1173-tmp-dir\") pod \"dns-default-kxhq6\" (UID: \"021eeff4-26d3-4e17-890e-19e1f81c1173\") " pod="openshift-dns/dns-default-kxhq6" Jul 19 14:58:14.073642 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:58:14.073394 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/b50d3eba-b458-44a1-9949-0ee8c16e1e3c-registry-certificates\") pod \"image-registry-575ddf7c75-4blsl\" (UID: \"b50d3eba-b458-44a1-9949-0ee8c16e1e3c\") " pod="openshift-image-registry/image-registry-575ddf7c75-4blsl" Jul 19 14:58:14.073642 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:58:14.073478 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/b50d3eba-b458-44a1-9949-0ee8c16e1e3c-installation-pull-secrets\") pod \"image-registry-575ddf7c75-4blsl\" (UID: \"b50d3eba-b458-44a1-9949-0ee8c16e1e3c\") " pod="openshift-image-registry/image-registry-575ddf7c75-4blsl" Jul 19 14:58:14.073642 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:58:14.073533 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/021eeff4-26d3-4e17-890e-19e1f81c1173-config-volume\") pod \"dns-default-kxhq6\" (UID: \"021eeff4-26d3-4e17-890e-19e1f81c1173\") " pod="openshift-dns/dns-default-kxhq6" Jul 19 14:58:14.073642 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:58:14.073568 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/b50d3eba-b458-44a1-9949-0ee8c16e1e3c-trusted-ca\") pod \"image-registry-575ddf7c75-4blsl\" (UID: \"b50d3eba-b458-44a1-9949-0ee8c16e1e3c\") " pod="openshift-image-registry/image-registry-575ddf7c75-4blsl" Jul 19 14:58:14.073642 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:58:14.073596 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/021eeff4-26d3-4e17-890e-19e1f81c1173-metrics-tls\") pod \"dns-default-kxhq6\" (UID: \"021eeff4-26d3-4e17-890e-19e1f81c1173\") " pod="openshift-dns/dns-default-kxhq6" Jul 19 14:58:14.073642 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:58:14.073630 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k4c8s\" (UniqueName: \"kubernetes.io/projected/021eeff4-26d3-4e17-890e-19e1f81c1173-kube-api-access-k4c8s\") pod \"dns-default-kxhq6\" (UID: \"021eeff4-26d3-4e17-890e-19e1f81c1173\") " pod="openshift-dns/dns-default-kxhq6" Jul 19 14:58:14.073931 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:58:14.073667 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/47e880d5-eff9-4f7b-9c47-b7af7f74dc41-cert\") pod \"ingress-canary-7prf4\" (UID: \"47e880d5-eff9-4f7b-9c47-b7af7f74dc41\") " pod="openshift-ingress-canary/ingress-canary-7prf4" Jul 19 14:58:14.073931 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:58:14.073689 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/b50d3eba-b458-44a1-9949-0ee8c16e1e3c-image-registry-private-configuration\") pod \"image-registry-575ddf7c75-4blsl\" (UID: \"b50d3eba-b458-44a1-9949-0ee8c16e1e3c\") " pod="openshift-image-registry/image-registry-575ddf7c75-4blsl" Jul 19 14:58:14.073931 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:58:14.073726 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/b50d3eba-b458-44a1-9949-0ee8c16e1e3c-ca-trust-extracted\") pod \"image-registry-575ddf7c75-4blsl\" (UID: \"b50d3eba-b458-44a1-9949-0ee8c16e1e3c\") " pod="openshift-image-registry/image-registry-575ddf7c75-4blsl" Jul 19 14:58:14.073931 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:58:14.073743 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b50d3eba-b458-44a1-9949-0ee8c16e1e3c-bound-sa-token\") pod \"image-registry-575ddf7c75-4blsl\" (UID: \"b50d3eba-b458-44a1-9949-0ee8c16e1e3c\") " pod="openshift-image-registry/image-registry-575ddf7c75-4blsl" Jul 19 14:58:14.073931 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:58:14.073760 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b50d3eba-b458-44a1-9949-0ee8c16e1e3c-registry-tls\") pod \"image-registry-575ddf7c75-4blsl\" (UID: \"b50d3eba-b458-44a1-9949-0ee8c16e1e3c\") " pod="openshift-image-registry/image-registry-575ddf7c75-4blsl" Jul 19 14:58:14.175015 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:58:14.174985 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6nzqw\" (UniqueName: \"kubernetes.io/projected/b50d3eba-b458-44a1-9949-0ee8c16e1e3c-kube-api-access-6nzqw\") pod \"image-registry-575ddf7c75-4blsl\" (UID: \"b50d3eba-b458-44a1-9949-0ee8c16e1e3c\") " pod="openshift-image-registry/image-registry-575ddf7c75-4blsl" Jul 19 14:58:14.175015 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:58:14.175014 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8vj7n\" (UniqueName: \"kubernetes.io/projected/47e880d5-eff9-4f7b-9c47-b7af7f74dc41-kube-api-access-8vj7n\") pod \"ingress-canary-7prf4\" (UID: \"47e880d5-eff9-4f7b-9c47-b7af7f74dc41\") " pod="openshift-ingress-canary/ingress-canary-7prf4" Jul 19 14:58:14.175237 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:58:14.175070 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/021eeff4-26d3-4e17-890e-19e1f81c1173-tmp-dir\") pod \"dns-default-kxhq6\" (UID: \"021eeff4-26d3-4e17-890e-19e1f81c1173\") " pod="openshift-dns/dns-default-kxhq6" Jul 19 14:58:14.175237 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:58:14.175100 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/b50d3eba-b458-44a1-9949-0ee8c16e1e3c-registry-certificates\") pod \"image-registry-575ddf7c75-4blsl\" (UID: \"b50d3eba-b458-44a1-9949-0ee8c16e1e3c\") " pod="openshift-image-registry/image-registry-575ddf7c75-4blsl" Jul 19 14:58:14.175237 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:58:14.175121 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/b50d3eba-b458-44a1-9949-0ee8c16e1e3c-installation-pull-secrets\") pod \"image-registry-575ddf7c75-4blsl\" (UID: \"b50d3eba-b458-44a1-9949-0ee8c16e1e3c\") " pod="openshift-image-registry/image-registry-575ddf7c75-4blsl" Jul 19 14:58:14.175237 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:58:14.175155 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/021eeff4-26d3-4e17-890e-19e1f81c1173-config-volume\") pod \"dns-default-kxhq6\" (UID: \"021eeff4-26d3-4e17-890e-19e1f81c1173\") " pod="openshift-dns/dns-default-kxhq6" Jul 19 14:58:14.175237 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:58:14.175188 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/b50d3eba-b458-44a1-9949-0ee8c16e1e3c-trusted-ca\") pod \"image-registry-575ddf7c75-4blsl\" (UID: \"b50d3eba-b458-44a1-9949-0ee8c16e1e3c\") " pod="openshift-image-registry/image-registry-575ddf7c75-4blsl" Jul 19 14:58:14.175237 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:58:14.175212 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/021eeff4-26d3-4e17-890e-19e1f81c1173-metrics-tls\") pod \"dns-default-kxhq6\" (UID: \"021eeff4-26d3-4e17-890e-19e1f81c1173\") " pod="openshift-dns/dns-default-kxhq6" Jul 19 14:58:14.175237 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:58:14.175233 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k4c8s\" (UniqueName: \"kubernetes.io/projected/021eeff4-26d3-4e17-890e-19e1f81c1173-kube-api-access-k4c8s\") pod \"dns-default-kxhq6\" (UID: \"021eeff4-26d3-4e17-890e-19e1f81c1173\") " pod="openshift-dns/dns-default-kxhq6" Jul 19 14:58:14.175538 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:58:14.175257 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/47e880d5-eff9-4f7b-9c47-b7af7f74dc41-cert\") pod \"ingress-canary-7prf4\" (UID: \"47e880d5-eff9-4f7b-9c47-b7af7f74dc41\") " pod="openshift-ingress-canary/ingress-canary-7prf4" Jul 19 14:58:14.175538 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:58:14.175284 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/b50d3eba-b458-44a1-9949-0ee8c16e1e3c-image-registry-private-configuration\") pod \"image-registry-575ddf7c75-4blsl\" (UID: \"b50d3eba-b458-44a1-9949-0ee8c16e1e3c\") " pod="openshift-image-registry/image-registry-575ddf7c75-4blsl" Jul 19 14:58:14.175538 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:58:14.175314 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/b50d3eba-b458-44a1-9949-0ee8c16e1e3c-ca-trust-extracted\") pod \"image-registry-575ddf7c75-4blsl\" (UID: \"b50d3eba-b458-44a1-9949-0ee8c16e1e3c\") " pod="openshift-image-registry/image-registry-575ddf7c75-4blsl" Jul 19 14:58:14.175538 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:58:14.175336 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b50d3eba-b458-44a1-9949-0ee8c16e1e3c-bound-sa-token\") pod \"image-registry-575ddf7c75-4blsl\" (UID: \"b50d3eba-b458-44a1-9949-0ee8c16e1e3c\") " pod="openshift-image-registry/image-registry-575ddf7c75-4blsl" Jul 19 14:58:14.175538 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:58:14.175364 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b50d3eba-b458-44a1-9949-0ee8c16e1e3c-registry-tls\") pod \"image-registry-575ddf7c75-4blsl\" (UID: \"b50d3eba-b458-44a1-9949-0ee8c16e1e3c\") " pod="openshift-image-registry/image-registry-575ddf7c75-4blsl" Jul 19 14:58:14.175538 ip-10-0-138-220 kubenswrapper[2558]: E0719 14:58:14.175485 2558 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 19 14:58:14.175538 ip-10-0-138-220 kubenswrapper[2558]: E0719 14:58:14.175502 2558 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-575ddf7c75-4blsl: secret "image-registry-tls" not found Jul 19 14:58:14.175855 ip-10-0-138-220 kubenswrapper[2558]: E0719 14:58:14.175563 2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b50d3eba-b458-44a1-9949-0ee8c16e1e3c-registry-tls podName:b50d3eba-b458-44a1-9949-0ee8c16e1e3c nodeName:}" failed. No retries permitted until 2026-07-19 14:58:14.675544642 +0000 UTC m=+35.597247350 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/b50d3eba-b458-44a1-9949-0ee8c16e1e3c-registry-tls") pod "image-registry-575ddf7c75-4blsl" (UID: "b50d3eba-b458-44a1-9949-0ee8c16e1e3c") : secret "image-registry-tls" not found Jul 19 14:58:14.175855 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:58:14.175695 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/021eeff4-26d3-4e17-890e-19e1f81c1173-tmp-dir\") pod \"dns-default-kxhq6\" (UID: \"021eeff4-26d3-4e17-890e-19e1f81c1173\") " pod="openshift-dns/dns-default-kxhq6" Jul 19 14:58:14.176248 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:58:14.176014 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/b50d3eba-b458-44a1-9949-0ee8c16e1e3c-ca-trust-extracted\") pod \"image-registry-575ddf7c75-4blsl\" (UID: \"b50d3eba-b458-44a1-9949-0ee8c16e1e3c\") " pod="openshift-image-registry/image-registry-575ddf7c75-4blsl" Jul 19 14:58:14.176248 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:58:14.176017 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/b50d3eba-b458-44a1-9949-0ee8c16e1e3c-registry-certificates\") pod \"image-registry-575ddf7c75-4blsl\" (UID: \"b50d3eba-b458-44a1-9949-0ee8c16e1e3c\") " pod="openshift-image-registry/image-registry-575ddf7c75-4blsl" Jul 19 14:58:14.176248 ip-10-0-138-220 kubenswrapper[2558]: E0719 14:58:14.176114 2558 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 19 14:58:14.176248 ip-10-0-138-220 kubenswrapper[2558]: E0719 14:58:14.176174 2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/021eeff4-26d3-4e17-890e-19e1f81c1173-metrics-tls podName:021eeff4-26d3-4e17-890e-19e1f81c1173 nodeName:}" failed. No retries permitted until 2026-07-19 14:58:14.676154993 +0000 UTC m=+35.597857687 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/021eeff4-26d3-4e17-890e-19e1f81c1173-metrics-tls") pod "dns-default-kxhq6" (UID: "021eeff4-26d3-4e17-890e-19e1f81c1173") : secret "dns-default-metrics-tls" not found Jul 19 14:58:14.176516 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:58:14.176419 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/b50d3eba-b458-44a1-9949-0ee8c16e1e3c-trusted-ca\") pod \"image-registry-575ddf7c75-4blsl\" (UID: \"b50d3eba-b458-44a1-9949-0ee8c16e1e3c\") " pod="openshift-image-registry/image-registry-575ddf7c75-4blsl" Jul 19 14:58:14.176564 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:58:14.176529 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/021eeff4-26d3-4e17-890e-19e1f81c1173-config-volume\") pod \"dns-default-kxhq6\" (UID: \"021eeff4-26d3-4e17-890e-19e1f81c1173\") " pod="openshift-dns/dns-default-kxhq6" Jul 19 14:58:14.179937 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:58:14.179913 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/b50d3eba-b458-44a1-9949-0ee8c16e1e3c-image-registry-private-configuration\") pod \"image-registry-575ddf7c75-4blsl\" (UID: \"b50d3eba-b458-44a1-9949-0ee8c16e1e3c\") " pod="openshift-image-registry/image-registry-575ddf7c75-4blsl" Jul 19 14:58:14.179937 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:58:14.179925 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/b50d3eba-b458-44a1-9949-0ee8c16e1e3c-installation-pull-secrets\") pod \"image-registry-575ddf7c75-4blsl\" (UID: \"b50d3eba-b458-44a1-9949-0ee8c16e1e3c\") " pod="openshift-image-registry/image-registry-575ddf7c75-4blsl" Jul 19 14:58:14.180152 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:58:14.179929 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/47e880d5-eff9-4f7b-9c47-b7af7f74dc41-cert\") pod \"ingress-canary-7prf4\" (UID: \"47e880d5-eff9-4f7b-9c47-b7af7f74dc41\") " pod="openshift-ingress-canary/ingress-canary-7prf4" Jul 19 14:58:14.183852 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:58:14.183820 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8vj7n\" (UniqueName: \"kubernetes.io/projected/47e880d5-eff9-4f7b-9c47-b7af7f74dc41-kube-api-access-8vj7n\") pod \"ingress-canary-7prf4\" (UID: \"47e880d5-eff9-4f7b-9c47-b7af7f74dc41\") " pod="openshift-ingress-canary/ingress-canary-7prf4" Jul 19 14:58:14.184226 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:58:14.184206 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b50d3eba-b458-44a1-9949-0ee8c16e1e3c-bound-sa-token\") pod \"image-registry-575ddf7c75-4blsl\" (UID: \"b50d3eba-b458-44a1-9949-0ee8c16e1e3c\") " pod="openshift-image-registry/image-registry-575ddf7c75-4blsl" Jul 19 14:58:14.184457 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:58:14.184438 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6nzqw\" (UniqueName: \"kubernetes.io/projected/b50d3eba-b458-44a1-9949-0ee8c16e1e3c-kube-api-access-6nzqw\") pod \"image-registry-575ddf7c75-4blsl\" (UID: \"b50d3eba-b458-44a1-9949-0ee8c16e1e3c\") " pod="openshift-image-registry/image-registry-575ddf7c75-4blsl" Jul 19 14:58:14.184853 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:58:14.184829 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k4c8s\" (UniqueName: \"kubernetes.io/projected/021eeff4-26d3-4e17-890e-19e1f81c1173-kube-api-access-k4c8s\") pod \"dns-default-kxhq6\" (UID: \"021eeff4-26d3-4e17-890e-19e1f81c1173\") " pod="openshift-dns/dns-default-kxhq6" Jul 19 14:58:14.227198 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:58:14.227173 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-7prf4" Jul 19 14:58:14.377016 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:58:14.376916 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-7prf4"] Jul 19 14:58:14.380199 ip-10-0-138-220 kubenswrapper[2558]: W0719 14:58:14.380171 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod47e880d5_eff9_4f7b_9c47_b7af7f74dc41.slice/crio-bfe130f9b274f3f0226b3904dbaa4911286f9e34562a13a8d0adf18d53bf7919 WatchSource:0}: Error finding container bfe130f9b274f3f0226b3904dbaa4911286f9e34562a13a8d0adf18d53bf7919: Status 404 returned error can't find the container with id bfe130f9b274f3f0226b3904dbaa4911286f9e34562a13a8d0adf18d53bf7919 Jul 19 14:58:14.560075 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:58:14.560045 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kj72j" Jul 19 14:58:14.560075 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:58:14.560061 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-g52v5" Jul 19 14:58:14.560075 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:58:14.560078 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xpl4t" Jul 19 14:58:14.563302 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:58:14.563167 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 19 14:58:14.563302 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:58:14.563178 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 19 14:58:14.563302 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:58:14.563187 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-v7fqw\"" Jul 19 14:58:14.563302 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:58:14.563236 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-hc8w7\"" Jul 19 14:58:14.563302 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:58:14.563171 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 19 14:58:14.563981 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:58:14.563962 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 19 14:58:14.679118 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:58:14.679091 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/021eeff4-26d3-4e17-890e-19e1f81c1173-metrics-tls\") pod \"dns-default-kxhq6\" (UID: \"021eeff4-26d3-4e17-890e-19e1f81c1173\") " pod="openshift-dns/dns-default-kxhq6" Jul 19 14:58:14.679288 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:58:14.679145 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b50d3eba-b458-44a1-9949-0ee8c16e1e3c-registry-tls\") pod \"image-registry-575ddf7c75-4blsl\" (UID: \"b50d3eba-b458-44a1-9949-0ee8c16e1e3c\") " pod="openshift-image-registry/image-registry-575ddf7c75-4blsl" Jul 19 14:58:14.679288 ip-10-0-138-220 kubenswrapper[2558]: E0719 14:58:14.679275 2558 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 19 14:58:14.679414 ip-10-0-138-220 kubenswrapper[2558]: E0719 14:58:14.679337 2558 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 19 14:58:14.679414 ip-10-0-138-220 kubenswrapper[2558]: E0719 14:58:14.679355 2558 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-575ddf7c75-4blsl: secret "image-registry-tls" not found Jul 19 14:58:14.679414 ip-10-0-138-220 kubenswrapper[2558]: E0719 14:58:14.679371 2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/021eeff4-26d3-4e17-890e-19e1f81c1173-metrics-tls podName:021eeff4-26d3-4e17-890e-19e1f81c1173 nodeName:}" failed. No retries permitted until 2026-07-19 14:58:15.679348722 +0000 UTC m=+36.601051437 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/021eeff4-26d3-4e17-890e-19e1f81c1173-metrics-tls") pod "dns-default-kxhq6" (UID: "021eeff4-26d3-4e17-890e-19e1f81c1173") : secret "dns-default-metrics-tls" not found Jul 19 14:58:14.679545 ip-10-0-138-220 kubenswrapper[2558]: E0719 14:58:14.679419 2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b50d3eba-b458-44a1-9949-0ee8c16e1e3c-registry-tls podName:b50d3eba-b458-44a1-9949-0ee8c16e1e3c nodeName:}" failed. No retries permitted until 2026-07-19 14:58:15.679401108 +0000 UTC m=+36.601103813 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/b50d3eba-b458-44a1-9949-0ee8c16e1e3c-registry-tls") pod "image-registry-575ddf7c75-4blsl" (UID: "b50d3eba-b458-44a1-9949-0ee8c16e1e3c") : secret "image-registry-tls" not found Jul 19 14:58:14.848873 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:58:14.848787 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-7prf4" event={"ID":"47e880d5-eff9-4f7b-9c47-b7af7f74dc41","Type":"ContainerStarted","Data":"bfe130f9b274f3f0226b3904dbaa4911286f9e34562a13a8d0adf18d53bf7919"} Jul 19 14:58:15.688230 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:58:15.688179 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/021eeff4-26d3-4e17-890e-19e1f81c1173-metrics-tls\") pod \"dns-default-kxhq6\" (UID: \"021eeff4-26d3-4e17-890e-19e1f81c1173\") " pod="openshift-dns/dns-default-kxhq6" Jul 19 14:58:15.689120 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:58:15.688247 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b50d3eba-b458-44a1-9949-0ee8c16e1e3c-registry-tls\") pod \"image-registry-575ddf7c75-4blsl\" (UID: \"b50d3eba-b458-44a1-9949-0ee8c16e1e3c\") " pod="openshift-image-registry/image-registry-575ddf7c75-4blsl" Jul 19 14:58:15.689120 ip-10-0-138-220 kubenswrapper[2558]: E0719 14:58:15.688353 2558 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 19 14:58:15.689120 ip-10-0-138-220 kubenswrapper[2558]: E0719 14:58:15.688426 2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/021eeff4-26d3-4e17-890e-19e1f81c1173-metrics-tls podName:021eeff4-26d3-4e17-890e-19e1f81c1173 nodeName:}" failed. No retries permitted until 2026-07-19 14:58:17.688404098 +0000 UTC m=+38.610106809 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/021eeff4-26d3-4e17-890e-19e1f81c1173-metrics-tls") pod "dns-default-kxhq6" (UID: "021eeff4-26d3-4e17-890e-19e1f81c1173") : secret "dns-default-metrics-tls" not found Jul 19 14:58:15.689120 ip-10-0-138-220 kubenswrapper[2558]: E0719 14:58:15.688357 2558 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 19 14:58:15.689120 ip-10-0-138-220 kubenswrapper[2558]: E0719 14:58:15.688467 2558 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-575ddf7c75-4blsl: secret "image-registry-tls" not found Jul 19 14:58:15.689120 ip-10-0-138-220 kubenswrapper[2558]: E0719 14:58:15.688538 2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b50d3eba-b458-44a1-9949-0ee8c16e1e3c-registry-tls podName:b50d3eba-b458-44a1-9949-0ee8c16e1e3c nodeName:}" failed. No retries permitted until 2026-07-19 14:58:17.688518073 +0000 UTC m=+38.610220765 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/b50d3eba-b458-44a1-9949-0ee8c16e1e3c-registry-tls") pod "image-registry-575ddf7c75-4blsl" (UID: "b50d3eba-b458-44a1-9949-0ee8c16e1e3c") : secret "image-registry-tls" not found Jul 19 14:58:16.294532 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:58:16.294307 2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-hp977"] Jul 19 14:58:16.298073 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:58:16.298051 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-hp977" Jul 19 14:58:16.300731 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:58:16.300530 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-j74hj\"" Jul 19 14:58:16.301134 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:58:16.301117 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Jul 19 14:58:16.301388 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:58:16.301374 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Jul 19 14:58:16.301480 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:58:16.301407 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Jul 19 14:58:16.301480 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:58:16.301435 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Jul 19 14:58:16.306051 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:58:16.305924 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-hp977"] Jul 19 14:58:16.395366 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:58:16.395327 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/7fec0b2b-b831-414c-9a64-bf1b67c04fdd-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-hp977\" (UID: \"7fec0b2b-b831-414c-9a64-bf1b67c04fdd\") " pod="openshift-insights/insights-runtime-extractor-hp977" Jul 19 14:58:16.395454 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:58:16.395442 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/7fec0b2b-b831-414c-9a64-bf1b67c04fdd-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-hp977\" (UID: \"7fec0b2b-b831-414c-9a64-bf1b67c04fdd\") " pod="openshift-insights/insights-runtime-extractor-hp977" Jul 19 14:58:16.395510 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:58:16.395495 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/7fec0b2b-b831-414c-9a64-bf1b67c04fdd-crio-socket\") pod \"insights-runtime-extractor-hp977\" (UID: \"7fec0b2b-b831-414c-9a64-bf1b67c04fdd\") " pod="openshift-insights/insights-runtime-extractor-hp977" Jul 19 14:58:16.395556 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:58:16.395529 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/7fec0b2b-b831-414c-9a64-bf1b67c04fdd-data-volume\") pod \"insights-runtime-extractor-hp977\" (UID: \"7fec0b2b-b831-414c-9a64-bf1b67c04fdd\") " pod="openshift-insights/insights-runtime-extractor-hp977" Jul 19 14:58:16.395606 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:58:16.395556 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m4688\" (UniqueName: \"kubernetes.io/projected/7fec0b2b-b831-414c-9a64-bf1b67c04fdd-kube-api-access-m4688\") pod \"insights-runtime-extractor-hp977\" (UID: \"7fec0b2b-b831-414c-9a64-bf1b67c04fdd\") " pod="openshift-insights/insights-runtime-extractor-hp977" Jul 19 14:58:16.496862 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:58:16.496762 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/7fec0b2b-b831-414c-9a64-bf1b67c04fdd-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-hp977\" (UID: \"7fec0b2b-b831-414c-9a64-bf1b67c04fdd\") " pod="openshift-insights/insights-runtime-extractor-hp977" Jul 19 14:58:16.496862 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:58:16.496819 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/7fec0b2b-b831-414c-9a64-bf1b67c04fdd-crio-socket\") pod \"insights-runtime-extractor-hp977\" (UID: \"7fec0b2b-b831-414c-9a64-bf1b67c04fdd\") " pod="openshift-insights/insights-runtime-extractor-hp977" Jul 19 14:58:16.497017 ip-10-0-138-220 kubenswrapper[2558]: E0719 14:58:16.496894 2558 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 19 14:58:16.497017 ip-10-0-138-220 kubenswrapper[2558]: E0719 14:58:16.496963 2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/7fec0b2b-b831-414c-9a64-bf1b67c04fdd-kube-rbac-proxy-cm podName:7fec0b2b-b831-414c-9a64-bf1b67c04fdd nodeName:}" failed. No retries permitted until 2026-07-19 14:58:16.996947387 +0000 UTC m=+37.918650082 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/7fec0b2b-b831-414c-9a64-bf1b67c04fdd-kube-rbac-proxy-cm") pod "insights-runtime-extractor-hp977" (UID: "7fec0b2b-b831-414c-9a64-bf1b67c04fdd") : configmap "kube-rbac-proxy" not found Jul 19 14:58:16.497017 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:58:16.497005 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/7fec0b2b-b831-414c-9a64-bf1b67c04fdd-crio-socket\") pod \"insights-runtime-extractor-hp977\" (UID: \"7fec0b2b-b831-414c-9a64-bf1b67c04fdd\") " pod="openshift-insights/insights-runtime-extractor-hp977" Jul 19 14:58:16.497151 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:58:16.497068 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/7fec0b2b-b831-414c-9a64-bf1b67c04fdd-data-volume\") pod \"insights-runtime-extractor-hp977\" (UID: \"7fec0b2b-b831-414c-9a64-bf1b67c04fdd\") " pod="openshift-insights/insights-runtime-extractor-hp977" Jul 19 14:58:16.497151 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:58:16.497090 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m4688\" (UniqueName: \"kubernetes.io/projected/7fec0b2b-b831-414c-9a64-bf1b67c04fdd-kube-api-access-m4688\") pod \"insights-runtime-extractor-hp977\" (UID: \"7fec0b2b-b831-414c-9a64-bf1b67c04fdd\") " pod="openshift-insights/insights-runtime-extractor-hp977" Jul 19 14:58:16.497250 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:58:16.497148 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/7fec0b2b-b831-414c-9a64-bf1b67c04fdd-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-hp977\" (UID: \"7fec0b2b-b831-414c-9a64-bf1b67c04fdd\") " pod="openshift-insights/insights-runtime-extractor-hp977" Jul 19 14:58:16.497302 ip-10-0-138-220 kubenswrapper[2558]: E0719 14:58:16.497256 2558 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 19 14:58:16.497350 ip-10-0-138-220 kubenswrapper[2558]: E0719 14:58:16.497310 2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7fec0b2b-b831-414c-9a64-bf1b67c04fdd-insights-runtime-extractor-tls podName:7fec0b2b-b831-414c-9a64-bf1b67c04fdd nodeName:}" failed. No retries permitted until 2026-07-19 14:58:16.997294907 +0000 UTC m=+37.918997600 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/7fec0b2b-b831-414c-9a64-bf1b67c04fdd-insights-runtime-extractor-tls") pod "insights-runtime-extractor-hp977" (UID: "7fec0b2b-b831-414c-9a64-bf1b67c04fdd") : secret "insights-runtime-extractor-tls" not found Jul 19 14:58:16.497445 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:58:16.497426 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/7fec0b2b-b831-414c-9a64-bf1b67c04fdd-data-volume\") pod \"insights-runtime-extractor-hp977\" (UID: \"7fec0b2b-b831-414c-9a64-bf1b67c04fdd\") " pod="openshift-insights/insights-runtime-extractor-hp977" Jul 19 14:58:16.505733 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:58:16.505709 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m4688\" (UniqueName: \"kubernetes.io/projected/7fec0b2b-b831-414c-9a64-bf1b67c04fdd-kube-api-access-m4688\") pod \"insights-runtime-extractor-hp977\" (UID: \"7fec0b2b-b831-414c-9a64-bf1b67c04fdd\") " pod="openshift-insights/insights-runtime-extractor-hp977" Jul 19 14:58:16.826183 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:58:16.826102 2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-2k6h8"] Jul 19 14:58:16.829302 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:58:16.829282 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-2k6h8" Jul 19 14:58:16.831446 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:58:16.831421 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\"" Jul 19 14:58:16.831524 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:58:16.831474 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\"" Jul 19 14:58:16.831816 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:58:16.831799 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-srgmg\"" Jul 19 14:58:16.839943 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:58:16.838392 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-2k6h8"] Jul 19 14:58:16.857099 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:58:16.857073 2558 generic.go:356] "Generic (PLEG): container finished" podID="2571b663-0e56-4668-9831-fcdaf3793a1f" containerID="49e9483017acecd568dd8c20c4067962cb7fc27ff31aa30cc9aa87b5230e4b33" exitCode=0 Jul 19 14:58:16.857219 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:58:16.857130 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-vzds8" event={"ID":"2571b663-0e56-4668-9831-fcdaf3793a1f","Type":"ContainerDied","Data":"49e9483017acecd568dd8c20c4067962cb7fc27ff31aa30cc9aa87b5230e4b33"} Jul 19 14:58:17.002177 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:58:17.002147 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/7fec0b2b-b831-414c-9a64-bf1b67c04fdd-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-hp977\" (UID: \"7fec0b2b-b831-414c-9a64-bf1b67c04fdd\") " pod="openshift-insights/insights-runtime-extractor-hp977" Jul 19 14:58:17.002296 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:58:17.002258 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/7fec0b2b-b831-414c-9a64-bf1b67c04fdd-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-hp977\" (UID: \"7fec0b2b-b831-414c-9a64-bf1b67c04fdd\") " pod="openshift-insights/insights-runtime-extractor-hp977" Jul 19 14:58:17.002369 ip-10-0-138-220 kubenswrapper[2558]: E0719 14:58:17.002317 2558 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 19 14:58:17.002369 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:58:17.002335 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9qc55\" (UniqueName: \"kubernetes.io/projected/f6b9fc46-d49c-4ef3-98f8-244658f46ea9-kube-api-access-9qc55\") pod \"migrator-7cc9c7ffd7-2k6h8\" (UID: \"f6b9fc46-d49c-4ef3-98f8-244658f46ea9\") " pod="openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-2k6h8" Jul 19 14:58:17.002489 ip-10-0-138-220 kubenswrapper[2558]: E0719 14:58:17.002474 2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7fec0b2b-b831-414c-9a64-bf1b67c04fdd-insights-runtime-extractor-tls podName:7fec0b2b-b831-414c-9a64-bf1b67c04fdd nodeName:}" failed. No retries permitted until 2026-07-19 14:58:18.002451264 +0000 UTC m=+38.924153960 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/7fec0b2b-b831-414c-9a64-bf1b67c04fdd-insights-runtime-extractor-tls") pod "insights-runtime-extractor-hp977" (UID: "7fec0b2b-b831-414c-9a64-bf1b67c04fdd") : secret "insights-runtime-extractor-tls" not found Jul 19 14:58:17.002854 ip-10-0-138-220 kubenswrapper[2558]: E0719 14:58:17.002582 2558 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 19 14:58:17.002854 ip-10-0-138-220 kubenswrapper[2558]: E0719 14:58:17.002783 2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/7fec0b2b-b831-414c-9a64-bf1b67c04fdd-kube-rbac-proxy-cm podName:7fec0b2b-b831-414c-9a64-bf1b67c04fdd nodeName:}" failed. No retries permitted until 2026-07-19 14:58:18.002660843 +0000 UTC m=+38.924363546 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/7fec0b2b-b831-414c-9a64-bf1b67c04fdd-kube-rbac-proxy-cm") pod "insights-runtime-extractor-hp977" (UID: "7fec0b2b-b831-414c-9a64-bf1b67c04fdd") : configmap "kube-rbac-proxy" not found Jul 19 14:58:17.102960 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:58:17.102888 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9qc55\" (UniqueName: \"kubernetes.io/projected/f6b9fc46-d49c-4ef3-98f8-244658f46ea9-kube-api-access-9qc55\") pod \"migrator-7cc9c7ffd7-2k6h8\" (UID: \"f6b9fc46-d49c-4ef3-98f8-244658f46ea9\") " pod="openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-2k6h8" Jul 19 14:58:17.111500 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:58:17.111478 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9qc55\" (UniqueName: \"kubernetes.io/projected/f6b9fc46-d49c-4ef3-98f8-244658f46ea9-kube-api-access-9qc55\") pod \"migrator-7cc9c7ffd7-2k6h8\" (UID: \"f6b9fc46-d49c-4ef3-98f8-244658f46ea9\") " pod="openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-2k6h8" Jul 19 14:58:17.140329 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:58:17.140304 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-2k6h8" Jul 19 14:58:17.285547 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:58:17.285433 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-2k6h8"] Jul 19 14:58:17.710614 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:58:17.710570 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b50d3eba-b458-44a1-9949-0ee8c16e1e3c-registry-tls\") pod \"image-registry-575ddf7c75-4blsl\" (UID: \"b50d3eba-b458-44a1-9949-0ee8c16e1e3c\") " pod="openshift-image-registry/image-registry-575ddf7c75-4blsl" Jul 19 14:58:17.710787 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:58:17.710689 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/021eeff4-26d3-4e17-890e-19e1f81c1173-metrics-tls\") pod \"dns-default-kxhq6\" (UID: \"021eeff4-26d3-4e17-890e-19e1f81c1173\") " pod="openshift-dns/dns-default-kxhq6" Jul 19 14:58:17.710787 ip-10-0-138-220 kubenswrapper[2558]: E0719 14:58:17.710712 2558 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 19 14:58:17.710787 ip-10-0-138-220 kubenswrapper[2558]: E0719 14:58:17.710736 2558 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-575ddf7c75-4blsl: secret "image-registry-tls" not found Jul 19 14:58:17.710787 ip-10-0-138-220 kubenswrapper[2558]: E0719 14:58:17.710780 2558 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 19 14:58:17.710787 ip-10-0-138-220 kubenswrapper[2558]: E0719 14:58:17.710788 2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b50d3eba-b458-44a1-9949-0ee8c16e1e3c-registry-tls podName:b50d3eba-b458-44a1-9949-0ee8c16e1e3c nodeName:}" failed. No retries permitted until 2026-07-19 14:58:21.710772113 +0000 UTC m=+42.632474808 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/b50d3eba-b458-44a1-9949-0ee8c16e1e3c-registry-tls") pod "image-registry-575ddf7c75-4blsl" (UID: "b50d3eba-b458-44a1-9949-0ee8c16e1e3c") : secret "image-registry-tls" not found Jul 19 14:58:17.710978 ip-10-0-138-220 kubenswrapper[2558]: E0719 14:58:17.710830 2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/021eeff4-26d3-4e17-890e-19e1f81c1173-metrics-tls podName:021eeff4-26d3-4e17-890e-19e1f81c1173 nodeName:}" failed. No retries permitted until 2026-07-19 14:58:21.710813875 +0000 UTC m=+42.632516586 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/021eeff4-26d3-4e17-890e-19e1f81c1173-metrics-tls") pod "dns-default-kxhq6" (UID: "021eeff4-26d3-4e17-890e-19e1f81c1173") : secret "dns-default-metrics-tls" not found Jul 19 14:58:17.864054 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:58:17.863997 2558 generic.go:356] "Generic (PLEG): container finished" podID="2571b663-0e56-4668-9831-fcdaf3793a1f" containerID="15722c6145eccfdefcfa63381a8a1ba866557c832e034cc23cbcab1ccfaf33c0" exitCode=0 Jul 19 14:58:17.864423 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:58:17.864084 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-vzds8" event={"ID":"2571b663-0e56-4668-9831-fcdaf3793a1f","Type":"ContainerDied","Data":"15722c6145eccfdefcfa63381a8a1ba866557c832e034cc23cbcab1ccfaf33c0"} Jul 19 14:58:17.959236 ip-10-0-138-220 kubenswrapper[2558]: W0719 14:58:17.959203 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf6b9fc46_d49c_4ef3_98f8_244658f46ea9.slice/crio-2e20dec513f21363cb4b46c6196603241090ffbfd4c4d862bf011744a28b8fb0 WatchSource:0}: Error finding container 2e20dec513f21363cb4b46c6196603241090ffbfd4c4d862bf011744a28b8fb0: Status 404 returned error can't find the container with id 2e20dec513f21363cb4b46c6196603241090ffbfd4c4d862bf011744a28b8fb0 Jul 19 14:58:18.013079 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:58:18.012901 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/7fec0b2b-b831-414c-9a64-bf1b67c04fdd-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-hp977\" (UID: \"7fec0b2b-b831-414c-9a64-bf1b67c04fdd\") " pod="openshift-insights/insights-runtime-extractor-hp977" Jul 19 14:58:18.013168 ip-10-0-138-220 kubenswrapper[2558]: E0719 14:58:18.013096 2558 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 19 14:58:18.013205 ip-10-0-138-220 kubenswrapper[2558]: E0719 14:58:18.013178 2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7fec0b2b-b831-414c-9a64-bf1b67c04fdd-insights-runtime-extractor-tls podName:7fec0b2b-b831-414c-9a64-bf1b67c04fdd nodeName:}" failed. No retries permitted until 2026-07-19 14:58:20.013161223 +0000 UTC m=+40.934863915 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/7fec0b2b-b831-414c-9a64-bf1b67c04fdd-insights-runtime-extractor-tls") pod "insights-runtime-extractor-hp977" (UID: "7fec0b2b-b831-414c-9a64-bf1b67c04fdd") : secret "insights-runtime-extractor-tls" not found Jul 19 14:58:18.013253 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:58:18.013220 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/7fec0b2b-b831-414c-9a64-bf1b67c04fdd-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-hp977\" (UID: \"7fec0b2b-b831-414c-9a64-bf1b67c04fdd\") " pod="openshift-insights/insights-runtime-extractor-hp977" Jul 19 14:58:18.013298 ip-10-0-138-220 kubenswrapper[2558]: E0719 14:58:18.013286 2558 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 19 14:58:18.013330 ip-10-0-138-220 kubenswrapper[2558]: E0719 14:58:18.013322 2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/7fec0b2b-b831-414c-9a64-bf1b67c04fdd-kube-rbac-proxy-cm podName:7fec0b2b-b831-414c-9a64-bf1b67c04fdd nodeName:}" failed. No retries permitted until 2026-07-19 14:58:20.013312262 +0000 UTC m=+40.935014953 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/7fec0b2b-b831-414c-9a64-bf1b67c04fdd-kube-rbac-proxy-cm") pod "insights-runtime-extractor-hp977" (UID: "7fec0b2b-b831-414c-9a64-bf1b67c04fdd") : configmap "kube-rbac-proxy" not found Jul 19 14:58:18.080250 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:58:18.080230 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-qsm4z_8c1b2505-369a-4989-ac89-344f9d9991f9/dns-node-resolver/0.log" Jul 19 14:58:18.871221 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:58:18.871188 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-vzds8" event={"ID":"2571b663-0e56-4668-9831-fcdaf3793a1f","Type":"ContainerStarted","Data":"7ca247191232701adf4ccc6bb0e6a42391aca461d4f6b05913c150f903ee8f0e"} Jul 19 14:58:18.873206 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:58:18.873179 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-7prf4" event={"ID":"47e880d5-eff9-4f7b-9c47-b7af7f74dc41","Type":"ContainerStarted","Data":"d4e460e14e24e97508b77f1e066d481f4190000f2da355571a9762bdf9900992"} Jul 19 14:58:18.874457 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:58:18.874430 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-2k6h8" event={"ID":"f6b9fc46-d49c-4ef3-98f8-244658f46ea9","Type":"ContainerStarted","Data":"2e20dec513f21363cb4b46c6196603241090ffbfd4c4d862bf011744a28b8fb0"} Jul 19 14:58:18.880131 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:58:18.880091 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-tnrrh_80a8f70e-f1a5-408d-bb95-b347a0ec8715/node-ca/0.log" Jul 19 14:58:18.892848 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:58:18.892796 2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-vzds8" podStartSLOduration=9.009507117 podStartE2EDuration="39.892782172s" totalImagesPullingTime="30.883275055s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-19 14:57:39 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-19 14:58:18.892588907 +0000 UTC m=+39.814291704" watchObservedRunningTime="2026-07-19 14:58:18.892782172 +0000 UTC m=+39.814484885" Jul 19 14:58:18.907144 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:58:18.907092 2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-7prf4" podStartSLOduration=2.26613704 podStartE2EDuration="5.907082315s" totalImagesPullingTime="3.640945275s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-19 14:58:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-19 14:58:18.906163193 +0000 UTC m=+39.827865907" watchObservedRunningTime="2026-07-19 14:58:18.907082315 +0000 UTC m=+39.828785028" Jul 19 14:58:19.482259 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:58:19.482236 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-7prf4_47e880d5-eff9-4f7b-9c47-b7af7f74dc41/serve-healthcheck-canary/0.log" Jul 19 14:58:19.507393 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:58:19.507365 2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-58598695f7-7rprz"] Jul 19 14:58:19.532349 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:58:19.532329 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-58598695f7-7rprz"] Jul 19 14:58:19.532485 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:58:19.532407 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-58598695f7-7rprz" Jul 19 14:58:19.535223 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:58:19.535207 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\"" Jul 19 14:58:19.535223 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:58:19.535210 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\"" Jul 19 14:58:19.535353 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:58:19.535243 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\"" Jul 19 14:58:19.535985 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:58:19.535959 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-nsprw\"" Jul 19 14:58:19.536082 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:58:19.536067 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\"" Jul 19 14:58:19.627629 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:58:19.627599 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-87k9w\" (UniqueName: \"kubernetes.io/projected/50367d07-e381-4218-90a8-29c3b81e29db-kube-api-access-87k9w\") pod \"service-ca-58598695f7-7rprz\" (UID: \"50367d07-e381-4218-90a8-29c3b81e29db\") " pod="openshift-service-ca/service-ca-58598695f7-7rprz" Jul 19 14:58:19.627767 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:58:19.627641 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/50367d07-e381-4218-90a8-29c3b81e29db-signing-cabundle\") pod \"service-ca-58598695f7-7rprz\" (UID: \"50367d07-e381-4218-90a8-29c3b81e29db\") " pod="openshift-service-ca/service-ca-58598695f7-7rprz" Jul 19 14:58:19.627767 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:58:19.627709 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/50367d07-e381-4218-90a8-29c3b81e29db-signing-key\") pod \"service-ca-58598695f7-7rprz\" (UID: \"50367d07-e381-4218-90a8-29c3b81e29db\") " pod="openshift-service-ca/service-ca-58598695f7-7rprz" Jul 19 14:58:19.728667 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:58:19.728637 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-87k9w\" (UniqueName: \"kubernetes.io/projected/50367d07-e381-4218-90a8-29c3b81e29db-kube-api-access-87k9w\") pod \"service-ca-58598695f7-7rprz\" (UID: \"50367d07-e381-4218-90a8-29c3b81e29db\") " pod="openshift-service-ca/service-ca-58598695f7-7rprz" Jul 19 14:58:19.728793 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:58:19.728706 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/50367d07-e381-4218-90a8-29c3b81e29db-signing-cabundle\") pod \"service-ca-58598695f7-7rprz\" (UID: \"50367d07-e381-4218-90a8-29c3b81e29db\") " pod="openshift-service-ca/service-ca-58598695f7-7rprz" Jul 19 14:58:19.728939 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:58:19.728914 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/50367d07-e381-4218-90a8-29c3b81e29db-signing-key\") pod \"service-ca-58598695f7-7rprz\" (UID: \"50367d07-e381-4218-90a8-29c3b81e29db\") " pod="openshift-service-ca/service-ca-58598695f7-7rprz" Jul 19 14:58:19.729279 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:58:19.729261 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/50367d07-e381-4218-90a8-29c3b81e29db-signing-cabundle\") pod \"service-ca-58598695f7-7rprz\" (UID: \"50367d07-e381-4218-90a8-29c3b81e29db\") " pod="openshift-service-ca/service-ca-58598695f7-7rprz" Jul 19 14:58:19.732345 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:58:19.732305 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/50367d07-e381-4218-90a8-29c3b81e29db-signing-key\") pod \"service-ca-58598695f7-7rprz\" (UID: \"50367d07-e381-4218-90a8-29c3b81e29db\") " pod="openshift-service-ca/service-ca-58598695f7-7rprz" Jul 19 14:58:19.735947 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:58:19.735930 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-87k9w\" (UniqueName: \"kubernetes.io/projected/50367d07-e381-4218-90a8-29c3b81e29db-kube-api-access-87k9w\") pod \"service-ca-58598695f7-7rprz\" (UID: \"50367d07-e381-4218-90a8-29c3b81e29db\") " pod="openshift-service-ca/service-ca-58598695f7-7rprz" Jul 19 14:58:19.841272 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:58:19.841250 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-58598695f7-7rprz" Jul 19 14:58:19.879334 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:58:19.878675 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-2k6h8" event={"ID":"f6b9fc46-d49c-4ef3-98f8-244658f46ea9","Type":"ContainerStarted","Data":"31fba25c76e103eaa6c16353df12ecc7cafd4e337e369004c44ad5ca95d1a316"} Jul 19 14:58:19.990999 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:58:19.990933 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-58598695f7-7rprz"] Jul 19 14:58:19.994538 ip-10-0-138-220 kubenswrapper[2558]: W0719 14:58:19.994513 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod50367d07_e381_4218_90a8_29c3b81e29db.slice/crio-7087f8ec478d332f7241d685697a64cd85aba1abacb56b05db37c795766358f5 WatchSource:0}: Error finding container 7087f8ec478d332f7241d685697a64cd85aba1abacb56b05db37c795766358f5: Status 404 returned error can't find the container with id 7087f8ec478d332f7241d685697a64cd85aba1abacb56b05db37c795766358f5 Jul 19 14:58:20.030491 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:58:20.030467 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/7fec0b2b-b831-414c-9a64-bf1b67c04fdd-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-hp977\" (UID: \"7fec0b2b-b831-414c-9a64-bf1b67c04fdd\") " pod="openshift-insights/insights-runtime-extractor-hp977" Jul 19 14:58:20.030576 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:58:20.030516 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/7fec0b2b-b831-414c-9a64-bf1b67c04fdd-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-hp977\" (UID: \"7fec0b2b-b831-414c-9a64-bf1b67c04fdd\") " pod="openshift-insights/insights-runtime-extractor-hp977" Jul 19 14:58:20.030613 ip-10-0-138-220 kubenswrapper[2558]: E0719 14:58:20.030590 2558 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 19 14:58:20.030644 ip-10-0-138-220 kubenswrapper[2558]: E0719 14:58:20.030608 2558 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 19 14:58:20.030644 ip-10-0-138-220 kubenswrapper[2558]: E0719 14:58:20.030637 2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/7fec0b2b-b831-414c-9a64-bf1b67c04fdd-kube-rbac-proxy-cm podName:7fec0b2b-b831-414c-9a64-bf1b67c04fdd nodeName:}" failed. No retries permitted until 2026-07-19 14:58:24.030620752 +0000 UTC m=+44.952323444 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/7fec0b2b-b831-414c-9a64-bf1b67c04fdd-kube-rbac-proxy-cm") pod "insights-runtime-extractor-hp977" (UID: "7fec0b2b-b831-414c-9a64-bf1b67c04fdd") : configmap "kube-rbac-proxy" not found Jul 19 14:58:20.030711 ip-10-0-138-220 kubenswrapper[2558]: E0719 14:58:20.030667 2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7fec0b2b-b831-414c-9a64-bf1b67c04fdd-insights-runtime-extractor-tls podName:7fec0b2b-b831-414c-9a64-bf1b67c04fdd nodeName:}" failed. No retries permitted until 2026-07-19 14:58:24.030653685 +0000 UTC m=+44.952356381 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/7fec0b2b-b831-414c-9a64-bf1b67c04fdd-insights-runtime-extractor-tls") pod "insights-runtime-extractor-hp977" (UID: "7fec0b2b-b831-414c-9a64-bf1b67c04fdd") : secret "insights-runtime-extractor-tls" not found Jul 19 14:58:20.883218 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:58:20.883171 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-2k6h8" event={"ID":"f6b9fc46-d49c-4ef3-98f8-244658f46ea9","Type":"ContainerStarted","Data":"2db9ccee7e0e5d9f0351d3c3ade73351b50c63ca09b0e37b21578bafbca45b2f"} Jul 19 14:58:20.884510 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:58:20.884481 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-58598695f7-7rprz" event={"ID":"50367d07-e381-4218-90a8-29c3b81e29db","Type":"ContainerStarted","Data":"7087f8ec478d332f7241d685697a64cd85aba1abacb56b05db37c795766358f5"} Jul 19 14:58:20.897890 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:58:20.897843 2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-2k6h8" podStartSLOduration=3.172875236 podStartE2EDuration="4.897823923s" totalImagesPullingTime="1.724948687s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-19 14:58:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-19 14:58:20.897066598 +0000 UTC m=+41.818769312" watchObservedRunningTime="2026-07-19 14:58:20.897823923 +0000 UTC m=+41.819526638" Jul 19 14:58:21.744228 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:58:21.744179 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b50d3eba-b458-44a1-9949-0ee8c16e1e3c-registry-tls\") pod \"image-registry-575ddf7c75-4blsl\" (UID: \"b50d3eba-b458-44a1-9949-0ee8c16e1e3c\") " pod="openshift-image-registry/image-registry-575ddf7c75-4blsl" Jul 19 14:58:21.744397 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:58:21.744284 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/021eeff4-26d3-4e17-890e-19e1f81c1173-metrics-tls\") pod \"dns-default-kxhq6\" (UID: \"021eeff4-26d3-4e17-890e-19e1f81c1173\") " pod="openshift-dns/dns-default-kxhq6" Jul 19 14:58:21.744397 ip-10-0-138-220 kubenswrapper[2558]: E0719 14:58:21.744330 2558 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 19 14:58:21.744397 ip-10-0-138-220 kubenswrapper[2558]: E0719 14:58:21.744351 2558 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-575ddf7c75-4blsl: secret "image-registry-tls" not found Jul 19 14:58:21.744397 ip-10-0-138-220 kubenswrapper[2558]: E0719 14:58:21.744366 2558 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 19 14:58:21.744512 ip-10-0-138-220 kubenswrapper[2558]: E0719 14:58:21.744407 2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b50d3eba-b458-44a1-9949-0ee8c16e1e3c-registry-tls podName:b50d3eba-b458-44a1-9949-0ee8c16e1e3c nodeName:}" failed. No retries permitted until 2026-07-19 14:58:29.744389886 +0000 UTC m=+50.666092578 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/b50d3eba-b458-44a1-9949-0ee8c16e1e3c-registry-tls") pod "image-registry-575ddf7c75-4blsl" (UID: "b50d3eba-b458-44a1-9949-0ee8c16e1e3c") : secret "image-registry-tls" not found Jul 19 14:58:21.744512 ip-10-0-138-220 kubenswrapper[2558]: E0719 14:58:21.744420 2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/021eeff4-26d3-4e17-890e-19e1f81c1173-metrics-tls podName:021eeff4-26d3-4e17-890e-19e1f81c1173 nodeName:}" failed. No retries permitted until 2026-07-19 14:58:29.744414382 +0000 UTC m=+50.666117073 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/021eeff4-26d3-4e17-890e-19e1f81c1173-metrics-tls") pod "dns-default-kxhq6" (UID: "021eeff4-26d3-4e17-890e-19e1f81c1173") : secret "dns-default-metrics-tls" not found Jul 19 14:58:22.890715 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:58:22.890680 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-58598695f7-7rprz" event={"ID":"50367d07-e381-4218-90a8-29c3b81e29db","Type":"ContainerStarted","Data":"57c306871e64b7ae18d324ce3c79e6660c2eaa3d5137ecd0d5c1d01b0f66b532"} Jul 19 14:58:22.910785 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:58:22.910729 2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca/service-ca-58598695f7-7rprz" podStartSLOduration=1.694318434 podStartE2EDuration="3.910705282s" totalImagesPullingTime="2.216386848s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-19 14:58:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-19 14:58:22.908919121 +0000 UTC m=+43.830621832" watchObservedRunningTime="2026-07-19 14:58:22.910705282 +0000 UTC m=+43.832407994" Jul 19 14:58:24.063495 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:58:24.063458 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/7fec0b2b-b831-414c-9a64-bf1b67c04fdd-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-hp977\" (UID: \"7fec0b2b-b831-414c-9a64-bf1b67c04fdd\") " pod="openshift-insights/insights-runtime-extractor-hp977" Jul 19 14:58:24.063959 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:58:24.063577 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/7fec0b2b-b831-414c-9a64-bf1b67c04fdd-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-hp977\" (UID: \"7fec0b2b-b831-414c-9a64-bf1b67c04fdd\") " pod="openshift-insights/insights-runtime-extractor-hp977" Jul 19 14:58:24.063959 ip-10-0-138-220 kubenswrapper[2558]: E0719 14:58:24.063705 2558 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 19 14:58:24.063959 ip-10-0-138-220 kubenswrapper[2558]: E0719 14:58:24.063767 2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7fec0b2b-b831-414c-9a64-bf1b67c04fdd-insights-runtime-extractor-tls podName:7fec0b2b-b831-414c-9a64-bf1b67c04fdd nodeName:}" failed. No retries permitted until 2026-07-19 14:58:32.063748855 +0000 UTC m=+52.985451550 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/7fec0b2b-b831-414c-9a64-bf1b67c04fdd-insights-runtime-extractor-tls") pod "insights-runtime-extractor-hp977" (UID: "7fec0b2b-b831-414c-9a64-bf1b67c04fdd") : secret "insights-runtime-extractor-tls" not found Jul 19 14:58:24.064143 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:58:24.063998 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/7fec0b2b-b831-414c-9a64-bf1b67c04fdd-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-hp977\" (UID: \"7fec0b2b-b831-414c-9a64-bf1b67c04fdd\") " pod="openshift-insights/insights-runtime-extractor-hp977" Jul 19 14:58:29.805666 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:58:29.805628 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/021eeff4-26d3-4e17-890e-19e1f81c1173-metrics-tls\") pod \"dns-default-kxhq6\" (UID: \"021eeff4-26d3-4e17-890e-19e1f81c1173\") " pod="openshift-dns/dns-default-kxhq6" Jul 19 14:58:29.806162 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:58:29.805675 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b50d3eba-b458-44a1-9949-0ee8c16e1e3c-registry-tls\") pod \"image-registry-575ddf7c75-4blsl\" (UID: \"b50d3eba-b458-44a1-9949-0ee8c16e1e3c\") " pod="openshift-image-registry/image-registry-575ddf7c75-4blsl" Jul 19 14:58:29.808183 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:58:29.808155 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/021eeff4-26d3-4e17-890e-19e1f81c1173-metrics-tls\") pod \"dns-default-kxhq6\" (UID: \"021eeff4-26d3-4e17-890e-19e1f81c1173\") " pod="openshift-dns/dns-default-kxhq6" Jul 19 14:58:29.808296 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:58:29.808202 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b50d3eba-b458-44a1-9949-0ee8c16e1e3c-registry-tls\") pod \"image-registry-575ddf7c75-4blsl\" (UID: \"b50d3eba-b458-44a1-9949-0ee8c16e1e3c\") " pod="openshift-image-registry/image-registry-575ddf7c75-4blsl" Jul 19 14:58:29.820177 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:58:29.820153 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-kxhq6" Jul 19 14:58:29.960100 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:58:29.960056 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-kxhq6"] Jul 19 14:58:29.963375 ip-10-0-138-220 kubenswrapper[2558]: W0719 14:58:29.963348 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod021eeff4_26d3_4e17_890e_19e1f81c1173.slice/crio-6788e28b612224d8a5de95159daf140a0368f8eebca26d9b15f4642a074f3789 WatchSource:0}: Error finding container 6788e28b612224d8a5de95159daf140a0368f8eebca26d9b15f4642a074f3789: Status 404 returned error can't find the container with id 6788e28b612224d8a5de95159daf140a0368f8eebca26d9b15f4642a074f3789 Jul 19 14:58:30.104533 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:58:30.104472 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-575ddf7c75-4blsl" Jul 19 14:58:30.256991 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:58:30.256963 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-575ddf7c75-4blsl"] Jul 19 14:58:30.260093 ip-10-0-138-220 kubenswrapper[2558]: W0719 14:58:30.260064 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb50d3eba_b458_44a1_9949_0ee8c16e1e3c.slice/crio-140ddc9c3bb23c93627262006741503cee3e63ffd9e224f080c099f274fe8636 WatchSource:0}: Error finding container 140ddc9c3bb23c93627262006741503cee3e63ffd9e224f080c099f274fe8636: Status 404 returned error can't find the container with id 140ddc9c3bb23c93627262006741503cee3e63ffd9e224f080c099f274fe8636 Jul 19 14:58:30.916542 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:58:30.916503 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-575ddf7c75-4blsl" event={"ID":"b50d3eba-b458-44a1-9949-0ee8c16e1e3c","Type":"ContainerStarted","Data":"0065fc694e0c2fcee12923fcbdd3a62ba01f09be55f5074b1c3da2495b991ca2"} Jul 19 14:58:30.916994 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:58:30.916561 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-575ddf7c75-4blsl" event={"ID":"b50d3eba-b458-44a1-9949-0ee8c16e1e3c","Type":"ContainerStarted","Data":"140ddc9c3bb23c93627262006741503cee3e63ffd9e224f080c099f274fe8636"} Jul 19 14:58:30.916994 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:58:30.916695 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-575ddf7c75-4blsl" Jul 19 14:58:30.919205 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:58:30.919180 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-kxhq6" event={"ID":"021eeff4-26d3-4e17-890e-19e1f81c1173","Type":"ContainerStarted","Data":"6788e28b612224d8a5de95159daf140a0368f8eebca26d9b15f4642a074f3789"} Jul 19 14:58:30.936788 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:58:30.936738 2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-575ddf7c75-4blsl" podStartSLOduration=25.936719919 podStartE2EDuration="25.936719919s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-19 14:58:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-19 14:58:30.935793352 +0000 UTC m=+51.857496068" watchObservedRunningTime="2026-07-19 14:58:30.936719919 +0000 UTC m=+51.858422634" Jul 19 14:58:32.126373 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:58:32.126343 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/7fec0b2b-b831-414c-9a64-bf1b67c04fdd-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-hp977\" (UID: \"7fec0b2b-b831-414c-9a64-bf1b67c04fdd\") " pod="openshift-insights/insights-runtime-extractor-hp977" Jul 19 14:58:32.128691 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:58:32.128670 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/7fec0b2b-b831-414c-9a64-bf1b67c04fdd-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-hp977\" (UID: \"7fec0b2b-b831-414c-9a64-bf1b67c04fdd\") " pod="openshift-insights/insights-runtime-extractor-hp977" Jul 19 14:58:32.209057 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:58:32.209015 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-hp977" Jul 19 14:58:32.362387 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:58:32.362364 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-hp977"] Jul 19 14:58:32.368193 ip-10-0-138-220 kubenswrapper[2558]: W0719 14:58:32.367188 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7fec0b2b_b831_414c_9a64_bf1b67c04fdd.slice/crio-cf530745e5864d9128f7fedd22a65ba6e3caf7b782fdad295a0823b060fb2b00 WatchSource:0}: Error finding container cf530745e5864d9128f7fedd22a65ba6e3caf7b782fdad295a0823b060fb2b00: Status 404 returned error can't find the container with id cf530745e5864d9128f7fedd22a65ba6e3caf7b782fdad295a0823b060fb2b00 Jul 19 14:58:32.930002 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:58:32.929964 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-kxhq6" event={"ID":"021eeff4-26d3-4e17-890e-19e1f81c1173","Type":"ContainerStarted","Data":"a67312016ad4ea13c9bddfcc9f4928e54a2c2b0282dcabff3b83b260676bd4d0"} Jul 19 14:58:32.930204 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:58:32.930097 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-kxhq6" Jul 19 14:58:32.930204 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:58:32.930127 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-kxhq6" event={"ID":"021eeff4-26d3-4e17-890e-19e1f81c1173","Type":"ContainerStarted","Data":"286c29f9df1de07acd92e6395138717d4195710fa7fe8942db926c55ad4d3342"} Jul 19 14:58:32.932347 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:58:32.932315 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-hp977" event={"ID":"7fec0b2b-b831-414c-9a64-bf1b67c04fdd","Type":"ContainerStarted","Data":"cf530745e5864d9128f7fedd22a65ba6e3caf7b782fdad295a0823b060fb2b00"} Jul 19 14:58:32.946074 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:58:32.946008 2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-kxhq6" podStartSLOduration=17.721193751 podStartE2EDuration="19.945989547s" totalImagesPullingTime="2.224795796s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-19 14:58:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-19 14:58:32.94523187 +0000 UTC m=+53.866934585" watchObservedRunningTime="2026-07-19 14:58:32.945989547 +0000 UTC m=+53.867692262" Jul 19 14:58:37.951276 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:58:37.951228 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-hp977" event={"ID":"7fec0b2b-b831-414c-9a64-bf1b67c04fdd","Type":"ContainerStarted","Data":"7dab6805aeff547d07359bfa25fd71b81caae748c392e848542c4f83bd2249d6"} Jul 19 14:58:39.852695 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:58:39.852660 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-5wfzs" Jul 19 14:58:41.340289 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:58:41.340211 2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-5b86c47c46-vjgpv"] Jul 19 14:58:41.342252 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:58:41.342229 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5b86c47c46-vjgpv" Jul 19 14:58:41.344952 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:58:41.344931 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\"" Jul 19 14:58:41.344952 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:58:41.344945 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"work-manager-hub-kubeconfig\"" Jul 19 14:58:41.345643 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:58:41.345625 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\"" Jul 19 14:58:41.345643 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:58:41.345633 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\"" Jul 19 14:58:41.353998 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:58:41.353980 2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b775f4854-5blzr"] Jul 19 14:58:41.355836 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:58:41.355820 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-5b86c47c46-vjgpv"] Jul 19 14:58:41.355933 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:58:41.355910 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b775f4854-5blzr" Jul 19 14:58:41.358246 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:58:41.358229 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-open-cluster-management.io-proxy-agent-signer-client-cert\"" Jul 19 14:58:41.358578 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:58:41.358561 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-hub-kubeconfig\"" Jul 19 14:58:41.358794 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:58:41.358776 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-service-proxy-server-certificates\"" Jul 19 14:58:41.358858 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:58:41.358848 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-ca\"" Jul 19 14:58:41.363148 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:58:41.363132 2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-575ddf7c75-4blsl"] Jul 19 14:58:41.372127 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:58:41.372109 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b775f4854-5blzr"] Jul 19 14:58:41.406204 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:58:41.406176 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/3a074892-b609-4143-86f3-9c3102f66c36-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-6b775f4854-5blzr\" (UID: \"3a074892-b609-4143-86f3-9c3102f66c36\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b775f4854-5blzr" Jul 19 14:58:41.406322 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:58:41.406209 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/3a074892-b609-4143-86f3-9c3102f66c36-additional-service-ca\") pod \"cluster-proxy-proxy-agent-6b775f4854-5blzr\" (UID: \"3a074892-b609-4143-86f3-9c3102f66c36\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b775f4854-5blzr" Jul 19 14:58:41.406322 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:58:41.406233 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/3a074892-b609-4143-86f3-9c3102f66c36-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-6b775f4854-5blzr\" (UID: \"3a074892-b609-4143-86f3-9c3102f66c36\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b775f4854-5blzr" Jul 19 14:58:41.406322 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:58:41.406265 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/3a074892-b609-4143-86f3-9c3102f66c36-hub\") pod \"cluster-proxy-proxy-agent-6b775f4854-5blzr\" (UID: \"3a074892-b609-4143-86f3-9c3102f66c36\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b775f4854-5blzr" Jul 19 14:58:41.406444 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:58:41.406322 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/06032cf2-951e-4323-8b39-19401b6f9c4f-tmp\") pod \"klusterlet-addon-workmgr-5b86c47c46-vjgpv\" (UID: \"06032cf2-951e-4323-8b39-19401b6f9c4f\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5b86c47c46-vjgpv" Jul 19 14:58:41.406444 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:58:41.406337 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/3a074892-b609-4143-86f3-9c3102f66c36-ca\") pod \"cluster-proxy-proxy-agent-6b775f4854-5blzr\" (UID: \"3a074892-b609-4143-86f3-9c3102f66c36\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b775f4854-5blzr" Jul 19 14:58:41.406444 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:58:41.406381 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-htwpd\" (UniqueName: \"kubernetes.io/projected/06032cf2-951e-4323-8b39-19401b6f9c4f-kube-api-access-htwpd\") pod \"klusterlet-addon-workmgr-5b86c47c46-vjgpv\" (UID: \"06032cf2-951e-4323-8b39-19401b6f9c4f\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5b86c47c46-vjgpv" Jul 19 14:58:41.406444 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:58:41.406411 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sbzht\" (UniqueName: \"kubernetes.io/projected/3a074892-b609-4143-86f3-9c3102f66c36-kube-api-access-sbzht\") pod \"cluster-proxy-proxy-agent-6b775f4854-5blzr\" (UID: \"3a074892-b609-4143-86f3-9c3102f66c36\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b775f4854-5blzr" Jul 19 14:58:41.406575 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:58:41.406444 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/06032cf2-951e-4323-8b39-19401b6f9c4f-klusterlet-config\") pod \"klusterlet-addon-workmgr-5b86c47c46-vjgpv\" (UID: \"06032cf2-951e-4323-8b39-19401b6f9c4f\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5b86c47c46-vjgpv" Jul 19 14:58:41.441470 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:58:41.441445 2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-6f485ff8ff-jrz6w"] Jul 19 14:58:41.443205 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:58:41.443189 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6f485ff8ff-jrz6w" Jul 19 14:58:41.460847 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:58:41.460821 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-6f485ff8ff-jrz6w"] Jul 19 14:58:41.507751 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:58:41.507719 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/06032cf2-951e-4323-8b39-19401b6f9c4f-klusterlet-config\") pod \"klusterlet-addon-workmgr-5b86c47c46-vjgpv\" (UID: \"06032cf2-951e-4323-8b39-19401b6f9c4f\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5b86c47c46-vjgpv" Jul 19 14:58:41.507914 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:58:41.507773 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/1e4e44f7-d447-4d70-bf92-656f49c76bb0-image-registry-private-configuration\") pod \"image-registry-6f485ff8ff-jrz6w\" (UID: \"1e4e44f7-d447-4d70-bf92-656f49c76bb0\") " pod="openshift-image-registry/image-registry-6f485ff8ff-jrz6w" Jul 19 14:58:41.507914 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:58:41.507809 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/1e4e44f7-d447-4d70-bf92-656f49c76bb0-installation-pull-secrets\") pod \"image-registry-6f485ff8ff-jrz6w\" (UID: \"1e4e44f7-d447-4d70-bf92-656f49c76bb0\") " pod="openshift-image-registry/image-registry-6f485ff8ff-jrz6w" Jul 19 14:58:41.507914 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:58:41.507841 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/3a074892-b609-4143-86f3-9c3102f66c36-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-6b775f4854-5blzr\" (UID: \"3a074892-b609-4143-86f3-9c3102f66c36\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b775f4854-5blzr" Jul 19 14:58:41.507914 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:58:41.507902 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2jzpq\" (UniqueName: \"kubernetes.io/projected/1e4e44f7-d447-4d70-bf92-656f49c76bb0-kube-api-access-2jzpq\") pod \"image-registry-6f485ff8ff-jrz6w\" (UID: \"1e4e44f7-d447-4d70-bf92-656f49c76bb0\") " pod="openshift-image-registry/image-registry-6f485ff8ff-jrz6w" Jul 19 14:58:41.508154 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:58:41.507941 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/1e4e44f7-d447-4d70-bf92-656f49c76bb0-registry-tls\") pod \"image-registry-6f485ff8ff-jrz6w\" (UID: \"1e4e44f7-d447-4d70-bf92-656f49c76bb0\") " pod="openshift-image-registry/image-registry-6f485ff8ff-jrz6w" Jul 19 14:58:41.508154 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:58:41.507979 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/3a074892-b609-4143-86f3-9c3102f66c36-additional-service-ca\") pod \"cluster-proxy-proxy-agent-6b775f4854-5blzr\" (UID: \"3a074892-b609-4143-86f3-9c3102f66c36\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b775f4854-5blzr" Jul 19 14:58:41.508154 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:58:41.508017 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/3a074892-b609-4143-86f3-9c3102f66c36-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-6b775f4854-5blzr\" (UID: \"3a074892-b609-4143-86f3-9c3102f66c36\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b775f4854-5blzr" Jul 19 14:58:41.508154 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:58:41.508064 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/1e4e44f7-d447-4d70-bf92-656f49c76bb0-registry-certificates\") pod \"image-registry-6f485ff8ff-jrz6w\" (UID: \"1e4e44f7-d447-4d70-bf92-656f49c76bb0\") " pod="openshift-image-registry/image-registry-6f485ff8ff-jrz6w" Jul 19 14:58:41.508154 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:58:41.508091 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/3a074892-b609-4143-86f3-9c3102f66c36-hub\") pod \"cluster-proxy-proxy-agent-6b775f4854-5blzr\" (UID: \"3a074892-b609-4143-86f3-9c3102f66c36\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b775f4854-5blzr" Jul 19 14:58:41.508154 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:58:41.508138 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/1e4e44f7-d447-4d70-bf92-656f49c76bb0-trusted-ca\") pod \"image-registry-6f485ff8ff-jrz6w\" (UID: \"1e4e44f7-d447-4d70-bf92-656f49c76bb0\") " pod="openshift-image-registry/image-registry-6f485ff8ff-jrz6w" Jul 19 14:58:41.508439 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:58:41.508182 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/1e4e44f7-d447-4d70-bf92-656f49c76bb0-ca-trust-extracted\") pod \"image-registry-6f485ff8ff-jrz6w\" (UID: \"1e4e44f7-d447-4d70-bf92-656f49c76bb0\") " pod="openshift-image-registry/image-registry-6f485ff8ff-jrz6w" Jul 19 14:58:41.508439 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:58:41.508290 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/06032cf2-951e-4323-8b39-19401b6f9c4f-tmp\") pod \"klusterlet-addon-workmgr-5b86c47c46-vjgpv\" (UID: \"06032cf2-951e-4323-8b39-19401b6f9c4f\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5b86c47c46-vjgpv" Jul 19 14:58:41.508439 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:58:41.508315 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/3a074892-b609-4143-86f3-9c3102f66c36-ca\") pod \"cluster-proxy-proxy-agent-6b775f4854-5blzr\" (UID: \"3a074892-b609-4143-86f3-9c3102f66c36\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b775f4854-5blzr" Jul 19 14:58:41.508589 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:58:41.508441 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-htwpd\" (UniqueName: \"kubernetes.io/projected/06032cf2-951e-4323-8b39-19401b6f9c4f-kube-api-access-htwpd\") pod \"klusterlet-addon-workmgr-5b86c47c46-vjgpv\" (UID: \"06032cf2-951e-4323-8b39-19401b6f9c4f\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5b86c47c46-vjgpv" Jul 19 14:58:41.508589 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:58:41.508489 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sbzht\" (UniqueName: \"kubernetes.io/projected/3a074892-b609-4143-86f3-9c3102f66c36-kube-api-access-sbzht\") pod \"cluster-proxy-proxy-agent-6b775f4854-5blzr\" (UID: \"3a074892-b609-4143-86f3-9c3102f66c36\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b775f4854-5blzr" Jul 19 14:58:41.508589 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:58:41.508516 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/1e4e44f7-d447-4d70-bf92-656f49c76bb0-bound-sa-token\") pod \"image-registry-6f485ff8ff-jrz6w\" (UID: \"1e4e44f7-d447-4d70-bf92-656f49c76bb0\") " pod="openshift-image-registry/image-registry-6f485ff8ff-jrz6w" Jul 19 14:58:41.508751 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:58:41.508731 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/3a074892-b609-4143-86f3-9c3102f66c36-additional-service-ca\") pod \"cluster-proxy-proxy-agent-6b775f4854-5blzr\" (UID: \"3a074892-b609-4143-86f3-9c3102f66c36\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b775f4854-5blzr" Jul 19 14:58:41.509548 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:58:41.509052 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/06032cf2-951e-4323-8b39-19401b6f9c4f-tmp\") pod \"klusterlet-addon-workmgr-5b86c47c46-vjgpv\" (UID: \"06032cf2-951e-4323-8b39-19401b6f9c4f\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5b86c47c46-vjgpv" Jul 19 14:58:41.510983 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:58:41.510955 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/3a074892-b609-4143-86f3-9c3102f66c36-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-6b775f4854-5blzr\" (UID: \"3a074892-b609-4143-86f3-9c3102f66c36\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b775f4854-5blzr" Jul 19 14:58:41.511426 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:58:41.511401 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca\" (UniqueName: \"kubernetes.io/secret/3a074892-b609-4143-86f3-9c3102f66c36-ca\") pod \"cluster-proxy-proxy-agent-6b775f4854-5blzr\" (UID: \"3a074892-b609-4143-86f3-9c3102f66c36\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b775f4854-5blzr" Jul 19 14:58:41.511520 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:58:41.511494 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/3a074892-b609-4143-86f3-9c3102f66c36-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-6b775f4854-5blzr\" (UID: \"3a074892-b609-4143-86f3-9c3102f66c36\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b775f4854-5blzr" Jul 19 14:58:41.511903 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:58:41.511882 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/06032cf2-951e-4323-8b39-19401b6f9c4f-klusterlet-config\") pod \"klusterlet-addon-workmgr-5b86c47c46-vjgpv\" (UID: \"06032cf2-951e-4323-8b39-19401b6f9c4f\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5b86c47c46-vjgpv" Jul 19 14:58:41.511986 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:58:41.511965 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub\" (UniqueName: \"kubernetes.io/secret/3a074892-b609-4143-86f3-9c3102f66c36-hub\") pod \"cluster-proxy-proxy-agent-6b775f4854-5blzr\" (UID: \"3a074892-b609-4143-86f3-9c3102f66c36\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b775f4854-5blzr" Jul 19 14:58:41.535184 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:58:41.535148 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-htwpd\" (UniqueName: \"kubernetes.io/projected/06032cf2-951e-4323-8b39-19401b6f9c4f-kube-api-access-htwpd\") pod \"klusterlet-addon-workmgr-5b86c47c46-vjgpv\" (UID: \"06032cf2-951e-4323-8b39-19401b6f9c4f\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5b86c47c46-vjgpv" Jul 19 14:58:41.535770 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:58:41.535749 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sbzht\" (UniqueName: \"kubernetes.io/projected/3a074892-b609-4143-86f3-9c3102f66c36-kube-api-access-sbzht\") pod \"cluster-proxy-proxy-agent-6b775f4854-5blzr\" (UID: \"3a074892-b609-4143-86f3-9c3102f66c36\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b775f4854-5blzr" Jul 19 14:58:41.609368 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:58:41.609303 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/1e4e44f7-d447-4d70-bf92-656f49c76bb0-registry-certificates\") pod \"image-registry-6f485ff8ff-jrz6w\" (UID: \"1e4e44f7-d447-4d70-bf92-656f49c76bb0\") " pod="openshift-image-registry/image-registry-6f485ff8ff-jrz6w" Jul 19 14:58:41.609368 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:58:41.609362 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/1e4e44f7-d447-4d70-bf92-656f49c76bb0-trusted-ca\") pod \"image-registry-6f485ff8ff-jrz6w\" (UID: \"1e4e44f7-d447-4d70-bf92-656f49c76bb0\") " pod="openshift-image-registry/image-registry-6f485ff8ff-jrz6w" Jul 19 14:58:41.609541 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:58:41.609390 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/1e4e44f7-d447-4d70-bf92-656f49c76bb0-ca-trust-extracted\") pod \"image-registry-6f485ff8ff-jrz6w\" (UID: \"1e4e44f7-d447-4d70-bf92-656f49c76bb0\") " pod="openshift-image-registry/image-registry-6f485ff8ff-jrz6w" Jul 19 14:58:41.609541 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:58:41.609443 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/1e4e44f7-d447-4d70-bf92-656f49c76bb0-bound-sa-token\") pod \"image-registry-6f485ff8ff-jrz6w\" (UID: \"1e4e44f7-d447-4d70-bf92-656f49c76bb0\") " pod="openshift-image-registry/image-registry-6f485ff8ff-jrz6w" Jul 19 14:58:41.609541 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:58:41.609502 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/1e4e44f7-d447-4d70-bf92-656f49c76bb0-image-registry-private-configuration\") pod \"image-registry-6f485ff8ff-jrz6w\" (UID: \"1e4e44f7-d447-4d70-bf92-656f49c76bb0\") " pod="openshift-image-registry/image-registry-6f485ff8ff-jrz6w" Jul 19 14:58:41.609541 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:58:41.609536 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/1e4e44f7-d447-4d70-bf92-656f49c76bb0-installation-pull-secrets\") pod \"image-registry-6f485ff8ff-jrz6w\" (UID: \"1e4e44f7-d447-4d70-bf92-656f49c76bb0\") " pod="openshift-image-registry/image-registry-6f485ff8ff-jrz6w" Jul 19 14:58:41.609706 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:58:41.609570 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2jzpq\" (UniqueName: \"kubernetes.io/projected/1e4e44f7-d447-4d70-bf92-656f49c76bb0-kube-api-access-2jzpq\") pod \"image-registry-6f485ff8ff-jrz6w\" (UID: \"1e4e44f7-d447-4d70-bf92-656f49c76bb0\") " pod="openshift-image-registry/image-registry-6f485ff8ff-jrz6w" Jul 19 14:58:41.609706 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:58:41.609596 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/1e4e44f7-d447-4d70-bf92-656f49c76bb0-registry-tls\") pod \"image-registry-6f485ff8ff-jrz6w\" (UID: \"1e4e44f7-d447-4d70-bf92-656f49c76bb0\") " pod="openshift-image-registry/image-registry-6f485ff8ff-jrz6w" Jul 19 14:58:41.609785 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:58:41.609768 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/1e4e44f7-d447-4d70-bf92-656f49c76bb0-ca-trust-extracted\") pod \"image-registry-6f485ff8ff-jrz6w\" (UID: \"1e4e44f7-d447-4d70-bf92-656f49c76bb0\") " pod="openshift-image-registry/image-registry-6f485ff8ff-jrz6w" Jul 19 14:58:41.610247 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:58:41.610217 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/1e4e44f7-d447-4d70-bf92-656f49c76bb0-registry-certificates\") pod \"image-registry-6f485ff8ff-jrz6w\" (UID: \"1e4e44f7-d447-4d70-bf92-656f49c76bb0\") " pod="openshift-image-registry/image-registry-6f485ff8ff-jrz6w" Jul 19 14:58:41.610518 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:58:41.610350 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/1e4e44f7-d447-4d70-bf92-656f49c76bb0-trusted-ca\") pod \"image-registry-6f485ff8ff-jrz6w\" (UID: \"1e4e44f7-d447-4d70-bf92-656f49c76bb0\") " pod="openshift-image-registry/image-registry-6f485ff8ff-jrz6w" Jul 19 14:58:41.612602 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:58:41.612484 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/1e4e44f7-d447-4d70-bf92-656f49c76bb0-image-registry-private-configuration\") pod \"image-registry-6f485ff8ff-jrz6w\" (UID: \"1e4e44f7-d447-4d70-bf92-656f49c76bb0\") " pod="openshift-image-registry/image-registry-6f485ff8ff-jrz6w" Jul 19 14:58:41.612709 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:58:41.612647 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/1e4e44f7-d447-4d70-bf92-656f49c76bb0-registry-tls\") pod \"image-registry-6f485ff8ff-jrz6w\" (UID: \"1e4e44f7-d447-4d70-bf92-656f49c76bb0\") " pod="openshift-image-registry/image-registry-6f485ff8ff-jrz6w" Jul 19 14:58:41.612811 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:58:41.612791 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/1e4e44f7-d447-4d70-bf92-656f49c76bb0-installation-pull-secrets\") pod \"image-registry-6f485ff8ff-jrz6w\" (UID: \"1e4e44f7-d447-4d70-bf92-656f49c76bb0\") " pod="openshift-image-registry/image-registry-6f485ff8ff-jrz6w" Jul 19 14:58:41.622166 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:58:41.622144 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/1e4e44f7-d447-4d70-bf92-656f49c76bb0-bound-sa-token\") pod \"image-registry-6f485ff8ff-jrz6w\" (UID: \"1e4e44f7-d447-4d70-bf92-656f49c76bb0\") " pod="openshift-image-registry/image-registry-6f485ff8ff-jrz6w" Jul 19 14:58:41.622285 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:58:41.622265 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2jzpq\" (UniqueName: \"kubernetes.io/projected/1e4e44f7-d447-4d70-bf92-656f49c76bb0-kube-api-access-2jzpq\") pod \"image-registry-6f485ff8ff-jrz6w\" (UID: \"1e4e44f7-d447-4d70-bf92-656f49c76bb0\") " pod="openshift-image-registry/image-registry-6f485ff8ff-jrz6w" Jul 19 14:58:41.651659 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:58:41.651633 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5b86c47c46-vjgpv" Jul 19 14:58:41.676442 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:58:41.676409 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b775f4854-5blzr" Jul 19 14:58:41.752259 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:58:41.752076 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6f485ff8ff-jrz6w" Jul 19 14:58:41.813941 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:58:41.813893 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-5b86c47c46-vjgpv"] Jul 19 14:58:41.817421 ip-10-0-138-220 kubenswrapper[2558]: W0719 14:58:41.817393 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod06032cf2_951e_4323_8b39_19401b6f9c4f.slice/crio-71991e575bee17eef8deaa223933bdd937d1b9504de95f5586097fb5ac8343ed WatchSource:0}: Error finding container 71991e575bee17eef8deaa223933bdd937d1b9504de95f5586097fb5ac8343ed: Status 404 returned error can't find the container with id 71991e575bee17eef8deaa223933bdd937d1b9504de95f5586097fb5ac8343ed Jul 19 14:58:41.844356 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:58:41.844292 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b775f4854-5blzr"] Jul 19 14:58:41.847239 ip-10-0-138-220 kubenswrapper[2558]: W0719 14:58:41.847209 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3a074892_b609_4143_86f3_9c3102f66c36.slice/crio-eaeae437ac487c6493a1dcfc3613dfcde98021253386a23e6f1385858d9fe693 WatchSource:0}: Error finding container eaeae437ac487c6493a1dcfc3613dfcde98021253386a23e6f1385858d9fe693: Status 404 returned error can't find the container with id eaeae437ac487c6493a1dcfc3613dfcde98021253386a23e6f1385858d9fe693 Jul 19 14:58:41.909518 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:58:41.909490 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-6f485ff8ff-jrz6w"] Jul 19 14:58:41.913204 ip-10-0-138-220 kubenswrapper[2558]: W0719 14:58:41.913167 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1e4e44f7_d447_4d70_bf92_656f49c76bb0.slice/crio-529aee74cbcc3debdda3ce9fa42d9d0b30c9b47324e6e3e173d2d1a58fd13def WatchSource:0}: Error finding container 529aee74cbcc3debdda3ce9fa42d9d0b30c9b47324e6e3e173d2d1a58fd13def: Status 404 returned error can't find the container with id 529aee74cbcc3debdda3ce9fa42d9d0b30c9b47324e6e3e173d2d1a58fd13def Jul 19 14:58:41.967163 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:58:41.967133 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-hp977" event={"ID":"7fec0b2b-b831-414c-9a64-bf1b67c04fdd","Type":"ContainerStarted","Data":"a1027a74efbfa236503b91e31152a4c34d1f8bba1de04c81b75236274a64d7ae"} Jul 19 14:58:41.968552 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:58:41.968527 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6f485ff8ff-jrz6w" event={"ID":"1e4e44f7-d447-4d70-bf92-656f49c76bb0","Type":"ContainerStarted","Data":"529aee74cbcc3debdda3ce9fa42d9d0b30c9b47324e6e3e173d2d1a58fd13def"} Jul 19 14:58:41.969861 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:58:41.969812 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5b86c47c46-vjgpv" event={"ID":"06032cf2-951e-4323-8b39-19401b6f9c4f","Type":"ContainerStarted","Data":"71991e575bee17eef8deaa223933bdd937d1b9504de95f5586097fb5ac8343ed"} Jul 19 14:58:41.971108 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:58:41.971083 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b775f4854-5blzr" event={"ID":"3a074892-b609-4143-86f3-9c3102f66c36","Type":"ContainerStarted","Data":"eaeae437ac487c6493a1dcfc3613dfcde98021253386a23e6f1385858d9fe693"} Jul 19 14:58:42.939615 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:58:42.938741 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-kxhq6" Jul 19 14:58:42.980559 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:58:42.980518 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6f485ff8ff-jrz6w" event={"ID":"1e4e44f7-d447-4d70-bf92-656f49c76bb0","Type":"ContainerStarted","Data":"784898bb763a16d108b7922ce0cf271e73e21e944f98ffadd39e416bc996a6af"} Jul 19 14:58:42.981277 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:58:42.981253 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-6f485ff8ff-jrz6w" Jul 19 14:58:43.000567 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:58:43.000136 2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-6f485ff8ff-jrz6w" podStartSLOduration=2.000104281 podStartE2EDuration="2.000104281s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-19 14:58:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-19 14:58:42.99900349 +0000 UTC m=+63.920706196" watchObservedRunningTime="2026-07-19 14:58:43.000104281 +0000 UTC m=+63.921807010" Jul 19 14:58:44.334137 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:58:44.334036 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c9a1c57d-99f6-49a1-bd75-d00fcffc86f1-metrics-certs\") pod \"network-metrics-daemon-g52v5\" (UID: \"c9a1c57d-99f6-49a1-bd75-d00fcffc86f1\") " pod="openshift-multus/network-metrics-daemon-g52v5" Jul 19 14:58:44.336438 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:58:44.336229 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 19 14:58:44.354865 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:58:44.354823 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c9a1c57d-99f6-49a1-bd75-d00fcffc86f1-metrics-certs\") pod \"network-metrics-daemon-g52v5\" (UID: \"c9a1c57d-99f6-49a1-bd75-d00fcffc86f1\") " pod="openshift-multus/network-metrics-daemon-g52v5" Jul 19 14:58:44.435289 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:58:44.435259 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5994a848-5c3c-4d6c-898c-bb659e1a41b2-original-pull-secret\") pod \"global-pull-secret-syncer-kj72j\" (UID: \"5994a848-5c3c-4d6c-898c-bb659e1a41b2\") " pod="kube-system/global-pull-secret-syncer-kj72j" Jul 19 14:58:44.437671 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:58:44.437642 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 19 14:58:44.449198 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:58:44.449173 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5994a848-5c3c-4d6c-898c-bb659e1a41b2-original-pull-secret\") pod \"global-pull-secret-syncer-kj72j\" (UID: \"5994a848-5c3c-4d6c-898c-bb659e1a41b2\") " pod="kube-system/global-pull-secret-syncer-kj72j" Jul 19 14:58:44.535637 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:58:44.535607 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dnc4v\" (UniqueName: \"kubernetes.io/projected/11a5463d-a4e8-47d2-8791-5a10ea52cfc2-kube-api-access-dnc4v\") pod \"network-check-target-xpl4t\" (UID: \"11a5463d-a4e8-47d2-8791-5a10ea52cfc2\") " pod="openshift-network-diagnostics/network-check-target-xpl4t" Jul 19 14:58:44.538040 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:58:44.537858 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 19 14:58:44.548470 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:58:44.548446 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 19 14:58:44.561344 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:58:44.561295 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dnc4v\" (UniqueName: \"kubernetes.io/projected/11a5463d-a4e8-47d2-8791-5a10ea52cfc2-kube-api-access-dnc4v\") pod \"network-check-target-xpl4t\" (UID: \"11a5463d-a4e8-47d2-8791-5a10ea52cfc2\") " pod="openshift-network-diagnostics/network-check-target-xpl4t" Jul 19 14:58:44.572994 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:58:44.572965 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-kj72j" Jul 19 14:58:44.582365 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:58:44.582167 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-hc8w7\"" Jul 19 14:58:44.587729 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:58:44.587709 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-v7fqw\"" Jul 19 14:58:44.590318 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:58:44.590300 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-g52v5" Jul 19 14:58:44.596739 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:58:44.596720 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xpl4t" Jul 19 14:58:49.368950 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:58:49.368920 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-g52v5"] Jul 19 14:58:49.371950 ip-10-0-138-220 kubenswrapper[2558]: W0719 14:58:49.371927 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc9a1c57d_99f6_49a1_bd75_d00fcffc86f1.slice/crio-ec83e860e3b2e9f6a011ed7fc8208c7721fcbb4579c9a5cfc06108695715a300 WatchSource:0}: Error finding container ec83e860e3b2e9f6a011ed7fc8208c7721fcbb4579c9a5cfc06108695715a300: Status 404 returned error can't find the container with id ec83e860e3b2e9f6a011ed7fc8208c7721fcbb4579c9a5cfc06108695715a300 Jul 19 14:58:49.387734 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:58:49.387606 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-kj72j"] Jul 19 14:58:49.390689 ip-10-0-138-220 kubenswrapper[2558]: W0719 14:58:49.390652 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5994a848_5c3c_4d6c_898c_bb659e1a41b2.slice/crio-68b3ca144290ee48c145530881d247c807c4bfa392b79eee796f3fde1dbafad7 WatchSource:0}: Error finding container 68b3ca144290ee48c145530881d247c807c4bfa392b79eee796f3fde1dbafad7: Status 404 returned error can't find the container with id 68b3ca144290ee48c145530881d247c807c4bfa392b79eee796f3fde1dbafad7 Jul 19 14:58:49.404879 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:58:49.403902 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-xpl4t"] Jul 19 14:58:49.407992 ip-10-0-138-220 kubenswrapper[2558]: W0719 14:58:49.407962 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod11a5463d_a4e8_47d2_8791_5a10ea52cfc2.slice/crio-5983610c6c49e5606b1eea4f5d3d8a482d518783990906f0fc7b915e3ab19a70 WatchSource:0}: Error finding container 5983610c6c49e5606b1eea4f5d3d8a482d518783990906f0fc7b915e3ab19a70: Status 404 returned error can't find the container with id 5983610c6c49e5606b1eea4f5d3d8a482d518783990906f0fc7b915e3ab19a70 Jul 19 14:58:50.014300 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:58:50.013236 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5b86c47c46-vjgpv" event={"ID":"06032cf2-951e-4323-8b39-19401b6f9c4f","Type":"ContainerStarted","Data":"7ecfe39016a09e4f4aceedda96946ead47877826506c0d871939ec9a7ca2df22"} Jul 19 14:58:50.014300 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:58:50.014225 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5b86c47c46-vjgpv" Jul 19 14:58:50.016069 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:58:50.016039 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-xpl4t" event={"ID":"11a5463d-a4e8-47d2-8791-5a10ea52cfc2","Type":"ContainerStarted","Data":"5983610c6c49e5606b1eea4f5d3d8a482d518783990906f0fc7b915e3ab19a70"} Jul 19 14:58:50.016190 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:58:50.016162 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5b86c47c46-vjgpv" Jul 19 14:58:50.018425 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:58:50.018364 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-kj72j" event={"ID":"5994a848-5c3c-4d6c-898c-bb659e1a41b2","Type":"ContainerStarted","Data":"68b3ca144290ee48c145530881d247c807c4bfa392b79eee796f3fde1dbafad7"} Jul 19 14:58:50.021604 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:58:50.021204 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b775f4854-5blzr" event={"ID":"3a074892-b609-4143-86f3-9c3102f66c36","Type":"ContainerStarted","Data":"c8baf4b2235fdae74d459b677c7bd3e034718861177692acff8fc67c55a0e911"} Jul 19 14:58:50.021604 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:58:50.021237 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b775f4854-5blzr" event={"ID":"3a074892-b609-4143-86f3-9c3102f66c36","Type":"ContainerStarted","Data":"c8e2a88a5495ea14f73a82fc4c9675f6fe29f7a1977ce7f9b0b867c6ddfbb7a9"} Jul 19 14:58:50.021604 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:58:50.021255 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b775f4854-5blzr" event={"ID":"3a074892-b609-4143-86f3-9c3102f66c36","Type":"ContainerStarted","Data":"afbd4285a6f3feada09c393dac52bccc12fd3e05d2f8384a326f29f29752fa74"} Jul 19 14:58:50.023384 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:58:50.023356 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-g52v5" event={"ID":"c9a1c57d-99f6-49a1-bd75-d00fcffc86f1","Type":"ContainerStarted","Data":"ec83e860e3b2e9f6a011ed7fc8208c7721fcbb4579c9a5cfc06108695715a300"} Jul 19 14:58:50.026310 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:58:50.026286 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-hp977" event={"ID":"7fec0b2b-b831-414c-9a64-bf1b67c04fdd","Type":"ContainerStarted","Data":"c7002ec24dabe28ec65700cb8e66274267012ded4d28e366afb61a3e1081b055"} Jul 19 14:58:50.028332 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:58:50.028287 2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5b86c47c46-vjgpv" podStartSLOduration=1.635823982 podStartE2EDuration="9.028262549s" totalImagesPullingTime="7.392438567s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-19 14:58:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-19 14:58:50.027074484 +0000 UTC m=+70.948777198" watchObservedRunningTime="2026-07-19 14:58:50.028262549 +0000 UTC m=+70.949965260" Jul 19 14:58:50.057865 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:58:50.057820 2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-hp977" podStartSLOduration=17.31701119 podStartE2EDuration="34.057803007s" totalImagesPullingTime="16.740791817s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-19 14:58:16 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-19 14:58:50.057644648 +0000 UTC m=+70.979347370" watchObservedRunningTime="2026-07-19 14:58:50.057803007 +0000 UTC m=+70.979505722" Jul 19 14:58:50.073685 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:58:50.073640 2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b775f4854-5blzr" podStartSLOduration=1.690892234 podStartE2EDuration="9.073622799s" totalImagesPullingTime="7.382730565s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-19 14:58:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-19 14:58:50.07190517 +0000 UTC m=+70.993607885" watchObservedRunningTime="2026-07-19 14:58:50.073622799 +0000 UTC m=+70.995325516" Jul 19 14:58:51.037283 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:58:51.037107 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-g52v5" event={"ID":"c9a1c57d-99f6-49a1-bd75-d00fcffc86f1","Type":"ContainerStarted","Data":"a1dc05a5e4a90b23d7c58ae8a117652da6a184c735f570cb3d368af1b8d36870"} Jul 19 14:58:51.037283 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:58:51.037241 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-g52v5" event={"ID":"c9a1c57d-99f6-49a1-bd75-d00fcffc86f1","Type":"ContainerStarted","Data":"4f7f323a6021d57f8e192fbdd17ead2108b220517e3e92e0fadd297481741caa"} Jul 19 14:58:51.054111 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:58:51.054062 2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-g52v5" podStartSLOduration=70.954857457 podStartE2EDuration="1m12.054041984s" totalImagesPullingTime="1.099184527s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-19 14:57:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-19 14:58:51.052270372 +0000 UTC m=+71.973973086" watchObservedRunningTime="2026-07-19 14:58:51.054041984 +0000 UTC m=+71.975744698" Jul 19 14:58:51.369878 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:58:51.369807 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-575ddf7c75-4blsl" Jul 19 14:58:55.053439 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:58:55.053406 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-xpl4t" event={"ID":"11a5463d-a4e8-47d2-8791-5a10ea52cfc2","Type":"ContainerStarted","Data":"5320ee71991c48f91b804dedfb717c656033c6ef117148b0295275febb92b52c"} Jul 19 14:58:55.053838 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:58:55.053479 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-xpl4t" Jul 19 14:58:55.055002 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:58:55.054980 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-kj72j" event={"ID":"5994a848-5c3c-4d6c-898c-bb659e1a41b2","Type":"ContainerStarted","Data":"4a9968b2d35dfcf823271983127747ba20fa6ffdd815d4d690ff717ed2d93d98"} Jul 19 14:58:55.067810 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:58:55.067748 2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-xpl4t" podStartSLOduration=71.458062252 podStartE2EDuration="1m16.067732905s" totalImagesPullingTime="4.609670653s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-19 14:57:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-19 14:58:55.067696539 +0000 UTC m=+75.989399266" watchObservedRunningTime="2026-07-19 14:58:55.067732905 +0000 UTC m=+75.989435620" Jul 19 14:58:55.081946 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:58:55.081910 2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-kj72j" podStartSLOduration=71.450213372 podStartE2EDuration="1m16.081899158s" totalImagesPullingTime="4.631685786s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-19 14:57:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-19 14:58:55.080832206 +0000 UTC m=+76.002534921" watchObservedRunningTime="2026-07-19 14:58:55.081899158 +0000 UTC m=+76.003601872" Jul 19 14:59:01.981620 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:59:01.981499 2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-pwvsv"] Jul 19 14:59:01.985875 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:59:01.985853 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-pwvsv" Jul 19 14:59:01.988648 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:59:01.988427 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Jul 19 14:59:01.988648 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:59:01.988476 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Jul 19 14:59:01.988648 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:59:01.988427 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Jul 19 14:59:01.988898 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:59:01.988690 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Jul 19 14:59:01.988898 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:59:01.988876 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Jul 19 14:59:01.989003 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:59:01.988956 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Jul 19 14:59:01.989269 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:59:01.989119 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-tcft9\"" Jul 19 14:59:02.080746 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:59:02.080719 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zvrfz\" (UniqueName: \"kubernetes.io/projected/8281d5f9-51cc-46b4-a611-06ae3c073d41-kube-api-access-zvrfz\") pod \"node-exporter-pwvsv\" (UID: \"8281d5f9-51cc-46b4-a611-06ae3c073d41\") " pod="openshift-monitoring/node-exporter-pwvsv" Jul 19 14:59:02.080882 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:59:02.080756 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/8281d5f9-51cc-46b4-a611-06ae3c073d41-node-exporter-tls\") pod \"node-exporter-pwvsv\" (UID: \"8281d5f9-51cc-46b4-a611-06ae3c073d41\") " pod="openshift-monitoring/node-exporter-pwvsv" Jul 19 14:59:02.080882 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:59:02.080794 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/8281d5f9-51cc-46b4-a611-06ae3c073d41-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-pwvsv\" (UID: \"8281d5f9-51cc-46b4-a611-06ae3c073d41\") " pod="openshift-monitoring/node-exporter-pwvsv" Jul 19 14:59:02.080882 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:59:02.080843 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/8281d5f9-51cc-46b4-a611-06ae3c073d41-node-exporter-textfile\") pod \"node-exporter-pwvsv\" (UID: \"8281d5f9-51cc-46b4-a611-06ae3c073d41\") " pod="openshift-monitoring/node-exporter-pwvsv" Jul 19 14:59:02.081053 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:59:02.080906 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/8281d5f9-51cc-46b4-a611-06ae3c073d41-node-exporter-wtmp\") pod \"node-exporter-pwvsv\" (UID: \"8281d5f9-51cc-46b4-a611-06ae3c073d41\") " pod="openshift-monitoring/node-exporter-pwvsv" Jul 19 14:59:02.081053 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:59:02.080949 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/8281d5f9-51cc-46b4-a611-06ae3c073d41-root\") pod \"node-exporter-pwvsv\" (UID: \"8281d5f9-51cc-46b4-a611-06ae3c073d41\") " pod="openshift-monitoring/node-exporter-pwvsv" Jul 19 14:59:02.081053 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:59:02.080974 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/8281d5f9-51cc-46b4-a611-06ae3c073d41-node-exporter-accelerators-collector-config\") pod \"node-exporter-pwvsv\" (UID: \"8281d5f9-51cc-46b4-a611-06ae3c073d41\") " pod="openshift-monitoring/node-exporter-pwvsv" Jul 19 14:59:02.081053 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:59:02.081012 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8281d5f9-51cc-46b4-a611-06ae3c073d41-metrics-client-ca\") pod \"node-exporter-pwvsv\" (UID: \"8281d5f9-51cc-46b4-a611-06ae3c073d41\") " pod="openshift-monitoring/node-exporter-pwvsv" Jul 19 14:59:02.081260 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:59:02.081056 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8281d5f9-51cc-46b4-a611-06ae3c073d41-sys\") pod \"node-exporter-pwvsv\" (UID: \"8281d5f9-51cc-46b4-a611-06ae3c073d41\") " pod="openshift-monitoring/node-exporter-pwvsv" Jul 19 14:59:02.182192 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:59:02.182161 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/8281d5f9-51cc-46b4-a611-06ae3c073d41-node-exporter-textfile\") pod \"node-exporter-pwvsv\" (UID: \"8281d5f9-51cc-46b4-a611-06ae3c073d41\") " pod="openshift-monitoring/node-exporter-pwvsv" Jul 19 14:59:02.182345 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:59:02.182202 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/8281d5f9-51cc-46b4-a611-06ae3c073d41-node-exporter-wtmp\") pod \"node-exporter-pwvsv\" (UID: \"8281d5f9-51cc-46b4-a611-06ae3c073d41\") " pod="openshift-monitoring/node-exporter-pwvsv" Jul 19 14:59:02.182345 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:59:02.182232 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/8281d5f9-51cc-46b4-a611-06ae3c073d41-root\") pod \"node-exporter-pwvsv\" (UID: \"8281d5f9-51cc-46b4-a611-06ae3c073d41\") " pod="openshift-monitoring/node-exporter-pwvsv" Jul 19 14:59:02.182345 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:59:02.182293 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/8281d5f9-51cc-46b4-a611-06ae3c073d41-root\") pod \"node-exporter-pwvsv\" (UID: \"8281d5f9-51cc-46b4-a611-06ae3c073d41\") " pod="openshift-monitoring/node-exporter-pwvsv" Jul 19 14:59:02.182495 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:59:02.182346 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/8281d5f9-51cc-46b4-a611-06ae3c073d41-node-exporter-wtmp\") pod \"node-exporter-pwvsv\" (UID: \"8281d5f9-51cc-46b4-a611-06ae3c073d41\") " pod="openshift-monitoring/node-exporter-pwvsv" Jul 19 14:59:02.182495 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:59:02.182346 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/8281d5f9-51cc-46b4-a611-06ae3c073d41-node-exporter-accelerators-collector-config\") pod \"node-exporter-pwvsv\" (UID: \"8281d5f9-51cc-46b4-a611-06ae3c073d41\") " pod="openshift-monitoring/node-exporter-pwvsv" Jul 19 14:59:02.182495 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:59:02.182413 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8281d5f9-51cc-46b4-a611-06ae3c073d41-metrics-client-ca\") pod \"node-exporter-pwvsv\" (UID: \"8281d5f9-51cc-46b4-a611-06ae3c073d41\") " pod="openshift-monitoring/node-exporter-pwvsv" Jul 19 14:59:02.182495 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:59:02.182446 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8281d5f9-51cc-46b4-a611-06ae3c073d41-sys\") pod \"node-exporter-pwvsv\" (UID: \"8281d5f9-51cc-46b4-a611-06ae3c073d41\") " pod="openshift-monitoring/node-exporter-pwvsv" Jul 19 14:59:02.182495 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:59:02.182475 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zvrfz\" (UniqueName: \"kubernetes.io/projected/8281d5f9-51cc-46b4-a611-06ae3c073d41-kube-api-access-zvrfz\") pod \"node-exporter-pwvsv\" (UID: \"8281d5f9-51cc-46b4-a611-06ae3c073d41\") " pod="openshift-monitoring/node-exporter-pwvsv" Jul 19 14:59:02.182742 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:59:02.182501 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/8281d5f9-51cc-46b4-a611-06ae3c073d41-node-exporter-tls\") pod \"node-exporter-pwvsv\" (UID: \"8281d5f9-51cc-46b4-a611-06ae3c073d41\") " pod="openshift-monitoring/node-exporter-pwvsv" Jul 19 14:59:02.182742 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:59:02.182530 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/8281d5f9-51cc-46b4-a611-06ae3c073d41-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-pwvsv\" (UID: \"8281d5f9-51cc-46b4-a611-06ae3c073d41\") " pod="openshift-monitoring/node-exporter-pwvsv" Jul 19 14:59:02.182742 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:59:02.182546 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/8281d5f9-51cc-46b4-a611-06ae3c073d41-node-exporter-textfile\") pod \"node-exporter-pwvsv\" (UID: \"8281d5f9-51cc-46b4-a611-06ae3c073d41\") " pod="openshift-monitoring/node-exporter-pwvsv" Jul 19 14:59:02.182742 ip-10-0-138-220 kubenswrapper[2558]: E0719 14:59:02.182650 2558 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found Jul 19 14:59:02.182742 ip-10-0-138-220 kubenswrapper[2558]: E0719 14:59:02.182716 2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8281d5f9-51cc-46b4-a611-06ae3c073d41-node-exporter-tls podName:8281d5f9-51cc-46b4-a611-06ae3c073d41 nodeName:}" failed. No retries permitted until 2026-07-19 14:59:02.682696777 +0000 UTC m=+83.604399469 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/8281d5f9-51cc-46b4-a611-06ae3c073d41-node-exporter-tls") pod "node-exporter-pwvsv" (UID: "8281d5f9-51cc-46b4-a611-06ae3c073d41") : secret "node-exporter-tls" not found Jul 19 14:59:02.182989 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:59:02.182837 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8281d5f9-51cc-46b4-a611-06ae3c073d41-sys\") pod \"node-exporter-pwvsv\" (UID: \"8281d5f9-51cc-46b4-a611-06ae3c073d41\") " pod="openshift-monitoring/node-exporter-pwvsv" Jul 19 14:59:02.182989 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:59:02.182882 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/8281d5f9-51cc-46b4-a611-06ae3c073d41-node-exporter-accelerators-collector-config\") pod \"node-exporter-pwvsv\" (UID: \"8281d5f9-51cc-46b4-a611-06ae3c073d41\") " pod="openshift-monitoring/node-exporter-pwvsv" Jul 19 14:59:02.182989 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:59:02.182933 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8281d5f9-51cc-46b4-a611-06ae3c073d41-metrics-client-ca\") pod \"node-exporter-pwvsv\" (UID: \"8281d5f9-51cc-46b4-a611-06ae3c073d41\") " pod="openshift-monitoring/node-exporter-pwvsv" Jul 19 14:59:02.185091 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:59:02.185063 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/8281d5f9-51cc-46b4-a611-06ae3c073d41-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-pwvsv\" (UID: \"8281d5f9-51cc-46b4-a611-06ae3c073d41\") " pod="openshift-monitoring/node-exporter-pwvsv" Jul 19 14:59:02.192443 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:59:02.192420 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zvrfz\" (UniqueName: \"kubernetes.io/projected/8281d5f9-51cc-46b4-a611-06ae3c073d41-kube-api-access-zvrfz\") pod \"node-exporter-pwvsv\" (UID: \"8281d5f9-51cc-46b4-a611-06ae3c073d41\") " pod="openshift-monitoring/node-exporter-pwvsv" Jul 19 14:59:02.686847 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:59:02.686815 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/8281d5f9-51cc-46b4-a611-06ae3c073d41-node-exporter-tls\") pod \"node-exporter-pwvsv\" (UID: \"8281d5f9-51cc-46b4-a611-06ae3c073d41\") " pod="openshift-monitoring/node-exporter-pwvsv" Jul 19 14:59:02.689266 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:59:02.689244 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/8281d5f9-51cc-46b4-a611-06ae3c073d41-node-exporter-tls\") pod \"node-exporter-pwvsv\" (UID: \"8281d5f9-51cc-46b4-a611-06ae3c073d41\") " pod="openshift-monitoring/node-exporter-pwvsv" Jul 19 14:59:02.908506 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:59:02.908473 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-pwvsv" Jul 19 14:59:02.918987 ip-10-0-138-220 kubenswrapper[2558]: W0719 14:59:02.918960 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8281d5f9_51cc_46b4_a611_06ae3c073d41.slice/crio-61a4e2db7184c07238ed9b1e7bd7fd74a82310e5b72934d44788d3708173a7d1 WatchSource:0}: Error finding container 61a4e2db7184c07238ed9b1e7bd7fd74a82310e5b72934d44788d3708173a7d1: Status 404 returned error can't find the container with id 61a4e2db7184c07238ed9b1e7bd7fd74a82310e5b72934d44788d3708173a7d1 Jul 19 14:59:03.082587 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:59:03.082505 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-pwvsv" event={"ID":"8281d5f9-51cc-46b4-a611-06ae3c073d41","Type":"ContainerStarted","Data":"61a4e2db7184c07238ed9b1e7bd7fd74a82310e5b72934d44788d3708173a7d1"} Jul 19 14:59:04.087907 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:59:04.087873 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-pwvsv" event={"ID":"8281d5f9-51cc-46b4-a611-06ae3c073d41","Type":"ContainerStarted","Data":"58622067643a16cdaa1213799cfd4da901b96cf55227c6001432840a83cce781"} Jul 19 14:59:04.994509 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:59:04.994482 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-6f485ff8ff-jrz6w" Jul 19 14:59:05.093461 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:59:05.093429 2558 generic.go:356] "Generic (PLEG): container finished" podID="8281d5f9-51cc-46b4-a611-06ae3c073d41" containerID="58622067643a16cdaa1213799cfd4da901b96cf55227c6001432840a83cce781" exitCode=0 Jul 19 14:59:05.093852 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:59:05.093495 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-pwvsv" event={"ID":"8281d5f9-51cc-46b4-a611-06ae3c073d41","Type":"ContainerDied","Data":"58622067643a16cdaa1213799cfd4da901b96cf55227c6001432840a83cce781"} Jul 19 14:59:06.098927 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:59:06.098895 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-pwvsv" event={"ID":"8281d5f9-51cc-46b4-a611-06ae3c073d41","Type":"ContainerStarted","Data":"fd029d97f138779e5e396c5cf60e9e76f70def7ec8b3161126010f691030ef62"} Jul 19 14:59:06.099285 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:59:06.098945 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-pwvsv" event={"ID":"8281d5f9-51cc-46b4-a611-06ae3c073d41","Type":"ContainerStarted","Data":"3862fe556d712d82dafa381d57f1d22b31a8cf41e9e3e61382915f989ec6b8fa"} Jul 19 14:59:06.118714 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:59:06.118655 2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-pwvsv" podStartSLOduration=3.996085422 podStartE2EDuration="5.118637202s" totalImagesPullingTime="1.04917143s" totalInitContainerRuntime="73.38035ms" isStatefulPod=true podCreationTimestamp="2026-07-19 14:59:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-19 14:59:06.118604228 +0000 UTC m=+87.040306934" watchObservedRunningTime="2026-07-19 14:59:06.118637202 +0000 UTC m=+87.040339910" Jul 19 14:59:06.383862 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:59:06.383766 2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-575ddf7c75-4blsl" podUID="b50d3eba-b458-44a1-9949-0ee8c16e1e3c" containerName="registry" containerID="cri-o://0065fc694e0c2fcee12923fcbdd3a62ba01f09be55f5074b1c3da2495b991ca2" gracePeriod=30 Jul 19 14:59:06.695898 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:59:06.695869 2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-575ddf7c75-4blsl" Jul 19 14:59:06.718448 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:59:06.718421 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/b50d3eba-b458-44a1-9949-0ee8c16e1e3c-trusted-ca\") pod \"b50d3eba-b458-44a1-9949-0ee8c16e1e3c\" (UID: \"b50d3eba-b458-44a1-9949-0ee8c16e1e3c\") " Jul 19 14:59:06.718558 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:59:06.718469 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/b50d3eba-b458-44a1-9949-0ee8c16e1e3c-installation-pull-secrets\") pod \"b50d3eba-b458-44a1-9949-0ee8c16e1e3c\" (UID: \"b50d3eba-b458-44a1-9949-0ee8c16e1e3c\") " Jul 19 14:59:06.718558 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:59:06.718496 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b50d3eba-b458-44a1-9949-0ee8c16e1e3c-registry-tls\") pod \"b50d3eba-b458-44a1-9949-0ee8c16e1e3c\" (UID: \"b50d3eba-b458-44a1-9949-0ee8c16e1e3c\") " Jul 19 14:59:06.718558 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:59:06.718547 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/b50d3eba-b458-44a1-9949-0ee8c16e1e3c-ca-trust-extracted\") pod \"b50d3eba-b458-44a1-9949-0ee8c16e1e3c\" (UID: \"b50d3eba-b458-44a1-9949-0ee8c16e1e3c\") " Jul 19 14:59:06.718730 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:59:06.718590 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b50d3eba-b458-44a1-9949-0ee8c16e1e3c-bound-sa-token\") pod \"b50d3eba-b458-44a1-9949-0ee8c16e1e3c\" (UID: \"b50d3eba-b458-44a1-9949-0ee8c16e1e3c\") " Jul 19 14:59:06.718730 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:59:06.718619 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/b50d3eba-b458-44a1-9949-0ee8c16e1e3c-image-registry-private-configuration\") pod \"b50d3eba-b458-44a1-9949-0ee8c16e1e3c\" (UID: \"b50d3eba-b458-44a1-9949-0ee8c16e1e3c\") " Jul 19 14:59:06.718730 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:59:06.718650 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6nzqw\" (UniqueName: \"kubernetes.io/projected/b50d3eba-b458-44a1-9949-0ee8c16e1e3c-kube-api-access-6nzqw\") pod \"b50d3eba-b458-44a1-9949-0ee8c16e1e3c\" (UID: \"b50d3eba-b458-44a1-9949-0ee8c16e1e3c\") " Jul 19 14:59:06.718730 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:59:06.718674 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/b50d3eba-b458-44a1-9949-0ee8c16e1e3c-registry-certificates\") pod \"b50d3eba-b458-44a1-9949-0ee8c16e1e3c\" (UID: \"b50d3eba-b458-44a1-9949-0ee8c16e1e3c\") " Jul 19 14:59:06.718929 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:59:06.718849 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b50d3eba-b458-44a1-9949-0ee8c16e1e3c-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "b50d3eba-b458-44a1-9949-0ee8c16e1e3c" (UID: "b50d3eba-b458-44a1-9949-0ee8c16e1e3c"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 19 14:59:06.719235 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:59:06.719000 2558 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/b50d3eba-b458-44a1-9949-0ee8c16e1e3c-trusted-ca\") on node \"ip-10-0-138-220.ec2.internal\" DevicePath \"\"" Jul 19 14:59:06.719775 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:59:06.719743 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b50d3eba-b458-44a1-9949-0ee8c16e1e3c-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "b50d3eba-b458-44a1-9949-0ee8c16e1e3c" (UID: "b50d3eba-b458-44a1-9949-0ee8c16e1e3c"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 19 14:59:06.722495 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:59:06.722419 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b50d3eba-b458-44a1-9949-0ee8c16e1e3c-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "b50d3eba-b458-44a1-9949-0ee8c16e1e3c" (UID: "b50d3eba-b458-44a1-9949-0ee8c16e1e3c"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 19 14:59:06.722495 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:59:06.722464 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b50d3eba-b458-44a1-9949-0ee8c16e1e3c-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "b50d3eba-b458-44a1-9949-0ee8c16e1e3c" (UID: "b50d3eba-b458-44a1-9949-0ee8c16e1e3c"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 19 14:59:06.722691 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:59:06.722568 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b50d3eba-b458-44a1-9949-0ee8c16e1e3c-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "b50d3eba-b458-44a1-9949-0ee8c16e1e3c" (UID: "b50d3eba-b458-44a1-9949-0ee8c16e1e3c"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 19 14:59:06.722691 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:59:06.722618 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b50d3eba-b458-44a1-9949-0ee8c16e1e3c-kube-api-access-6nzqw" (OuterVolumeSpecName: "kube-api-access-6nzqw") pod "b50d3eba-b458-44a1-9949-0ee8c16e1e3c" (UID: "b50d3eba-b458-44a1-9949-0ee8c16e1e3c"). InnerVolumeSpecName "kube-api-access-6nzqw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 19 14:59:06.722691 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:59:06.722649 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b50d3eba-b458-44a1-9949-0ee8c16e1e3c-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "b50d3eba-b458-44a1-9949-0ee8c16e1e3c" (UID: "b50d3eba-b458-44a1-9949-0ee8c16e1e3c"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 19 14:59:06.730589 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:59:06.730564 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b50d3eba-b458-44a1-9949-0ee8c16e1e3c-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "b50d3eba-b458-44a1-9949-0ee8c16e1e3c" (UID: "b50d3eba-b458-44a1-9949-0ee8c16e1e3c"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 19 14:59:06.819672 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:59:06.819642 2558 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/b50d3eba-b458-44a1-9949-0ee8c16e1e3c-installation-pull-secrets\") on node \"ip-10-0-138-220.ec2.internal\" DevicePath \"\"" Jul 19 14:59:06.819672 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:59:06.819668 2558 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b50d3eba-b458-44a1-9949-0ee8c16e1e3c-registry-tls\") on node \"ip-10-0-138-220.ec2.internal\" DevicePath \"\"" Jul 19 14:59:06.819672 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:59:06.819677 2558 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/b50d3eba-b458-44a1-9949-0ee8c16e1e3c-ca-trust-extracted\") on node \"ip-10-0-138-220.ec2.internal\" DevicePath \"\"" Jul 19 14:59:06.819878 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:59:06.819687 2558 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b50d3eba-b458-44a1-9949-0ee8c16e1e3c-bound-sa-token\") on node \"ip-10-0-138-220.ec2.internal\" DevicePath \"\"" Jul 19 14:59:06.819878 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:59:06.819697 2558 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/b50d3eba-b458-44a1-9949-0ee8c16e1e3c-image-registry-private-configuration\") on node \"ip-10-0-138-220.ec2.internal\" DevicePath \"\"" Jul 19 14:59:06.819878 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:59:06.819705 2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6nzqw\" (UniqueName: \"kubernetes.io/projected/b50d3eba-b458-44a1-9949-0ee8c16e1e3c-kube-api-access-6nzqw\") on node \"ip-10-0-138-220.ec2.internal\" DevicePath \"\"" Jul 19 14:59:06.819878 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:59:06.819713 2558 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/b50d3eba-b458-44a1-9949-0ee8c16e1e3c-registry-certificates\") on node \"ip-10-0-138-220.ec2.internal\" DevicePath \"\"" Jul 19 14:59:07.103562 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:59:07.103490 2558 generic.go:356] "Generic (PLEG): container finished" podID="b50d3eba-b458-44a1-9949-0ee8c16e1e3c" containerID="0065fc694e0c2fcee12923fcbdd3a62ba01f09be55f5074b1c3da2495b991ca2" exitCode=0 Jul 19 14:59:07.103562 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:59:07.103537 2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-575ddf7c75-4blsl" Jul 19 14:59:07.104047 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:59:07.103598 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-575ddf7c75-4blsl" event={"ID":"b50d3eba-b458-44a1-9949-0ee8c16e1e3c","Type":"ContainerDied","Data":"0065fc694e0c2fcee12923fcbdd3a62ba01f09be55f5074b1c3da2495b991ca2"} Jul 19 14:59:07.104047 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:59:07.103671 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-575ddf7c75-4blsl" event={"ID":"b50d3eba-b458-44a1-9949-0ee8c16e1e3c","Type":"ContainerDied","Data":"140ddc9c3bb23c93627262006741503cee3e63ffd9e224f080c099f274fe8636"} Jul 19 14:59:07.104047 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:59:07.103691 2558 scope.go:117] "RemoveContainer" containerID="0065fc694e0c2fcee12923fcbdd3a62ba01f09be55f5074b1c3da2495b991ca2" Jul 19 14:59:07.112616 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:59:07.112599 2558 scope.go:117] "RemoveContainer" containerID="0065fc694e0c2fcee12923fcbdd3a62ba01f09be55f5074b1c3da2495b991ca2" Jul 19 14:59:07.112918 ip-10-0-138-220 kubenswrapper[2558]: E0719 14:59:07.112898 2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0065fc694e0c2fcee12923fcbdd3a62ba01f09be55f5074b1c3da2495b991ca2\": container with ID starting with 0065fc694e0c2fcee12923fcbdd3a62ba01f09be55f5074b1c3da2495b991ca2 not found: ID does not exist" containerID="0065fc694e0c2fcee12923fcbdd3a62ba01f09be55f5074b1c3da2495b991ca2" Jul 19 14:59:07.113003 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:59:07.112930 2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0065fc694e0c2fcee12923fcbdd3a62ba01f09be55f5074b1c3da2495b991ca2"} err="failed to get container status \"0065fc694e0c2fcee12923fcbdd3a62ba01f09be55f5074b1c3da2495b991ca2\": rpc error: code = NotFound desc = could not find container \"0065fc694e0c2fcee12923fcbdd3a62ba01f09be55f5074b1c3da2495b991ca2\": container with ID starting with 0065fc694e0c2fcee12923fcbdd3a62ba01f09be55f5074b1c3da2495b991ca2 not found: ID does not exist" Jul 19 14:59:07.125197 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:59:07.125177 2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-575ddf7c75-4blsl"] Jul 19 14:59:07.130751 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:59:07.130733 2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-575ddf7c75-4blsl"] Jul 19 14:59:07.563388 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:59:07.563359 2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b50d3eba-b458-44a1-9949-0ee8c16e1e3c" path="/var/lib/kubelet/pods/b50d3eba-b458-44a1-9949-0ee8c16e1e3c/volumes" Jul 19 14:59:22.634201 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:59:22.634173 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-7prf4_47e880d5-eff9-4f7b-9c47-b7af7f74dc41/serve-healthcheck-canary/0.log" Jul 19 14:59:26.060337 ip-10-0-138-220 kubenswrapper[2558]: I0719 14:59:26.060301 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-xpl4t" Jul 19 15:02:39.501688 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:02:39.501662 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-5wfzs_5b2d3a07-86b2-4ba2-8a71-c2d89ada79c6/ovn-acl-logging/0.log" Jul 19 15:02:39.505181 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:02:39.505162 2558 kubelet.go:1648] "Image garbage collection succeeded" Jul 19 15:02:39.508657 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:02:39.508641 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-5wfzs_5b2d3a07-86b2-4ba2-8a71-c2d89ada79c6/ovn-acl-logging/0.log" Jul 19 15:03:35.884306 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:03:35.884231 2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-7gqmr"] Jul 19 15:03:35.884697 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:03:35.884518 2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b50d3eba-b458-44a1-9949-0ee8c16e1e3c" containerName="registry" Jul 19 15:03:35.884697 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:03:35.884531 2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="b50d3eba-b458-44a1-9949-0ee8c16e1e3c" containerName="registry" Jul 19 15:03:35.884697 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:03:35.884584 2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="b50d3eba-b458-44a1-9949-0ee8c16e1e3c" containerName="registry" Jul 19 15:03:35.887199 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:03:35.887183 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-7gqmr" Jul 19 15:03:35.889281 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:03:35.889218 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\"" Jul 19 15:03:35.890060 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:03:35.890040 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\"" Jul 19 15:03:35.890155 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:03:35.890047 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"openshift-lws-operator-dockercfg-srqvc\"" Jul 19 15:03:35.894976 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:03:35.894953 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-7gqmr"] Jul 19 15:03:35.930366 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:03:35.930335 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9fc231d4-04b0-4462-b869-067b8a369a4f-tmp\") pod \"openshift-lws-operator-fd8ccff4c-7gqmr\" (UID: \"9fc231d4-04b0-4462-b869-067b8a369a4f\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-7gqmr" Jul 19 15:03:35.930476 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:03:35.930385 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2s8fx\" (UniqueName: \"kubernetes.io/projected/9fc231d4-04b0-4462-b869-067b8a369a4f-kube-api-access-2s8fx\") pod \"openshift-lws-operator-fd8ccff4c-7gqmr\" (UID: \"9fc231d4-04b0-4462-b869-067b8a369a4f\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-7gqmr" Jul 19 15:03:36.030840 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:03:36.030816 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9fc231d4-04b0-4462-b869-067b8a369a4f-tmp\") pod \"openshift-lws-operator-fd8ccff4c-7gqmr\" (UID: \"9fc231d4-04b0-4462-b869-067b8a369a4f\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-7gqmr" Jul 19 15:03:36.030937 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:03:36.030859 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2s8fx\" (UniqueName: \"kubernetes.io/projected/9fc231d4-04b0-4462-b869-067b8a369a4f-kube-api-access-2s8fx\") pod \"openshift-lws-operator-fd8ccff4c-7gqmr\" (UID: \"9fc231d4-04b0-4462-b869-067b8a369a4f\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-7gqmr" Jul 19 15:03:36.031189 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:03:36.031172 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9fc231d4-04b0-4462-b869-067b8a369a4f-tmp\") pod \"openshift-lws-operator-fd8ccff4c-7gqmr\" (UID: \"9fc231d4-04b0-4462-b869-067b8a369a4f\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-7gqmr" Jul 19 15:03:36.038744 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:03:36.038716 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2s8fx\" (UniqueName: \"kubernetes.io/projected/9fc231d4-04b0-4462-b869-067b8a369a4f-kube-api-access-2s8fx\") pod \"openshift-lws-operator-fd8ccff4c-7gqmr\" (UID: \"9fc231d4-04b0-4462-b869-067b8a369a4f\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-7gqmr" Jul 19 15:03:36.196428 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:03:36.196406 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-7gqmr" Jul 19 15:03:36.317688 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:03:36.317655 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-7gqmr"] Jul 19 15:03:36.320510 ip-10-0-138-220 kubenswrapper[2558]: W0719 15:03:36.320467 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9fc231d4_04b0_4462_b869_067b8a369a4f.slice/crio-a07c45992989a61ef0eab67d28345d09548957b823bacdcfcb37920fc921da0c WatchSource:0}: Error finding container a07c45992989a61ef0eab67d28345d09548957b823bacdcfcb37920fc921da0c: Status 404 returned error can't find the container with id a07c45992989a61ef0eab67d28345d09548957b823bacdcfcb37920fc921da0c Jul 19 15:03:36.321893 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:03:36.321874 2558 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 19 15:03:36.993917 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:03:36.993878 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-7gqmr" event={"ID":"9fc231d4-04b0-4462-b869-067b8a369a4f","Type":"ContainerStarted","Data":"a07c45992989a61ef0eab67d28345d09548957b823bacdcfcb37920fc921da0c"} Jul 19 15:03:39.001787 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:03:39.001749 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-7gqmr" event={"ID":"9fc231d4-04b0-4462-b869-067b8a369a4f","Type":"ContainerStarted","Data":"2be1c4582a07cbde9b13afcfcd85c3bf8c369fcc0a45bb283a96a0b8788407c2"} Jul 19 15:03:39.016461 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:03:39.016411 2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-7gqmr" podStartSLOduration=1.430520346 podStartE2EDuration="4.016391451s" totalImagesPullingTime="2.585871105s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-19 15:03:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-19 15:03:39.015529689 +0000 UTC m=+359.937232402" watchObservedRunningTime="2026-07-19 15:03:39.016391451 +0000 UTC m=+359.938094164" Jul 19 15:03:46.259484 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:03:46.259455 2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-57c99ccc67-wpkfm"] Jul 19 15:03:46.262656 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:03:46.262642 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-57c99ccc67-wpkfm" Jul 19 15:03:46.265482 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:03:46.265458 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\"" Jul 19 15:03:46.265604 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:03:46.265458 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-wtfmc\"" Jul 19 15:03:46.265604 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:03:46.265468 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\"" Jul 19 15:03:46.265604 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:03:46.265468 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\"" Jul 19 15:03:46.271386 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:03:46.271363 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-57c99ccc67-wpkfm"] Jul 19 15:03:46.303761 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:03:46.303729 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/031f67ac-e1a0-42d7-a44c-26e987378ffb-manager-config\") pod \"lws-controller-manager-57c99ccc67-wpkfm\" (UID: \"031f67ac-e1a0-42d7-a44c-26e987378ffb\") " pod="openshift-lws-operator/lws-controller-manager-57c99ccc67-wpkfm" Jul 19 15:03:46.303894 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:03:46.303782 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/031f67ac-e1a0-42d7-a44c-26e987378ffb-metrics-cert\") pod \"lws-controller-manager-57c99ccc67-wpkfm\" (UID: \"031f67ac-e1a0-42d7-a44c-26e987378ffb\") " pod="openshift-lws-operator/lws-controller-manager-57c99ccc67-wpkfm" Jul 19 15:03:46.303894 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:03:46.303799 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/031f67ac-e1a0-42d7-a44c-26e987378ffb-cert\") pod \"lws-controller-manager-57c99ccc67-wpkfm\" (UID: \"031f67ac-e1a0-42d7-a44c-26e987378ffb\") " pod="openshift-lws-operator/lws-controller-manager-57c99ccc67-wpkfm" Jul 19 15:03:46.303894 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:03:46.303854 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p2ljh\" (UniqueName: \"kubernetes.io/projected/031f67ac-e1a0-42d7-a44c-26e987378ffb-kube-api-access-p2ljh\") pod \"lws-controller-manager-57c99ccc67-wpkfm\" (UID: \"031f67ac-e1a0-42d7-a44c-26e987378ffb\") " pod="openshift-lws-operator/lws-controller-manager-57c99ccc67-wpkfm" Jul 19 15:03:46.404614 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:03:46.404587 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/031f67ac-e1a0-42d7-a44c-26e987378ffb-manager-config\") pod \"lws-controller-manager-57c99ccc67-wpkfm\" (UID: \"031f67ac-e1a0-42d7-a44c-26e987378ffb\") " pod="openshift-lws-operator/lws-controller-manager-57c99ccc67-wpkfm" Jul 19 15:03:46.404753 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:03:46.404630 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/031f67ac-e1a0-42d7-a44c-26e987378ffb-metrics-cert\") pod \"lws-controller-manager-57c99ccc67-wpkfm\" (UID: \"031f67ac-e1a0-42d7-a44c-26e987378ffb\") " pod="openshift-lws-operator/lws-controller-manager-57c99ccc67-wpkfm" Jul 19 15:03:46.404753 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:03:46.404651 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/031f67ac-e1a0-42d7-a44c-26e987378ffb-cert\") pod \"lws-controller-manager-57c99ccc67-wpkfm\" (UID: \"031f67ac-e1a0-42d7-a44c-26e987378ffb\") " pod="openshift-lws-operator/lws-controller-manager-57c99ccc67-wpkfm" Jul 19 15:03:46.404753 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:03:46.404677 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p2ljh\" (UniqueName: \"kubernetes.io/projected/031f67ac-e1a0-42d7-a44c-26e987378ffb-kube-api-access-p2ljh\") pod \"lws-controller-manager-57c99ccc67-wpkfm\" (UID: \"031f67ac-e1a0-42d7-a44c-26e987378ffb\") " pod="openshift-lws-operator/lws-controller-manager-57c99ccc67-wpkfm" Jul 19 15:03:46.405353 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:03:46.405327 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/031f67ac-e1a0-42d7-a44c-26e987378ffb-manager-config\") pod \"lws-controller-manager-57c99ccc67-wpkfm\" (UID: \"031f67ac-e1a0-42d7-a44c-26e987378ffb\") " pod="openshift-lws-operator/lws-controller-manager-57c99ccc67-wpkfm" Jul 19 15:03:46.407133 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:03:46.407115 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/031f67ac-e1a0-42d7-a44c-26e987378ffb-metrics-cert\") pod \"lws-controller-manager-57c99ccc67-wpkfm\" (UID: \"031f67ac-e1a0-42d7-a44c-26e987378ffb\") " pod="openshift-lws-operator/lws-controller-manager-57c99ccc67-wpkfm" Jul 19 15:03:46.407257 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:03:46.407241 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/031f67ac-e1a0-42d7-a44c-26e987378ffb-cert\") pod \"lws-controller-manager-57c99ccc67-wpkfm\" (UID: \"031f67ac-e1a0-42d7-a44c-26e987378ffb\") " pod="openshift-lws-operator/lws-controller-manager-57c99ccc67-wpkfm" Jul 19 15:03:46.412517 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:03:46.412484 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p2ljh\" (UniqueName: \"kubernetes.io/projected/031f67ac-e1a0-42d7-a44c-26e987378ffb-kube-api-access-p2ljh\") pod \"lws-controller-manager-57c99ccc67-wpkfm\" (UID: \"031f67ac-e1a0-42d7-a44c-26e987378ffb\") " pod="openshift-lws-operator/lws-controller-manager-57c99ccc67-wpkfm" Jul 19 15:03:46.571995 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:03:46.571903 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-57c99ccc67-wpkfm" Jul 19 15:03:46.696357 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:03:46.696326 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-57c99ccc67-wpkfm"] Jul 19 15:03:46.699401 ip-10-0-138-220 kubenswrapper[2558]: W0719 15:03:46.699370 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod031f67ac_e1a0_42d7_a44c_26e987378ffb.slice/crio-13885d8c9357cbce4b8f715e9d25e110e878227115ed3fa64240dade8121d3e8 WatchSource:0}: Error finding container 13885d8c9357cbce4b8f715e9d25e110e878227115ed3fa64240dade8121d3e8: Status 404 returned error can't find the container with id 13885d8c9357cbce4b8f715e9d25e110e878227115ed3fa64240dade8121d3e8 Jul 19 15:03:47.027872 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:03:47.027840 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-57c99ccc67-wpkfm" event={"ID":"031f67ac-e1a0-42d7-a44c-26e987378ffb","Type":"ContainerStarted","Data":"13885d8c9357cbce4b8f715e9d25e110e878227115ed3fa64240dade8121d3e8"} Jul 19 15:03:49.037080 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:03:49.037045 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-57c99ccc67-wpkfm" event={"ID":"031f67ac-e1a0-42d7-a44c-26e987378ffb","Type":"ContainerStarted","Data":"e4283f9e53e6e5e58ef89128d342dd9d89ce3db0dbbc1efc6b251aa22b3df6ee"} Jul 19 15:03:49.037531 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:03:49.037108 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-57c99ccc67-wpkfm" Jul 19 15:03:49.052544 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:03:49.052392 2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-57c99ccc67-wpkfm" podStartSLOduration=1.622662652 podStartE2EDuration="3.05237538s" totalImagesPullingTime="1.429712728s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-19 15:03:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-19 15:03:49.052253885 +0000 UTC m=+369.973956602" watchObservedRunningTime="2026-07-19 15:03:49.05237538 +0000 UTC m=+369.974078096" Jul 19 15:03:58.780115 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:03:58.780082 2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-74d6d868c8-sbwgw"] Jul 19 15:03:58.785585 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:03:58.785560 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-74d6d868c8-sbwgw" Jul 19 15:03:58.788449 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:03:58.788431 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Jul 19 15:03:58.788582 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:03:58.788546 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Jul 19 15:03:58.788734 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:03:58.788715 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-vwrkx\"" Jul 19 15:03:58.788871 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:03:58.788854 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Jul 19 15:03:58.789192 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:03:58.789175 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Jul 19 15:03:58.815311 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:03:58.815285 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-74d6d868c8-sbwgw"] Jul 19 15:03:58.895783 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:03:58.895761 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/080b867a-6665-459f-9a44-8b35cf3dafe0-webhook-cert\") pod \"opendatahub-operator-controller-manager-74d6d868c8-sbwgw\" (UID: \"080b867a-6665-459f-9a44-8b35cf3dafe0\") " pod="opendatahub/opendatahub-operator-controller-manager-74d6d868c8-sbwgw" Jul 19 15:03:58.895885 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:03:58.895814 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/080b867a-6665-459f-9a44-8b35cf3dafe0-tmp\") pod \"opendatahub-operator-controller-manager-74d6d868c8-sbwgw\" (UID: \"080b867a-6665-459f-9a44-8b35cf3dafe0\") " pod="opendatahub/opendatahub-operator-controller-manager-74d6d868c8-sbwgw" Jul 19 15:03:58.895885 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:03:58.895851 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/080b867a-6665-459f-9a44-8b35cf3dafe0-manifests\") pod \"opendatahub-operator-controller-manager-74d6d868c8-sbwgw\" (UID: \"080b867a-6665-459f-9a44-8b35cf3dafe0\") " pod="opendatahub/opendatahub-operator-controller-manager-74d6d868c8-sbwgw" Jul 19 15:03:58.895885 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:03:58.895875 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/080b867a-6665-459f-9a44-8b35cf3dafe0-apiservice-cert\") pod \"opendatahub-operator-controller-manager-74d6d868c8-sbwgw\" (UID: \"080b867a-6665-459f-9a44-8b35cf3dafe0\") " pod="opendatahub/opendatahub-operator-controller-manager-74d6d868c8-sbwgw" Jul 19 15:03:58.896007 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:03:58.895957 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w5r4w\" (UniqueName: \"kubernetes.io/projected/080b867a-6665-459f-9a44-8b35cf3dafe0-kube-api-access-w5r4w\") pod \"opendatahub-operator-controller-manager-74d6d868c8-sbwgw\" (UID: \"080b867a-6665-459f-9a44-8b35cf3dafe0\") " pod="opendatahub/opendatahub-operator-controller-manager-74d6d868c8-sbwgw" Jul 19 15:03:58.997056 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:03:58.997006 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/080b867a-6665-459f-9a44-8b35cf3dafe0-tmp\") pod \"opendatahub-operator-controller-manager-74d6d868c8-sbwgw\" (UID: \"080b867a-6665-459f-9a44-8b35cf3dafe0\") " pod="opendatahub/opendatahub-operator-controller-manager-74d6d868c8-sbwgw" Jul 19 15:03:58.997171 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:03:58.997060 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/080b867a-6665-459f-9a44-8b35cf3dafe0-manifests\") pod \"opendatahub-operator-controller-manager-74d6d868c8-sbwgw\" (UID: \"080b867a-6665-459f-9a44-8b35cf3dafe0\") " pod="opendatahub/opendatahub-operator-controller-manager-74d6d868c8-sbwgw" Jul 19 15:03:58.997206 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:03:58.997171 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/080b867a-6665-459f-9a44-8b35cf3dafe0-apiservice-cert\") pod \"opendatahub-operator-controller-manager-74d6d868c8-sbwgw\" (UID: \"080b867a-6665-459f-9a44-8b35cf3dafe0\") " pod="opendatahub/opendatahub-operator-controller-manager-74d6d868c8-sbwgw" Jul 19 15:03:58.997283 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:03:58.997261 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w5r4w\" (UniqueName: \"kubernetes.io/projected/080b867a-6665-459f-9a44-8b35cf3dafe0-kube-api-access-w5r4w\") pod \"opendatahub-operator-controller-manager-74d6d868c8-sbwgw\" (UID: \"080b867a-6665-459f-9a44-8b35cf3dafe0\") " pod="opendatahub/opendatahub-operator-controller-manager-74d6d868c8-sbwgw" Jul 19 15:03:58.997326 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:03:58.997310 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/080b867a-6665-459f-9a44-8b35cf3dafe0-webhook-cert\") pod \"opendatahub-operator-controller-manager-74d6d868c8-sbwgw\" (UID: \"080b867a-6665-459f-9a44-8b35cf3dafe0\") " pod="opendatahub/opendatahub-operator-controller-manager-74d6d868c8-sbwgw" Jul 19 15:03:58.997425 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:03:58.997406 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/080b867a-6665-459f-9a44-8b35cf3dafe0-manifests\") pod \"opendatahub-operator-controller-manager-74d6d868c8-sbwgw\" (UID: \"080b867a-6665-459f-9a44-8b35cf3dafe0\") " pod="opendatahub/opendatahub-operator-controller-manager-74d6d868c8-sbwgw" Jul 19 15:03:58.999511 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:03:58.999478 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/080b867a-6665-459f-9a44-8b35cf3dafe0-tmp\") pod \"opendatahub-operator-controller-manager-74d6d868c8-sbwgw\" (UID: \"080b867a-6665-459f-9a44-8b35cf3dafe0\") " pod="opendatahub/opendatahub-operator-controller-manager-74d6d868c8-sbwgw" Jul 19 15:03:58.999600 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:03:58.999581 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/080b867a-6665-459f-9a44-8b35cf3dafe0-apiservice-cert\") pod \"opendatahub-operator-controller-manager-74d6d868c8-sbwgw\" (UID: \"080b867a-6665-459f-9a44-8b35cf3dafe0\") " pod="opendatahub/opendatahub-operator-controller-manager-74d6d868c8-sbwgw" Jul 19 15:03:58.999703 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:03:58.999686 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/080b867a-6665-459f-9a44-8b35cf3dafe0-webhook-cert\") pod \"opendatahub-operator-controller-manager-74d6d868c8-sbwgw\" (UID: \"080b867a-6665-459f-9a44-8b35cf3dafe0\") " pod="opendatahub/opendatahub-operator-controller-manager-74d6d868c8-sbwgw" Jul 19 15:03:59.005705 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:03:59.005681 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w5r4w\" (UniqueName: \"kubernetes.io/projected/080b867a-6665-459f-9a44-8b35cf3dafe0-kube-api-access-w5r4w\") pod \"opendatahub-operator-controller-manager-74d6d868c8-sbwgw\" (UID: \"080b867a-6665-459f-9a44-8b35cf3dafe0\") " pod="opendatahub/opendatahub-operator-controller-manager-74d6d868c8-sbwgw" Jul 19 15:03:59.096040 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:03:59.095962 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-74d6d868c8-sbwgw" Jul 19 15:03:59.234882 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:03:59.234852 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-74d6d868c8-sbwgw"] Jul 19 15:03:59.237844 ip-10-0-138-220 kubenswrapper[2558]: W0719 15:03:59.237820 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod080b867a_6665_459f_9a44_8b35cf3dafe0.slice/crio-8f8d8a93d09adce150a806e4491d1a1013b0c9378ac6dd52a10c02a61e7a8d2a WatchSource:0}: Error finding container 8f8d8a93d09adce150a806e4491d1a1013b0c9378ac6dd52a10c02a61e7a8d2a: Status 404 returned error can't find the container with id 8f8d8a93d09adce150a806e4491d1a1013b0c9378ac6dd52a10c02a61e7a8d2a Jul 19 15:04:00.042544 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:04:00.042503 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-57c99ccc67-wpkfm" Jul 19 15:04:00.079725 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:04:00.079689 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-74d6d868c8-sbwgw" event={"ID":"080b867a-6665-459f-9a44-8b35cf3dafe0","Type":"ContainerStarted","Data":"8f8d8a93d09adce150a806e4491d1a1013b0c9378ac6dd52a10c02a61e7a8d2a"} Jul 19 15:04:03.092234 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:04:03.092195 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-74d6d868c8-sbwgw" event={"ID":"080b867a-6665-459f-9a44-8b35cf3dafe0","Type":"ContainerStarted","Data":"d29bb045ebaa81c8a3860032454d2b1f71b0238d53ff260cfb1cdfdd943dabae"} Jul 19 15:04:04.097600 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:04:04.097556 2558 generic.go:356] "Generic (PLEG): container finished" podID="080b867a-6665-459f-9a44-8b35cf3dafe0" containerID="d29bb045ebaa81c8a3860032454d2b1f71b0238d53ff260cfb1cdfdd943dabae" exitCode=0 Jul 19 15:04:04.098096 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:04:04.097646 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-74d6d868c8-sbwgw" event={"ID":"080b867a-6665-459f-9a44-8b35cf3dafe0","Type":"ContainerDied","Data":"d29bb045ebaa81c8a3860032454d2b1f71b0238d53ff260cfb1cdfdd943dabae"} Jul 19 15:04:05.103043 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:04:05.102979 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-74d6d868c8-sbwgw" event={"ID":"080b867a-6665-459f-9a44-8b35cf3dafe0","Type":"ContainerStarted","Data":"9fe85a4b0faab6c41098478d7c96484dc2422d47a8c090c01dccd48e6972d4ff"} Jul 19 15:04:05.103443 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:04:05.103189 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-74d6d868c8-sbwgw" Jul 19 15:04:05.125101 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:04:05.125050 2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-74d6d868c8-sbwgw" podStartSLOduration=2.743983001 podStartE2EDuration="7.125012748s" totalImagesPullingTime="3.323788396s" totalInitContainerRuntime="1.057241351s" isStatefulPod=false podCreationTimestamp="2026-07-19 15:03:58 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-19 15:04:05.123095103 +0000 UTC m=+386.044797818" watchObservedRunningTime="2026-07-19 15:04:05.125012748 +0000 UTC m=+386.046715462" Jul 19 15:04:16.108396 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:04:16.108361 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-74d6d868c8-sbwgw" Jul 19 15:04:25.713143 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:04:25.713108 2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-94bb8fbfd-jbnb7"] Jul 19 15:04:25.724343 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:04:25.724318 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-jbnb7" Jul 19 15:04:25.726616 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:04:25.726593 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Jul 19 15:04:25.726747 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:04:25.726691 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Jul 19 15:04:25.726810 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:04:25.726742 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Jul 19 15:04:25.727018 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:04:25.727003 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Jul 19 15:04:25.727293 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:04:25.727277 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-447ml\"" Jul 19 15:04:25.727371 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:04:25.727354 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Jul 19 15:04:25.727427 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:04:25.727366 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Jul 19 15:04:25.733438 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:04:25.733415 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-94bb8fbfd-jbnb7"] Jul 19 15:04:25.808929 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:04:25.808901 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/2928718a-b22e-4315-b01a-4f2cabed02fa-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-94bb8fbfd-jbnb7\" (UID: \"2928718a-b22e-4315-b01a-4f2cabed02fa\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-jbnb7" Jul 19 15:04:25.809116 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:04:25.808936 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/2928718a-b22e-4315-b01a-4f2cabed02fa-cacerts\") pod \"istiod-openshift-gateway-94bb8fbfd-jbnb7\" (UID: \"2928718a-b22e-4315-b01a-4f2cabed02fa\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-jbnb7" Jul 19 15:04:25.809116 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:04:25.808960 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j2rp7\" (UniqueName: \"kubernetes.io/projected/2928718a-b22e-4315-b01a-4f2cabed02fa-kube-api-access-j2rp7\") pod \"istiod-openshift-gateway-94bb8fbfd-jbnb7\" (UID: \"2928718a-b22e-4315-b01a-4f2cabed02fa\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-jbnb7" Jul 19 15:04:25.809116 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:04:25.809062 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2928718a-b22e-4315-b01a-4f2cabed02fa-istio-token\") pod \"istiod-openshift-gateway-94bb8fbfd-jbnb7\" (UID: \"2928718a-b22e-4315-b01a-4f2cabed02fa\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-jbnb7" Jul 19 15:04:25.809116 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:04:25.809105 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/2928718a-b22e-4315-b01a-4f2cabed02fa-local-certs\") pod \"istiod-openshift-gateway-94bb8fbfd-jbnb7\" (UID: \"2928718a-b22e-4315-b01a-4f2cabed02fa\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-jbnb7" Jul 19 15:04:25.809305 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:04:25.809136 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/2928718a-b22e-4315-b01a-4f2cabed02fa-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-94bb8fbfd-jbnb7\" (UID: \"2928718a-b22e-4315-b01a-4f2cabed02fa\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-jbnb7" Jul 19 15:04:25.809305 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:04:25.809172 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/2928718a-b22e-4315-b01a-4f2cabed02fa-istio-kubeconfig\") pod \"istiod-openshift-gateway-94bb8fbfd-jbnb7\" (UID: \"2928718a-b22e-4315-b01a-4f2cabed02fa\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-jbnb7" Jul 19 15:04:25.910379 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:04:25.910347 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j2rp7\" (UniqueName: \"kubernetes.io/projected/2928718a-b22e-4315-b01a-4f2cabed02fa-kube-api-access-j2rp7\") pod \"istiod-openshift-gateway-94bb8fbfd-jbnb7\" (UID: \"2928718a-b22e-4315-b01a-4f2cabed02fa\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-jbnb7" Jul 19 15:04:25.910547 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:04:25.910388 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2928718a-b22e-4315-b01a-4f2cabed02fa-istio-token\") pod \"istiod-openshift-gateway-94bb8fbfd-jbnb7\" (UID: \"2928718a-b22e-4315-b01a-4f2cabed02fa\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-jbnb7" Jul 19 15:04:25.910547 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:04:25.910411 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/2928718a-b22e-4315-b01a-4f2cabed02fa-local-certs\") pod \"istiod-openshift-gateway-94bb8fbfd-jbnb7\" (UID: \"2928718a-b22e-4315-b01a-4f2cabed02fa\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-jbnb7" Jul 19 15:04:25.910547 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:04:25.910539 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/2928718a-b22e-4315-b01a-4f2cabed02fa-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-94bb8fbfd-jbnb7\" (UID: \"2928718a-b22e-4315-b01a-4f2cabed02fa\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-jbnb7" Jul 19 15:04:25.910711 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:04:25.910583 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/2928718a-b22e-4315-b01a-4f2cabed02fa-istio-kubeconfig\") pod \"istiod-openshift-gateway-94bb8fbfd-jbnb7\" (UID: \"2928718a-b22e-4315-b01a-4f2cabed02fa\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-jbnb7" Jul 19 15:04:25.910711 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:04:25.910623 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/2928718a-b22e-4315-b01a-4f2cabed02fa-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-94bb8fbfd-jbnb7\" (UID: \"2928718a-b22e-4315-b01a-4f2cabed02fa\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-jbnb7" Jul 19 15:04:25.910711 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:04:25.910653 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/2928718a-b22e-4315-b01a-4f2cabed02fa-cacerts\") pod \"istiod-openshift-gateway-94bb8fbfd-jbnb7\" (UID: \"2928718a-b22e-4315-b01a-4f2cabed02fa\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-jbnb7" Jul 19 15:04:25.911157 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:04:25.911135 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/2928718a-b22e-4315-b01a-4f2cabed02fa-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-94bb8fbfd-jbnb7\" (UID: \"2928718a-b22e-4315-b01a-4f2cabed02fa\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-jbnb7" Jul 19 15:04:25.912812 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:04:25.912786 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/2928718a-b22e-4315-b01a-4f2cabed02fa-local-certs\") pod \"istiod-openshift-gateway-94bb8fbfd-jbnb7\" (UID: \"2928718a-b22e-4315-b01a-4f2cabed02fa\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-jbnb7" Jul 19 15:04:25.912977 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:04:25.912957 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/2928718a-b22e-4315-b01a-4f2cabed02fa-istio-kubeconfig\") pod \"istiod-openshift-gateway-94bb8fbfd-jbnb7\" (UID: \"2928718a-b22e-4315-b01a-4f2cabed02fa\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-jbnb7" Jul 19 15:04:25.913073 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:04:25.913057 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/2928718a-b22e-4315-b01a-4f2cabed02fa-cacerts\") pod \"istiod-openshift-gateway-94bb8fbfd-jbnb7\" (UID: \"2928718a-b22e-4315-b01a-4f2cabed02fa\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-jbnb7" Jul 19 15:04:25.913160 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:04:25.913145 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/2928718a-b22e-4315-b01a-4f2cabed02fa-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-94bb8fbfd-jbnb7\" (UID: \"2928718a-b22e-4315-b01a-4f2cabed02fa\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-jbnb7" Jul 19 15:04:25.934819 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:04:25.934790 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2928718a-b22e-4315-b01a-4f2cabed02fa-istio-token\") pod \"istiod-openshift-gateway-94bb8fbfd-jbnb7\" (UID: \"2928718a-b22e-4315-b01a-4f2cabed02fa\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-jbnb7" Jul 19 15:04:25.934984 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:04:25.934968 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j2rp7\" (UniqueName: \"kubernetes.io/projected/2928718a-b22e-4315-b01a-4f2cabed02fa-kube-api-access-j2rp7\") pod \"istiod-openshift-gateway-94bb8fbfd-jbnb7\" (UID: \"2928718a-b22e-4315-b01a-4f2cabed02fa\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-jbnb7" Jul 19 15:04:26.034192 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:04:26.034124 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-jbnb7" Jul 19 15:04:26.164525 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:04:26.164419 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-94bb8fbfd-jbnb7"] Jul 19 15:04:26.166981 ip-10-0-138-220 kubenswrapper[2558]: W0719 15:04:26.166956 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2928718a_b22e_4315_b01a_4f2cabed02fa.slice/crio-fd7922245b3f78f3d8cc97f374b2fb44af5dedd2e440d60c4f7461c19d83f058 WatchSource:0}: Error finding container fd7922245b3f78f3d8cc97f374b2fb44af5dedd2e440d60c4f7461c19d83f058: Status 404 returned error can't find the container with id fd7922245b3f78f3d8cc97f374b2fb44af5dedd2e440d60c4f7461c19d83f058 Jul 19 15:04:26.180584 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:04:26.180557 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-jbnb7" event={"ID":"2928718a-b22e-4315-b01a-4f2cabed02fa","Type":"ContainerStarted","Data":"fd7922245b3f78f3d8cc97f374b2fb44af5dedd2e440d60c4f7461c19d83f058"} Jul 19 15:04:28.288217 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:04:28.288182 2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 19 15:04:28.288540 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:04:28.288255 2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 19 15:04:29.194817 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:04:29.194781 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-jbnb7" event={"ID":"2928718a-b22e-4315-b01a-4f2cabed02fa","Type":"ContainerStarted","Data":"fb008c36c1d5859baca60c58b356bdd89c5791ea1fa7d4755f12db2399fef15b"} Jul 19 15:04:29.195003 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:04:29.194919 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-jbnb7" Jul 19 15:04:29.196606 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:04:29.196583 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-jbnb7" Jul 19 15:04:29.215682 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:04:29.215636 2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-jbnb7" podStartSLOduration=2.096669879 podStartE2EDuration="4.215617698s" totalImagesPullingTime="2.118947819s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-19 15:04:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-19 15:04:29.213769374 +0000 UTC m=+410.135472088" watchObservedRunningTime="2026-07-19 15:04:29.215617698 +0000 UTC m=+410.137320414" Jul 19 15:07:12.169891 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:07:12.169851 2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-d4lll"] Jul 19 15:07:12.172980 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:07:12.172963 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-54d6d988d5-d4lll" Jul 19 15:07:12.175041 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:07:12.175002 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\"" Jul 19 15:07:12.175148 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:07:12.175078 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-7rbnp\"" Jul 19 15:07:12.175192 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:07:12.175168 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Jul 19 15:07:12.175746 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:07:12.175732 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Jul 19 15:07:12.182467 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:07:12.182445 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-d4lll"] Jul 19 15:07:12.207784 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:07:12.207754 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-d4lll"] Jul 19 15:07:12.230503 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:07:12.230473 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/8f25f7b4-20d5-483b-9d94-e01ed527dcb0-config-file\") pod \"limitador-limitador-54d6d988d5-d4lll\" (UID: \"8f25f7b4-20d5-483b-9d94-e01ed527dcb0\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-d4lll" Jul 19 15:07:12.230656 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:07:12.230534 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jpqrr\" (UniqueName: \"kubernetes.io/projected/8f25f7b4-20d5-483b-9d94-e01ed527dcb0-kube-api-access-jpqrr\") pod \"limitador-limitador-54d6d988d5-d4lll\" (UID: \"8f25f7b4-20d5-483b-9d94-e01ed527dcb0\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-d4lll" Jul 19 15:07:12.331336 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:07:12.331305 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/8f25f7b4-20d5-483b-9d94-e01ed527dcb0-config-file\") pod \"limitador-limitador-54d6d988d5-d4lll\" (UID: \"8f25f7b4-20d5-483b-9d94-e01ed527dcb0\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-d4lll" Jul 19 15:07:12.331497 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:07:12.331350 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jpqrr\" (UniqueName: \"kubernetes.io/projected/8f25f7b4-20d5-483b-9d94-e01ed527dcb0-kube-api-access-jpqrr\") pod \"limitador-limitador-54d6d988d5-d4lll\" (UID: \"8f25f7b4-20d5-483b-9d94-e01ed527dcb0\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-d4lll" Jul 19 15:07:12.331985 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:07:12.331960 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/8f25f7b4-20d5-483b-9d94-e01ed527dcb0-config-file\") pod \"limitador-limitador-54d6d988d5-d4lll\" (UID: \"8f25f7b4-20d5-483b-9d94-e01ed527dcb0\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-d4lll" Jul 19 15:07:12.341548 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:07:12.341529 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jpqrr\" (UniqueName: \"kubernetes.io/projected/8f25f7b4-20d5-483b-9d94-e01ed527dcb0-kube-api-access-jpqrr\") pod \"limitador-limitador-54d6d988d5-d4lll\" (UID: \"8f25f7b4-20d5-483b-9d94-e01ed527dcb0\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-d4lll" Jul 19 15:07:12.483896 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:07:12.483829 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-54d6d988d5-d4lll" Jul 19 15:07:12.606414 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:07:12.606386 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-d4lll"] Jul 19 15:07:12.609482 ip-10-0-138-220 kubenswrapper[2558]: W0719 15:07:12.609450 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8f25f7b4_20d5_483b_9d94_e01ed527dcb0.slice/crio-56610ef824c9dde49208dd35f05977c64f1343aaa13f65cd04e9225f17f70f70 WatchSource:0}: Error finding container 56610ef824c9dde49208dd35f05977c64f1343aaa13f65cd04e9225f17f70f70: Status 404 returned error can't find the container with id 56610ef824c9dde49208dd35f05977c64f1343aaa13f65cd04e9225f17f70f70 Jul 19 15:07:12.766557 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:07:12.766486 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-54d6d988d5-d4lll" event={"ID":"8f25f7b4-20d5-483b-9d94-e01ed527dcb0","Type":"ContainerStarted","Data":"56610ef824c9dde49208dd35f05977c64f1343aaa13f65cd04e9225f17f70f70"} Jul 19 15:07:15.781842 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:07:15.781802 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-54d6d988d5-d4lll" event={"ID":"8f25f7b4-20d5-483b-9d94-e01ed527dcb0","Type":"ContainerStarted","Data":"8cda01961a17af9274bd60c14e2dd0c8f9f08dc3a790f7be41c914f04d4af873"} Jul 19 15:07:15.782253 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:07:15.781901 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-54d6d988d5-d4lll" Jul 19 15:07:15.798665 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:07:15.798605 2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-54d6d988d5-d4lll" podStartSLOduration=1.347642788 podStartE2EDuration="3.79859176s" totalImagesPullingTime="2.450948972s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-19 15:07:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-19 15:07:15.796948629 +0000 UTC m=+576.718651342" watchObservedRunningTime="2026-07-19 15:07:15.79859176 +0000 UTC m=+576.720294473" Jul 19 15:07:26.787402 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:07:26.787367 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-54d6d988d5-d4lll" Jul 19 15:07:39.529623 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:07:39.529586 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-5wfzs_5b2d3a07-86b2-4ba2-8a71-c2d89ada79c6/ovn-acl-logging/0.log" Jul 19 15:07:39.534290 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:07:39.534270 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-5wfzs_5b2d3a07-86b2-4ba2-8a71-c2d89ada79c6/ovn-acl-logging/0.log" Jul 19 15:08:23.760951 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:08:23.760919 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-d4lll"] Jul 19 15:08:45.904905 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:08:45.904864 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-d4lll"] Jul 19 15:08:56.185738 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:08:56.185706 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-d4lll"] Jul 19 15:09:17.293530 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:09:17.293500 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-d4lll"] Jul 19 15:09:26.893659 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:09:26.893618 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-d4lll"] Jul 19 15:09:40.991125 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:09:40.991044 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-d4lll"] Jul 19 15:09:53.083053 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:09:53.083003 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-d4lll"] Jul 19 15:11:22.784581 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:11:22.784545 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-d4lll"] Jul 19 15:11:32.183577 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:11:32.183539 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-d4lll"] Jul 19 15:11:42.488722 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:11:42.488687 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-d4lll"] Jul 19 15:12:39.551912 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:12:39.551841 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-5wfzs_5b2d3a07-86b2-4ba2-8a71-c2d89ada79c6/ovn-acl-logging/0.log" Jul 19 15:12:39.553858 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:12:39.553837 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-5wfzs_5b2d3a07-86b2-4ba2-8a71-c2d89ada79c6/ovn-acl-logging/0.log" Jul 19 15:12:41.165710 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:12:41.165683 2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-6cb557cbc4-2jbk2"] Jul 19 15:12:41.168540 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:12:41.168522 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6cb557cbc4-2jbk2" Jul 19 15:12:41.170797 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:12:41.170771 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Jul 19 15:12:41.171513 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:12:41.171495 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-6dtfm\"" Jul 19 15:12:41.171607 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:12:41.171503 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Jul 19 15:12:41.177311 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:12:41.177291 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-6cb557cbc4-2jbk2"] Jul 19 15:12:41.210151 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:12:41.210114 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/f5172e55-cf90-4501-8ec2-6e4903a3b174-webhook-cert\") pod \"maas-controller-6cb557cbc4-2jbk2\" (UID: \"f5172e55-cf90-4501-8ec2-6e4903a3b174\") " pod="opendatahub/maas-controller-6cb557cbc4-2jbk2" Jul 19 15:12:41.210262 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:12:41.210187 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nrwpk\" (UniqueName: \"kubernetes.io/projected/f5172e55-cf90-4501-8ec2-6e4903a3b174-kube-api-access-nrwpk\") pod \"maas-controller-6cb557cbc4-2jbk2\" (UID: \"f5172e55-cf90-4501-8ec2-6e4903a3b174\") " pod="opendatahub/maas-controller-6cb557cbc4-2jbk2" Jul 19 15:12:41.311409 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:12:41.311381 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nrwpk\" (UniqueName: \"kubernetes.io/projected/f5172e55-cf90-4501-8ec2-6e4903a3b174-kube-api-access-nrwpk\") pod \"maas-controller-6cb557cbc4-2jbk2\" (UID: \"f5172e55-cf90-4501-8ec2-6e4903a3b174\") " pod="opendatahub/maas-controller-6cb557cbc4-2jbk2" Jul 19 15:12:41.311501 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:12:41.311438 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/f5172e55-cf90-4501-8ec2-6e4903a3b174-webhook-cert\") pod \"maas-controller-6cb557cbc4-2jbk2\" (UID: \"f5172e55-cf90-4501-8ec2-6e4903a3b174\") " pod="opendatahub/maas-controller-6cb557cbc4-2jbk2" Jul 19 15:12:41.313813 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:12:41.313796 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/f5172e55-cf90-4501-8ec2-6e4903a3b174-webhook-cert\") pod \"maas-controller-6cb557cbc4-2jbk2\" (UID: \"f5172e55-cf90-4501-8ec2-6e4903a3b174\") " pod="opendatahub/maas-controller-6cb557cbc4-2jbk2" Jul 19 15:12:41.318952 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:12:41.318935 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nrwpk\" (UniqueName: \"kubernetes.io/projected/f5172e55-cf90-4501-8ec2-6e4903a3b174-kube-api-access-nrwpk\") pod \"maas-controller-6cb557cbc4-2jbk2\" (UID: \"f5172e55-cf90-4501-8ec2-6e4903a3b174\") " pod="opendatahub/maas-controller-6cb557cbc4-2jbk2" Jul 19 15:12:41.479050 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:12:41.478981 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6cb557cbc4-2jbk2" Jul 19 15:12:41.605886 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:12:41.605724 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-6cb557cbc4-2jbk2"] Jul 19 15:12:41.608587 ip-10-0-138-220 kubenswrapper[2558]: W0719 15:12:41.608560 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf5172e55_cf90_4501_8ec2_6e4903a3b174.slice/crio-be3836247b02468a62c1fc2fff9c612da4f13f713d7c1068d98ac002034160d8 WatchSource:0}: Error finding container be3836247b02468a62c1fc2fff9c612da4f13f713d7c1068d98ac002034160d8: Status 404 returned error can't find the container with id be3836247b02468a62c1fc2fff9c612da4f13f713d7c1068d98ac002034160d8 Jul 19 15:12:41.609811 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:12:41.609792 2558 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 19 15:12:42.067086 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:12:42.067055 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6cb557cbc4-2jbk2" event={"ID":"f5172e55-cf90-4501-8ec2-6e4903a3b174","Type":"ContainerStarted","Data":"be3836247b02468a62c1fc2fff9c612da4f13f713d7c1068d98ac002034160d8"} Jul 19 15:12:44.076502 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:12:44.076467 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6cb557cbc4-2jbk2" event={"ID":"f5172e55-cf90-4501-8ec2-6e4903a3b174","Type":"ContainerStarted","Data":"cb9ead24d0c7fe8b3f6db694eef53363d08d44477ddadf7587ca9d150648a80b"} Jul 19 15:12:44.076844 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:12:44.076596 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-6cb557cbc4-2jbk2" Jul 19 15:12:44.106557 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:12:44.106481 2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-6cb557cbc4-2jbk2" podStartSLOduration=0.871692395 podStartE2EDuration="3.106466014s" totalImagesPullingTime="2.234773619s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-19 15:12:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-19 15:12:44.104686069 +0000 UTC m=+905.026388784" watchObservedRunningTime="2026-07-19 15:12:44.106466014 +0000 UTC m=+905.028168724" Jul 19 15:12:55.083484 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:12:55.083455 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-6cb557cbc4-2jbk2" Jul 19 15:12:59.719896 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:12:59.719861 2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-6cb557cbc4-2jbk2"] Jul 19 15:12:59.720299 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:12:59.720093 2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-6cb557cbc4-2jbk2" podUID="f5172e55-cf90-4501-8ec2-6e4903a3b174" containerName="manager" containerID="cri-o://cb9ead24d0c7fe8b3f6db694eef53363d08d44477ddadf7587ca9d150648a80b" gracePeriod=30 Jul 19 15:13:00.137406 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:13:00.137373 2558 generic.go:356] "Generic (PLEG): container finished" podID="f5172e55-cf90-4501-8ec2-6e4903a3b174" containerID="cb9ead24d0c7fe8b3f6db694eef53363d08d44477ddadf7587ca9d150648a80b" exitCode=0 Jul 19 15:13:00.137570 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:13:00.137405 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6cb557cbc4-2jbk2" event={"ID":"f5172e55-cf90-4501-8ec2-6e4903a3b174","Type":"ContainerDied","Data":"cb9ead24d0c7fe8b3f6db694eef53363d08d44477ddadf7587ca9d150648a80b"} Jul 19 15:13:00.167477 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:13:00.167452 2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6cb557cbc4-2jbk2" Jul 19 15:13:00.259161 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:13:00.259101 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nrwpk\" (UniqueName: \"kubernetes.io/projected/f5172e55-cf90-4501-8ec2-6e4903a3b174-kube-api-access-nrwpk\") pod \"f5172e55-cf90-4501-8ec2-6e4903a3b174\" (UID: \"f5172e55-cf90-4501-8ec2-6e4903a3b174\") " Jul 19 15:13:00.259287 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:13:00.259193 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/f5172e55-cf90-4501-8ec2-6e4903a3b174-webhook-cert\") pod \"f5172e55-cf90-4501-8ec2-6e4903a3b174\" (UID: \"f5172e55-cf90-4501-8ec2-6e4903a3b174\") " Jul 19 15:13:00.261456 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:13:00.261426 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f5172e55-cf90-4501-8ec2-6e4903a3b174-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "f5172e55-cf90-4501-8ec2-6e4903a3b174" (UID: "f5172e55-cf90-4501-8ec2-6e4903a3b174"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 19 15:13:00.261456 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:13:00.261426 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f5172e55-cf90-4501-8ec2-6e4903a3b174-kube-api-access-nrwpk" (OuterVolumeSpecName: "kube-api-access-nrwpk") pod "f5172e55-cf90-4501-8ec2-6e4903a3b174" (UID: "f5172e55-cf90-4501-8ec2-6e4903a3b174"). InnerVolumeSpecName "kube-api-access-nrwpk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 19 15:13:00.360003 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:13:00.359971 2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nrwpk\" (UniqueName: \"kubernetes.io/projected/f5172e55-cf90-4501-8ec2-6e4903a3b174-kube-api-access-nrwpk\") on node \"ip-10-0-138-220.ec2.internal\" DevicePath \"\"" Jul 19 15:13:00.360003 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:13:00.359998 2558 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/f5172e55-cf90-4501-8ec2-6e4903a3b174-webhook-cert\") on node \"ip-10-0-138-220.ec2.internal\" DevicePath \"\"" Jul 19 15:13:01.142887 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:13:01.142845 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6cb557cbc4-2jbk2" event={"ID":"f5172e55-cf90-4501-8ec2-6e4903a3b174","Type":"ContainerDied","Data":"be3836247b02468a62c1fc2fff9c612da4f13f713d7c1068d98ac002034160d8"} Jul 19 15:13:01.143334 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:13:01.142934 2558 scope.go:117] "RemoveContainer" containerID="cb9ead24d0c7fe8b3f6db694eef53363d08d44477ddadf7587ca9d150648a80b" Jul 19 15:13:01.143334 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:13:01.142865 2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6cb557cbc4-2jbk2" Jul 19 15:13:01.163085 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:13:01.163061 2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-6cb557cbc4-2jbk2"] Jul 19 15:13:01.166640 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:13:01.166620 2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-6cb557cbc4-2jbk2"] Jul 19 15:13:01.566156 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:13:01.566086 2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f5172e55-cf90-4501-8ec2-6e4903a3b174" path="/var/lib/kubelet/pods/f5172e55-cf90-4501-8ec2-6e4903a3b174/volumes" Jul 19 15:13:01.966304 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:13:01.966270 2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-6cb557cbc4-rqxpv"] Jul 19 15:13:01.966615 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:13:01.966598 2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f5172e55-cf90-4501-8ec2-6e4903a3b174" containerName="manager" Jul 19 15:13:01.966717 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:13:01.966627 2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="f5172e55-cf90-4501-8ec2-6e4903a3b174" containerName="manager" Jul 19 15:13:01.966773 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:13:01.966722 2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="f5172e55-cf90-4501-8ec2-6e4903a3b174" containerName="manager" Jul 19 15:13:01.971066 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:13:01.971045 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6cb557cbc4-rqxpv" Jul 19 15:13:01.973922 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:13:01.973902 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-6dtfm\"" Jul 19 15:13:01.973922 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:13:01.973917 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Jul 19 15:13:01.974093 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:13:01.973902 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Jul 19 15:13:01.977347 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:13:01.977321 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-6cb557cbc4-rqxpv"] Jul 19 15:13:02.074085 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:13:02.074056 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hnf84\" (UniqueName: \"kubernetes.io/projected/d50f665d-6f6e-436f-b1bc-94bd6aa10e6d-kube-api-access-hnf84\") pod \"maas-controller-6cb557cbc4-rqxpv\" (UID: \"d50f665d-6f6e-436f-b1bc-94bd6aa10e6d\") " pod="opendatahub/maas-controller-6cb557cbc4-rqxpv" Jul 19 15:13:02.074209 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:13:02.074110 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/d50f665d-6f6e-436f-b1bc-94bd6aa10e6d-webhook-cert\") pod \"maas-controller-6cb557cbc4-rqxpv\" (UID: \"d50f665d-6f6e-436f-b1bc-94bd6aa10e6d\") " pod="opendatahub/maas-controller-6cb557cbc4-rqxpv" Jul 19 15:13:02.175444 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:13:02.175411 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hnf84\" (UniqueName: \"kubernetes.io/projected/d50f665d-6f6e-436f-b1bc-94bd6aa10e6d-kube-api-access-hnf84\") pod \"maas-controller-6cb557cbc4-rqxpv\" (UID: \"d50f665d-6f6e-436f-b1bc-94bd6aa10e6d\") " pod="opendatahub/maas-controller-6cb557cbc4-rqxpv" Jul 19 15:13:02.175770 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:13:02.175455 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/d50f665d-6f6e-436f-b1bc-94bd6aa10e6d-webhook-cert\") pod \"maas-controller-6cb557cbc4-rqxpv\" (UID: \"d50f665d-6f6e-436f-b1bc-94bd6aa10e6d\") " pod="opendatahub/maas-controller-6cb557cbc4-rqxpv" Jul 19 15:13:02.177862 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:13:02.177844 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/d50f665d-6f6e-436f-b1bc-94bd6aa10e6d-webhook-cert\") pod \"maas-controller-6cb557cbc4-rqxpv\" (UID: \"d50f665d-6f6e-436f-b1bc-94bd6aa10e6d\") " pod="opendatahub/maas-controller-6cb557cbc4-rqxpv" Jul 19 15:13:02.182849 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:13:02.182826 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hnf84\" (UniqueName: \"kubernetes.io/projected/d50f665d-6f6e-436f-b1bc-94bd6aa10e6d-kube-api-access-hnf84\") pod \"maas-controller-6cb557cbc4-rqxpv\" (UID: \"d50f665d-6f6e-436f-b1bc-94bd6aa10e6d\") " pod="opendatahub/maas-controller-6cb557cbc4-rqxpv" Jul 19 15:13:02.282386 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:13:02.282313 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6cb557cbc4-rqxpv" Jul 19 15:13:02.404515 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:13:02.404494 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-6cb557cbc4-rqxpv"] Jul 19 15:13:02.407035 ip-10-0-138-220 kubenswrapper[2558]: W0719 15:13:02.406997 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd50f665d_6f6e_436f_b1bc_94bd6aa10e6d.slice/crio-175d25ba7507d6a816046c94a23a7fb6968d268eaf78f928bdf919c5bbea9345 WatchSource:0}: Error finding container 175d25ba7507d6a816046c94a23a7fb6968d268eaf78f928bdf919c5bbea9345: Status 404 returned error can't find the container with id 175d25ba7507d6a816046c94a23a7fb6968d268eaf78f928bdf919c5bbea9345 Jul 19 15:13:03.152616 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:13:03.152583 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6cb557cbc4-rqxpv" event={"ID":"d50f665d-6f6e-436f-b1bc-94bd6aa10e6d","Type":"ContainerStarted","Data":"add5d6087a96a86bacfa8ab5d82d36bed69c80c99e0126efbfc59223c92d00e7"} Jul 19 15:13:03.152772 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:13:03.152632 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6cb557cbc4-rqxpv" event={"ID":"d50f665d-6f6e-436f-b1bc-94bd6aa10e6d","Type":"ContainerStarted","Data":"175d25ba7507d6a816046c94a23a7fb6968d268eaf78f928bdf919c5bbea9345"} Jul 19 15:13:03.152772 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:13:03.152660 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-6cb557cbc4-rqxpv" Jul 19 15:13:03.172143 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:13:03.172104 2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-6cb557cbc4-rqxpv" podStartSLOduration=1.842169765 podStartE2EDuration="2.172089521s" totalImagesPullingTime="329.919756ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-19 15:13:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-19 15:13:03.170827159 +0000 UTC m=+924.092529872" watchObservedRunningTime="2026-07-19 15:13:03.172089521 +0000 UTC m=+924.093792233" Jul 19 15:13:14.158230 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:13:14.158199 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-6cb557cbc4-rqxpv" Jul 19 15:13:24.192124 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:13:24.192093 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-d4lll"] Jul 19 15:13:37.384901 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:13:37.384872 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-d4lll"] Jul 19 15:14:11.387711 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:14:11.387677 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-d4lll"] Jul 19 15:14:20.885819 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:14:20.885777 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-d4lll"] Jul 19 15:14:29.785384 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:14:29.785352 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-d4lll"] Jul 19 15:14:40.882741 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:14:40.882707 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-d4lll"] Jul 19 15:14:52.593415 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:14:52.593387 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-d4lll"] Jul 19 15:15:00.141162 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:15:00.141129 2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29741235-h4dwb"] Jul 19 15:15:00.144403 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:15:00.144385 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29741235-h4dwb" Jul 19 15:15:00.146540 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:15:00.146517 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qddbb\"" Jul 19 15:15:00.153687 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:15:00.153664 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29741235-h4dwb" Jul 19 15:15:00.156740 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:15:00.156711 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29741235-h4dwb"] Jul 19 15:15:00.280106 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:15:00.280079 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29741235-h4dwb"] Jul 19 15:15:00.283260 ip-10-0-138-220 kubenswrapper[2558]: W0719 15:15:00.283230 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode7d2d946_1495_4295_8b66_b64469d617bc.slice/crio-48c1b65775646d9ba080158fd69a5558b3e70cc55e4af5e4dc9b43314cffb489 WatchSource:0}: Error finding container 48c1b65775646d9ba080158fd69a5558b3e70cc55e4af5e4dc9b43314cffb489: Status 404 returned error can't find the container with id 48c1b65775646d9ba080158fd69a5558b3e70cc55e4af5e4dc9b43314cffb489 Jul 19 15:15:00.610498 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:15:00.610466 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29741235-h4dwb" event={"ID":"e7d2d946-1495-4295-8b66-b64469d617bc","Type":"ContainerStarted","Data":"48c1b65775646d9ba080158fd69a5558b3e70cc55e4af5e4dc9b43314cffb489"} Jul 19 15:15:02.293606 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:15:02.293577 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-d4lll"] Jul 19 15:15:03.624167 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:15:03.624076 2558 generic.go:356] "Generic (PLEG): container finished" podID="e7d2d946-1495-4295-8b66-b64469d617bc" containerID="8b48d462e6517bb018e508c4202322c94a9524a6ff0fd0f9b9d703bf84fb9d2a" exitCode=0 Jul 19 15:15:03.624509 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:15:03.624166 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29741235-h4dwb" event={"ID":"e7d2d946-1495-4295-8b66-b64469d617bc","Type":"ContainerDied","Data":"8b48d462e6517bb018e508c4202322c94a9524a6ff0fd0f9b9d703bf84fb9d2a"} Jul 19 15:15:04.758603 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:15:04.758581 2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29741235-h4dwb" Jul 19 15:15:05.634807 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:15:05.634779 2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29741235-h4dwb" Jul 19 15:15:05.634807 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:15:05.634792 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29741235-h4dwb" event={"ID":"e7d2d946-1495-4295-8b66-b64469d617bc","Type":"ContainerDied","Data":"48c1b65775646d9ba080158fd69a5558b3e70cc55e4af5e4dc9b43314cffb489"} Jul 19 15:15:05.634971 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:15:05.634838 2558 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="48c1b65775646d9ba080158fd69a5558b3e70cc55e4af5e4dc9b43314cffb489" Jul 19 15:15:12.982676 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:15:12.982642 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-d4lll"] Jul 19 15:15:22.487657 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:15:22.487625 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-d4lll"] Jul 19 15:15:45.192875 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:15:45.192793 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-d4lll"] Jul 19 15:15:56.787799 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:15:56.787768 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-d4lll"] Jul 19 15:16:20.190459 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:16:20.190423 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-d4lll"] Jul 19 15:16:31.188762 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:16:31.188729 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-d4lll"] Jul 19 15:16:46.200149 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:16:46.200110 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-d4lll"] Jul 19 15:16:57.148060 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:16:57.148006 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-d4lll"] Jul 19 15:17:06.213520 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:17:06.213442 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-d4lll"] Jul 19 15:17:15.700515 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:17:15.700484 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-d4lll"] Jul 19 15:17:25.292961 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:17:25.292924 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-d4lll"] Jul 19 15:17:39.576421 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:17:39.576385 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-5wfzs_5b2d3a07-86b2-4ba2-8a71-c2d89ada79c6/ovn-acl-logging/0.log" Jul 19 15:17:39.576959 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:17:39.576809 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-5wfzs_5b2d3a07-86b2-4ba2-8a71-c2d89ada79c6/ovn-acl-logging/0.log" Jul 19 15:19:14.494768 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:19:14.494730 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-d4lll"] Jul 19 15:19:23.788668 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:19:23.788634 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-d4lll"] Jul 19 15:19:39.190305 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:19:39.190272 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-d4lll"] Jul 19 15:19:49.986625 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:19:49.986582 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-d4lll"] Jul 19 15:20:00.187932 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:20:00.187888 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-d4lll"] Jul 19 15:20:11.582684 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:20:11.582651 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-d4lll"] Jul 19 15:20:20.291382 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:20:20.291348 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-d4lll"] Jul 19 15:20:29.687566 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:20:29.687533 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-d4lll"] Jul 19 15:20:40.391238 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:20:40.391202 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-d4lll"] Jul 19 15:20:50.189869 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:20:50.189833 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-d4lll"] Jul 19 15:21:01.790033 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:21:01.789988 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-d4lll"] Jul 19 15:21:27.382261 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:21:27.382227 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-d4lll"] Jul 19 15:21:34.687498 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:21:34.687422 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-d4lll"] Jul 19 15:21:44.383678 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:21:44.383638 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-d4lll"] Jul 19 15:21:51.991119 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:21:51.991077 2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-b-95a34797-kserve-6958d6545f-mbrjc"] Jul 19 15:21:51.991631 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:21:51.991500 2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e7d2d946-1495-4295-8b66-b64469d617bc" containerName="cleanup" Jul 19 15:21:51.991631 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:21:51.991519 2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="e7d2d946-1495-4295-8b66-b64469d617bc" containerName="cleanup" Jul 19 15:21:51.991631 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:21:51.991605 2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="e7d2d946-1495-4295-8b66-b64469d617bc" containerName="cleanup" Jul 19 15:21:51.994633 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:21:51.994600 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-95a34797-kserve-6958d6545f-mbrjc" Jul 19 15:21:51.997046 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:21:51.997003 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-c7xpt\"" Jul 19 15:21:51.997174 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:21:51.997006 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Jul 19 15:21:51.997848 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:21:51.997828 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Jul 19 15:21:51.997965 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:21:51.997851 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-b-95a34797-kserve-self-signed-certs\"" Jul 19 15:21:52.003602 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:21:52.003573 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-95a34797-kserve-6958d6545f-mbrjc"] Jul 19 15:21:52.052084 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:21:52.052054 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2a28a5a5-3a1e-4e67-a030-e39487ca0be0-dshm\") pod \"e2e-same-id-b-95a34797-kserve-6958d6545f-mbrjc\" (UID: \"2a28a5a5-3a1e-4e67-a030-e39487ca0be0\") " pod="llm/e2e-same-id-b-95a34797-kserve-6958d6545f-mbrjc" Jul 19 15:21:52.052247 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:21:52.052089 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2a28a5a5-3a1e-4e67-a030-e39487ca0be0-tls-certs\") pod \"e2e-same-id-b-95a34797-kserve-6958d6545f-mbrjc\" (UID: \"2a28a5a5-3a1e-4e67-a030-e39487ca0be0\") " pod="llm/e2e-same-id-b-95a34797-kserve-6958d6545f-mbrjc" Jul 19 15:21:52.052247 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:21:52.052124 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2a28a5a5-3a1e-4e67-a030-e39487ca0be0-model-cache\") pod \"e2e-same-id-b-95a34797-kserve-6958d6545f-mbrjc\" (UID: \"2a28a5a5-3a1e-4e67-a030-e39487ca0be0\") " pod="llm/e2e-same-id-b-95a34797-kserve-6958d6545f-mbrjc" Jul 19 15:21:52.052247 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:21:52.052191 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-44fxb\" (UniqueName: \"kubernetes.io/projected/2a28a5a5-3a1e-4e67-a030-e39487ca0be0-kube-api-access-44fxb\") pod \"e2e-same-id-b-95a34797-kserve-6958d6545f-mbrjc\" (UID: \"2a28a5a5-3a1e-4e67-a030-e39487ca0be0\") " pod="llm/e2e-same-id-b-95a34797-kserve-6958d6545f-mbrjc" Jul 19 15:21:52.052247 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:21:52.052230 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2a28a5a5-3a1e-4e67-a030-e39487ca0be0-tmp-dir\") pod \"e2e-same-id-b-95a34797-kserve-6958d6545f-mbrjc\" (UID: \"2a28a5a5-3a1e-4e67-a030-e39487ca0be0\") " pod="llm/e2e-same-id-b-95a34797-kserve-6958d6545f-mbrjc" Jul 19 15:21:52.052374 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:21:52.052258 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2a28a5a5-3a1e-4e67-a030-e39487ca0be0-home\") pod \"e2e-same-id-b-95a34797-kserve-6958d6545f-mbrjc\" (UID: \"2a28a5a5-3a1e-4e67-a030-e39487ca0be0\") " pod="llm/e2e-same-id-b-95a34797-kserve-6958d6545f-mbrjc" Jul 19 15:21:52.152687 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:21:52.152657 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2a28a5a5-3a1e-4e67-a030-e39487ca0be0-tls-certs\") pod \"e2e-same-id-b-95a34797-kserve-6958d6545f-mbrjc\" (UID: \"2a28a5a5-3a1e-4e67-a030-e39487ca0be0\") " pod="llm/e2e-same-id-b-95a34797-kserve-6958d6545f-mbrjc" Jul 19 15:21:52.152833 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:21:52.152703 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2a28a5a5-3a1e-4e67-a030-e39487ca0be0-model-cache\") pod \"e2e-same-id-b-95a34797-kserve-6958d6545f-mbrjc\" (UID: \"2a28a5a5-3a1e-4e67-a030-e39487ca0be0\") " pod="llm/e2e-same-id-b-95a34797-kserve-6958d6545f-mbrjc" Jul 19 15:21:52.152833 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:21:52.152726 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-44fxb\" (UniqueName: \"kubernetes.io/projected/2a28a5a5-3a1e-4e67-a030-e39487ca0be0-kube-api-access-44fxb\") pod \"e2e-same-id-b-95a34797-kserve-6958d6545f-mbrjc\" (UID: \"2a28a5a5-3a1e-4e67-a030-e39487ca0be0\") " pod="llm/e2e-same-id-b-95a34797-kserve-6958d6545f-mbrjc" Jul 19 15:21:52.152833 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:21:52.152752 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2a28a5a5-3a1e-4e67-a030-e39487ca0be0-tmp-dir\") pod \"e2e-same-id-b-95a34797-kserve-6958d6545f-mbrjc\" (UID: \"2a28a5a5-3a1e-4e67-a030-e39487ca0be0\") " pod="llm/e2e-same-id-b-95a34797-kserve-6958d6545f-mbrjc" Jul 19 15:21:52.152833 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:21:52.152790 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2a28a5a5-3a1e-4e67-a030-e39487ca0be0-home\") pod \"e2e-same-id-b-95a34797-kserve-6958d6545f-mbrjc\" (UID: \"2a28a5a5-3a1e-4e67-a030-e39487ca0be0\") " pod="llm/e2e-same-id-b-95a34797-kserve-6958d6545f-mbrjc" Jul 19 15:21:52.152833 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:21:52.152822 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2a28a5a5-3a1e-4e67-a030-e39487ca0be0-dshm\") pod \"e2e-same-id-b-95a34797-kserve-6958d6545f-mbrjc\" (UID: \"2a28a5a5-3a1e-4e67-a030-e39487ca0be0\") " pod="llm/e2e-same-id-b-95a34797-kserve-6958d6545f-mbrjc" Jul 19 15:21:52.153133 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:21:52.153075 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2a28a5a5-3a1e-4e67-a030-e39487ca0be0-model-cache\") pod \"e2e-same-id-b-95a34797-kserve-6958d6545f-mbrjc\" (UID: \"2a28a5a5-3a1e-4e67-a030-e39487ca0be0\") " pod="llm/e2e-same-id-b-95a34797-kserve-6958d6545f-mbrjc" Jul 19 15:21:52.153190 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:21:52.153163 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2a28a5a5-3a1e-4e67-a030-e39487ca0be0-tmp-dir\") pod \"e2e-same-id-b-95a34797-kserve-6958d6545f-mbrjc\" (UID: \"2a28a5a5-3a1e-4e67-a030-e39487ca0be0\") " pod="llm/e2e-same-id-b-95a34797-kserve-6958d6545f-mbrjc" Jul 19 15:21:52.153234 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:21:52.153218 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2a28a5a5-3a1e-4e67-a030-e39487ca0be0-home\") pod \"e2e-same-id-b-95a34797-kserve-6958d6545f-mbrjc\" (UID: \"2a28a5a5-3a1e-4e67-a030-e39487ca0be0\") " pod="llm/e2e-same-id-b-95a34797-kserve-6958d6545f-mbrjc" Jul 19 15:21:52.155200 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:21:52.155179 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2a28a5a5-3a1e-4e67-a030-e39487ca0be0-dshm\") pod \"e2e-same-id-b-95a34797-kserve-6958d6545f-mbrjc\" (UID: \"2a28a5a5-3a1e-4e67-a030-e39487ca0be0\") " pod="llm/e2e-same-id-b-95a34797-kserve-6958d6545f-mbrjc" Jul 19 15:21:52.155310 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:21:52.155298 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2a28a5a5-3a1e-4e67-a030-e39487ca0be0-tls-certs\") pod \"e2e-same-id-b-95a34797-kserve-6958d6545f-mbrjc\" (UID: \"2a28a5a5-3a1e-4e67-a030-e39487ca0be0\") " pod="llm/e2e-same-id-b-95a34797-kserve-6958d6545f-mbrjc" Jul 19 15:21:52.162032 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:21:52.161997 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-44fxb\" (UniqueName: \"kubernetes.io/projected/2a28a5a5-3a1e-4e67-a030-e39487ca0be0-kube-api-access-44fxb\") pod \"e2e-same-id-b-95a34797-kserve-6958d6545f-mbrjc\" (UID: \"2a28a5a5-3a1e-4e67-a030-e39487ca0be0\") " pod="llm/e2e-same-id-b-95a34797-kserve-6958d6545f-mbrjc" Jul 19 15:21:52.306607 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:21:52.306533 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-95a34797-kserve-6958d6545f-mbrjc" Jul 19 15:21:52.432569 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:21:52.432519 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-95a34797-kserve-6958d6545f-mbrjc"] Jul 19 15:21:52.434638 ip-10-0-138-220 kubenswrapper[2558]: W0719 15:21:52.434603 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2a28a5a5_3a1e_4e67_a030_e39487ca0be0.slice/crio-7b29192032c3805bb384a25540ce2afefa3f25409de00ff49a64a48b6654ecc3 WatchSource:0}: Error finding container 7b29192032c3805bb384a25540ce2afefa3f25409de00ff49a64a48b6654ecc3: Status 404 returned error can't find the container with id 7b29192032c3805bb384a25540ce2afefa3f25409de00ff49a64a48b6654ecc3 Jul 19 15:21:52.436486 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:21:52.436472 2558 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 19 15:21:53.216646 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:21:53.216613 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-95a34797-kserve-6958d6545f-mbrjc" event={"ID":"2a28a5a5-3a1e-4e67-a030-e39487ca0be0","Type":"ContainerStarted","Data":"7b29192032c3805bb384a25540ce2afefa3f25409de00ff49a64a48b6654ecc3"} Jul 19 15:21:54.222597 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:21:54.222567 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-95a34797-kserve-6958d6545f-mbrjc" event={"ID":"2a28a5a5-3a1e-4e67-a030-e39487ca0be0","Type":"ContainerStarted","Data":"af3577b08d33ca516e2589b021c344aeb559b9be713a4231dcd1d11996b9afb5"} Jul 19 15:21:54.238445 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:21:54.238405 2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-same-id-b-95a34797-kserve-6958d6545f-mbrjc" podStartSLOduration=2.12638597 podStartE2EDuration="3.238390583s" totalImagesPullingTime="1.112004613s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-19 15:21:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-19 15:21:54.236936627 +0000 UTC m=+1455.158639351" watchObservedRunningTime="2026-07-19 15:21:54.238390583 +0000 UTC m=+1455.160093296" Jul 19 15:21:55.886386 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:21:55.886353 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-d4lll"] Jul 19 15:22:02.307531 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:22:02.307503 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-b-95a34797-kserve-6958d6545f-mbrjc" Jul 19 15:22:02.307944 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:22:02.307549 2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-b-95a34797-kserve-6958d6545f-mbrjc" Jul 19 15:22:02.320285 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:22:02.320249 2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-b-95a34797-kserve-6958d6545f-mbrjc" Jul 19 15:22:03.270633 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:22:03.270599 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-b-95a34797-kserve-6958d6545f-mbrjc" Jul 19 15:22:37.209756 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:22:37.209724 2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-95a34797-kserve-6958d6545f-mbrjc"] Jul 19 15:22:39.597386 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:22:39.597357 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-5wfzs_5b2d3a07-86b2-4ba2-8a71-c2d89ada79c6/ovn-acl-logging/0.log" Jul 19 15:22:39.600003 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:22:39.599983 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-5wfzs_5b2d3a07-86b2-4ba2-8a71-c2d89ada79c6/ovn-acl-logging/0.log" Jul 19 15:22:40.394664 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:22:40.394629 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-d4lll"] Jul 19 15:22:52.230827 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:22:52.230780 2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-same-id-b-95a34797-kserve-6958d6545f-mbrjc" podUID="2a28a5a5-3a1e-4e67-a030-e39487ca0be0" containerName="main" containerID="cri-o://af3577b08d33ca516e2589b021c344aeb559b9be713a4231dcd1d11996b9afb5" gracePeriod=45 Jul 19 15:22:52.457667 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:22:52.457638 2558 generic.go:356] "Generic (PLEG): container finished" podID="2a28a5a5-3a1e-4e67-a030-e39487ca0be0" containerID="af3577b08d33ca516e2589b021c344aeb559b9be713a4231dcd1d11996b9afb5" exitCode=0 Jul 19 15:22:52.457809 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:22:52.457684 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-95a34797-kserve-6958d6545f-mbrjc" event={"ID":"2a28a5a5-3a1e-4e67-a030-e39487ca0be0","Type":"ContainerDied","Data":"af3577b08d33ca516e2589b021c344aeb559b9be713a4231dcd1d11996b9afb5"} Jul 19 15:22:52.482074 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:22:52.482008 2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-95a34797-kserve-6958d6545f-mbrjc" Jul 19 15:22:52.610178 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:22:52.610152 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-44fxb\" (UniqueName: \"kubernetes.io/projected/2a28a5a5-3a1e-4e67-a030-e39487ca0be0-kube-api-access-44fxb\") pod \"2a28a5a5-3a1e-4e67-a030-e39487ca0be0\" (UID: \"2a28a5a5-3a1e-4e67-a030-e39487ca0be0\") " Jul 19 15:22:52.610334 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:22:52.610192 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2a28a5a5-3a1e-4e67-a030-e39487ca0be0-home\") pod \"2a28a5a5-3a1e-4e67-a030-e39487ca0be0\" (UID: \"2a28a5a5-3a1e-4e67-a030-e39487ca0be0\") " Jul 19 15:22:52.610334 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:22:52.610216 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2a28a5a5-3a1e-4e67-a030-e39487ca0be0-dshm\") pod \"2a28a5a5-3a1e-4e67-a030-e39487ca0be0\" (UID: \"2a28a5a5-3a1e-4e67-a030-e39487ca0be0\") " Jul 19 15:22:52.610334 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:22:52.610230 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2a28a5a5-3a1e-4e67-a030-e39487ca0be0-model-cache\") pod \"2a28a5a5-3a1e-4e67-a030-e39487ca0be0\" (UID: \"2a28a5a5-3a1e-4e67-a030-e39487ca0be0\") " Jul 19 15:22:52.610334 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:22:52.610246 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2a28a5a5-3a1e-4e67-a030-e39487ca0be0-tls-certs\") pod \"2a28a5a5-3a1e-4e67-a030-e39487ca0be0\" (UID: \"2a28a5a5-3a1e-4e67-a030-e39487ca0be0\") " Jul 19 15:22:52.610334 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:22:52.610275 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2a28a5a5-3a1e-4e67-a030-e39487ca0be0-tmp-dir\") pod \"2a28a5a5-3a1e-4e67-a030-e39487ca0be0\" (UID: \"2a28a5a5-3a1e-4e67-a030-e39487ca0be0\") " Jul 19 15:22:52.610606 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:22:52.610481 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a28a5a5-3a1e-4e67-a030-e39487ca0be0-home" (OuterVolumeSpecName: "home") pod "2a28a5a5-3a1e-4e67-a030-e39487ca0be0" (UID: "2a28a5a5-3a1e-4e67-a030-e39487ca0be0"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 19 15:22:52.610606 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:22:52.610487 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a28a5a5-3a1e-4e67-a030-e39487ca0be0-model-cache" (OuterVolumeSpecName: "model-cache") pod "2a28a5a5-3a1e-4e67-a030-e39487ca0be0" (UID: "2a28a5a5-3a1e-4e67-a030-e39487ca0be0"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 19 15:22:52.610731 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:22:52.610699 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a28a5a5-3a1e-4e67-a030-e39487ca0be0-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "2a28a5a5-3a1e-4e67-a030-e39487ca0be0" (UID: "2a28a5a5-3a1e-4e67-a030-e39487ca0be0"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 19 15:22:52.612764 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:22:52.612653 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2a28a5a5-3a1e-4e67-a030-e39487ca0be0-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "2a28a5a5-3a1e-4e67-a030-e39487ca0be0" (UID: "2a28a5a5-3a1e-4e67-a030-e39487ca0be0"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 19 15:22:52.612764 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:22:52.612736 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a28a5a5-3a1e-4e67-a030-e39487ca0be0-dshm" (OuterVolumeSpecName: "dshm") pod "2a28a5a5-3a1e-4e67-a030-e39487ca0be0" (UID: "2a28a5a5-3a1e-4e67-a030-e39487ca0be0"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 19 15:22:52.612958 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:22:52.612767 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2a28a5a5-3a1e-4e67-a030-e39487ca0be0-kube-api-access-44fxb" (OuterVolumeSpecName: "kube-api-access-44fxb") pod "2a28a5a5-3a1e-4e67-a030-e39487ca0be0" (UID: "2a28a5a5-3a1e-4e67-a030-e39487ca0be0"). InnerVolumeSpecName "kube-api-access-44fxb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 19 15:22:52.711250 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:22:52.711219 2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-44fxb\" (UniqueName: \"kubernetes.io/projected/2a28a5a5-3a1e-4e67-a030-e39487ca0be0-kube-api-access-44fxb\") on node \"ip-10-0-138-220.ec2.internal\" DevicePath \"\"" Jul 19 15:22:52.711250 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:22:52.711246 2558 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2a28a5a5-3a1e-4e67-a030-e39487ca0be0-home\") on node \"ip-10-0-138-220.ec2.internal\" DevicePath \"\"" Jul 19 15:22:52.711250 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:22:52.711257 2558 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2a28a5a5-3a1e-4e67-a030-e39487ca0be0-dshm\") on node \"ip-10-0-138-220.ec2.internal\" DevicePath \"\"" Jul 19 15:22:52.711452 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:22:52.711266 2558 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2a28a5a5-3a1e-4e67-a030-e39487ca0be0-model-cache\") on node \"ip-10-0-138-220.ec2.internal\" DevicePath \"\"" Jul 19 15:22:52.711452 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:22:52.711275 2558 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2a28a5a5-3a1e-4e67-a030-e39487ca0be0-tls-certs\") on node \"ip-10-0-138-220.ec2.internal\" DevicePath \"\"" Jul 19 15:22:52.711452 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:22:52.711311 2558 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2a28a5a5-3a1e-4e67-a030-e39487ca0be0-tmp-dir\") on node \"ip-10-0-138-220.ec2.internal\" DevicePath \"\"" Jul 19 15:22:53.463856 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:22:53.463823 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-95a34797-kserve-6958d6545f-mbrjc" event={"ID":"2a28a5a5-3a1e-4e67-a030-e39487ca0be0","Type":"ContainerDied","Data":"7b29192032c3805bb384a25540ce2afefa3f25409de00ff49a64a48b6654ecc3"} Jul 19 15:22:53.463856 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:22:53.463858 2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-95a34797-kserve-6958d6545f-mbrjc" Jul 19 15:22:53.464343 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:22:53.463873 2558 scope.go:117] "RemoveContainer" containerID="af3577b08d33ca516e2589b021c344aeb559b9be713a4231dcd1d11996b9afb5" Jul 19 15:22:53.484056 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:22:53.484010 2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-95a34797-kserve-6958d6545f-mbrjc"] Jul 19 15:22:53.487746 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:22:53.487724 2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-b-95a34797-kserve-6958d6545f-mbrjc"] Jul 19 15:22:53.565397 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:22:53.565368 2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2a28a5a5-3a1e-4e67-a030-e39487ca0be0" path="/var/lib/kubelet/pods/2a28a5a5-3a1e-4e67-a030-e39487ca0be0/volumes" Jul 19 15:22:54.482295 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:22:54.482263 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-d4lll"] Jul 19 15:23:04.484761 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:23:04.484687 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-d4lll"] Jul 19 15:23:16.984058 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:23:16.984011 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-d4lll"] Jul 19 15:23:28.386606 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:23:28.386571 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-d4lll"] Jul 19 15:23:41.191244 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:23:41.191210 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-d4lll"] Jul 19 15:23:55.082978 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:23:55.082946 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-d4lll"] Jul 19 15:24:02.590829 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:24:02.590797 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-d4lll"] Jul 19 15:24:11.990452 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:24:11.990414 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-d4lll"] Jul 19 15:24:22.494324 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:24:22.494285 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-d4lll"] Jul 19 15:24:33.992400 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:24:33.992308 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-d4lll"] Jul 19 15:24:44.098150 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:24:44.098110 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-d4lll"] Jul 19 15:24:56.600178 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:24:56.600139 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-d4lll"] Jul 19 15:25:07.786846 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:25:07.786810 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-d4lll"] Jul 19 15:25:16.914470 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:25:16.914435 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-d4lll"] Jul 19 15:25:29.783725 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:25:29.783683 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-d4lll"] Jul 19 15:25:41.986608 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:25:41.986565 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-d4lll"] Jul 19 15:25:54.490312 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:25:54.490280 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-d4lll"] Jul 19 15:26:03.587204 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:26:03.587120 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-d4lll"] Jul 19 15:26:25.091721 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:26:25.091684 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-d4lll"] Jul 19 15:26:30.474684 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:26:30.474652 2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-44de5c81-openshift-default-74f765c8c5-hzzw9"] Jul 19 15:26:30.475144 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:26:30.474920 2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2a28a5a5-3a1e-4e67-a030-e39487ca0be0" containerName="main" Jul 19 15:26:30.475144 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:26:30.474932 2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="2a28a5a5-3a1e-4e67-a030-e39487ca0be0" containerName="main" Jul 19 15:26:30.475144 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:26:30.474991 2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="2a28a5a5-3a1e-4e67-a030-e39487ca0be0" containerName="main" Jul 19 15:26:30.477820 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:26:30.477804 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-44de5c81-openshift-default-74f765c8c5-hzzw9" Jul 19 15:26:30.479966 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:26:30.479947 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-44de5c81-openshift-default-dockercfg-8w7m5\"" Jul 19 15:26:30.497875 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:26:30.497850 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-44de5c81-openshift-default-74f765c8c5-hzzw9"] Jul 19 15:26:30.573642 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:26:30.573613 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ec616fc0-ba34-47b2-8423-93da5fe6e436-istio-data\") pod \"e2e-ait-44de5c81-openshift-default-74f765c8c5-hzzw9\" (UID: \"ec616fc0-ba34-47b2-8423-93da5fe6e436\") " pod="openshift-ingress/e2e-ait-44de5c81-openshift-default-74f765c8c5-hzzw9" Jul 19 15:26:30.573778 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:26:30.573649 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ec616fc0-ba34-47b2-8423-93da5fe6e436-istio-envoy\") pod \"e2e-ait-44de5c81-openshift-default-74f765c8c5-hzzw9\" (UID: \"ec616fc0-ba34-47b2-8423-93da5fe6e436\") " pod="openshift-ingress/e2e-ait-44de5c81-openshift-default-74f765c8c5-hzzw9" Jul 19 15:26:30.573778 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:26:30.573674 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zl64q\" (UniqueName: \"kubernetes.io/projected/ec616fc0-ba34-47b2-8423-93da5fe6e436-kube-api-access-zl64q\") pod \"e2e-ait-44de5c81-openshift-default-74f765c8c5-hzzw9\" (UID: \"ec616fc0-ba34-47b2-8423-93da5fe6e436\") " pod="openshift-ingress/e2e-ait-44de5c81-openshift-default-74f765c8c5-hzzw9" Jul 19 15:26:30.573778 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:26:30.573726 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ec616fc0-ba34-47b2-8423-93da5fe6e436-istio-token\") pod \"e2e-ait-44de5c81-openshift-default-74f765c8c5-hzzw9\" (UID: \"ec616fc0-ba34-47b2-8423-93da5fe6e436\") " pod="openshift-ingress/e2e-ait-44de5c81-openshift-default-74f765c8c5-hzzw9" Jul 19 15:26:30.573778 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:26:30.573749 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec616fc0-ba34-47b2-8423-93da5fe6e436-workload-socket\") pod \"e2e-ait-44de5c81-openshift-default-74f765c8c5-hzzw9\" (UID: \"ec616fc0-ba34-47b2-8423-93da5fe6e436\") " pod="openshift-ingress/e2e-ait-44de5c81-openshift-default-74f765c8c5-hzzw9" Jul 19 15:26:30.573778 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:26:30.573768 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ec616fc0-ba34-47b2-8423-93da5fe6e436-istio-podinfo\") pod \"e2e-ait-44de5c81-openshift-default-74f765c8c5-hzzw9\" (UID: \"ec616fc0-ba34-47b2-8423-93da5fe6e436\") " pod="openshift-ingress/e2e-ait-44de5c81-openshift-default-74f765c8c5-hzzw9" Jul 19 15:26:30.574116 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:26:30.573849 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ec616fc0-ba34-47b2-8423-93da5fe6e436-istiod-ca-cert\") pod \"e2e-ait-44de5c81-openshift-default-74f765c8c5-hzzw9\" (UID: \"ec616fc0-ba34-47b2-8423-93da5fe6e436\") " pod="openshift-ingress/e2e-ait-44de5c81-openshift-default-74f765c8c5-hzzw9" Jul 19 15:26:30.574116 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:26:30.573896 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec616fc0-ba34-47b2-8423-93da5fe6e436-credential-socket\") pod \"e2e-ait-44de5c81-openshift-default-74f765c8c5-hzzw9\" (UID: \"ec616fc0-ba34-47b2-8423-93da5fe6e436\") " pod="openshift-ingress/e2e-ait-44de5c81-openshift-default-74f765c8c5-hzzw9" Jul 19 15:26:30.574116 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:26:30.573922 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ec616fc0-ba34-47b2-8423-93da5fe6e436-workload-certs\") pod \"e2e-ait-44de5c81-openshift-default-74f765c8c5-hzzw9\" (UID: \"ec616fc0-ba34-47b2-8423-93da5fe6e436\") " pod="openshift-ingress/e2e-ait-44de5c81-openshift-default-74f765c8c5-hzzw9" Jul 19 15:26:30.675129 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:26:30.675092 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zl64q\" (UniqueName: \"kubernetes.io/projected/ec616fc0-ba34-47b2-8423-93da5fe6e436-kube-api-access-zl64q\") pod \"e2e-ait-44de5c81-openshift-default-74f765c8c5-hzzw9\" (UID: \"ec616fc0-ba34-47b2-8423-93da5fe6e436\") " pod="openshift-ingress/e2e-ait-44de5c81-openshift-default-74f765c8c5-hzzw9" Jul 19 15:26:30.675308 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:26:30.675141 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ec616fc0-ba34-47b2-8423-93da5fe6e436-istio-token\") pod \"e2e-ait-44de5c81-openshift-default-74f765c8c5-hzzw9\" (UID: \"ec616fc0-ba34-47b2-8423-93da5fe6e436\") " pod="openshift-ingress/e2e-ait-44de5c81-openshift-default-74f765c8c5-hzzw9" Jul 19 15:26:30.675308 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:26:30.675163 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec616fc0-ba34-47b2-8423-93da5fe6e436-workload-socket\") pod \"e2e-ait-44de5c81-openshift-default-74f765c8c5-hzzw9\" (UID: \"ec616fc0-ba34-47b2-8423-93da5fe6e436\") " pod="openshift-ingress/e2e-ait-44de5c81-openshift-default-74f765c8c5-hzzw9" Jul 19 15:26:30.675308 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:26:30.675190 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ec616fc0-ba34-47b2-8423-93da5fe6e436-istio-podinfo\") pod \"e2e-ait-44de5c81-openshift-default-74f765c8c5-hzzw9\" (UID: \"ec616fc0-ba34-47b2-8423-93da5fe6e436\") " pod="openshift-ingress/e2e-ait-44de5c81-openshift-default-74f765c8c5-hzzw9" Jul 19 15:26:30.675308 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:26:30.675228 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ec616fc0-ba34-47b2-8423-93da5fe6e436-istiod-ca-cert\") pod \"e2e-ait-44de5c81-openshift-default-74f765c8c5-hzzw9\" (UID: \"ec616fc0-ba34-47b2-8423-93da5fe6e436\") " pod="openshift-ingress/e2e-ait-44de5c81-openshift-default-74f765c8c5-hzzw9" Jul 19 15:26:30.675308 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:26:30.675258 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec616fc0-ba34-47b2-8423-93da5fe6e436-credential-socket\") pod \"e2e-ait-44de5c81-openshift-default-74f765c8c5-hzzw9\" (UID: \"ec616fc0-ba34-47b2-8423-93da5fe6e436\") " pod="openshift-ingress/e2e-ait-44de5c81-openshift-default-74f765c8c5-hzzw9" Jul 19 15:26:30.675308 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:26:30.675280 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ec616fc0-ba34-47b2-8423-93da5fe6e436-workload-certs\") pod \"e2e-ait-44de5c81-openshift-default-74f765c8c5-hzzw9\" (UID: \"ec616fc0-ba34-47b2-8423-93da5fe6e436\") " pod="openshift-ingress/e2e-ait-44de5c81-openshift-default-74f765c8c5-hzzw9" Jul 19 15:26:30.675308 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:26:30.675306 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ec616fc0-ba34-47b2-8423-93da5fe6e436-istio-data\") pod \"e2e-ait-44de5c81-openshift-default-74f765c8c5-hzzw9\" (UID: \"ec616fc0-ba34-47b2-8423-93da5fe6e436\") " pod="openshift-ingress/e2e-ait-44de5c81-openshift-default-74f765c8c5-hzzw9" Jul 19 15:26:30.675727 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:26:30.675347 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ec616fc0-ba34-47b2-8423-93da5fe6e436-istio-envoy\") pod \"e2e-ait-44de5c81-openshift-default-74f765c8c5-hzzw9\" (UID: \"ec616fc0-ba34-47b2-8423-93da5fe6e436\") " pod="openshift-ingress/e2e-ait-44de5c81-openshift-default-74f765c8c5-hzzw9" Jul 19 15:26:30.675727 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:26:30.675552 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec616fc0-ba34-47b2-8423-93da5fe6e436-workload-socket\") pod \"e2e-ait-44de5c81-openshift-default-74f765c8c5-hzzw9\" (UID: \"ec616fc0-ba34-47b2-8423-93da5fe6e436\") " pod="openshift-ingress/e2e-ait-44de5c81-openshift-default-74f765c8c5-hzzw9" Jul 19 15:26:30.675727 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:26:30.675663 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ec616fc0-ba34-47b2-8423-93da5fe6e436-workload-certs\") pod \"e2e-ait-44de5c81-openshift-default-74f765c8c5-hzzw9\" (UID: \"ec616fc0-ba34-47b2-8423-93da5fe6e436\") " pod="openshift-ingress/e2e-ait-44de5c81-openshift-default-74f765c8c5-hzzw9" Jul 19 15:26:30.675727 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:26:30.675687 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec616fc0-ba34-47b2-8423-93da5fe6e436-credential-socket\") pod \"e2e-ait-44de5c81-openshift-default-74f765c8c5-hzzw9\" (UID: \"ec616fc0-ba34-47b2-8423-93da5fe6e436\") " pod="openshift-ingress/e2e-ait-44de5c81-openshift-default-74f765c8c5-hzzw9" Jul 19 15:26:30.675902 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:26:30.675774 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ec616fc0-ba34-47b2-8423-93da5fe6e436-istio-data\") pod \"e2e-ait-44de5c81-openshift-default-74f765c8c5-hzzw9\" (UID: \"ec616fc0-ba34-47b2-8423-93da5fe6e436\") " pod="openshift-ingress/e2e-ait-44de5c81-openshift-default-74f765c8c5-hzzw9" Jul 19 15:26:30.675974 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:26:30.675956 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ec616fc0-ba34-47b2-8423-93da5fe6e436-istiod-ca-cert\") pod \"e2e-ait-44de5c81-openshift-default-74f765c8c5-hzzw9\" (UID: \"ec616fc0-ba34-47b2-8423-93da5fe6e436\") " pod="openshift-ingress/e2e-ait-44de5c81-openshift-default-74f765c8c5-hzzw9" Jul 19 15:26:30.677727 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:26:30.677702 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ec616fc0-ba34-47b2-8423-93da5fe6e436-istio-envoy\") pod \"e2e-ait-44de5c81-openshift-default-74f765c8c5-hzzw9\" (UID: \"ec616fc0-ba34-47b2-8423-93da5fe6e436\") " pod="openshift-ingress/e2e-ait-44de5c81-openshift-default-74f765c8c5-hzzw9" Jul 19 15:26:30.677927 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:26:30.677906 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ec616fc0-ba34-47b2-8423-93da5fe6e436-istio-podinfo\") pod \"e2e-ait-44de5c81-openshift-default-74f765c8c5-hzzw9\" (UID: \"ec616fc0-ba34-47b2-8423-93da5fe6e436\") " pod="openshift-ingress/e2e-ait-44de5c81-openshift-default-74f765c8c5-hzzw9" Jul 19 15:26:30.682707 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:26:30.682684 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ec616fc0-ba34-47b2-8423-93da5fe6e436-istio-token\") pod \"e2e-ait-44de5c81-openshift-default-74f765c8c5-hzzw9\" (UID: \"ec616fc0-ba34-47b2-8423-93da5fe6e436\") " pod="openshift-ingress/e2e-ait-44de5c81-openshift-default-74f765c8c5-hzzw9" Jul 19 15:26:30.682868 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:26:30.682851 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zl64q\" (UniqueName: \"kubernetes.io/projected/ec616fc0-ba34-47b2-8423-93da5fe6e436-kube-api-access-zl64q\") pod \"e2e-ait-44de5c81-openshift-default-74f765c8c5-hzzw9\" (UID: \"ec616fc0-ba34-47b2-8423-93da5fe6e436\") " pod="openshift-ingress/e2e-ait-44de5c81-openshift-default-74f765c8c5-hzzw9" Jul 19 15:26:30.788180 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:26:30.788121 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-44de5c81-openshift-default-74f765c8c5-hzzw9" Jul 19 15:26:30.918142 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:26:30.918119 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-44de5c81-openshift-default-74f765c8c5-hzzw9"] Jul 19 15:26:30.920278 ip-10-0-138-220 kubenswrapper[2558]: W0719 15:26:30.920253 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podec616fc0_ba34_47b2_8423_93da5fe6e436.slice/crio-619b4416d0a0e4703c419da08053586f9149a7e7747a5c919a07eedafbede4e7 WatchSource:0}: Error finding container 619b4416d0a0e4703c419da08053586f9149a7e7747a5c919a07eedafbede4e7: Status 404 returned error can't find the container with id 619b4416d0a0e4703c419da08053586f9149a7e7747a5c919a07eedafbede4e7 Jul 19 15:26:31.316974 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:26:31.316939 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-44de5c81-openshift-default-74f765c8c5-hzzw9" event={"ID":"ec616fc0-ba34-47b2-8423-93da5fe6e436","Type":"ContainerStarted","Data":"619b4416d0a0e4703c419da08053586f9149a7e7747a5c919a07eedafbede4e7"} Jul 19 15:26:33.525825 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:26:33.525792 2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 19 15:26:33.526096 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:26:33.525863 2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 19 15:26:33.526096 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:26:33.525888 2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 19 15:26:34.331303 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:26:34.331263 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-44de5c81-openshift-default-74f765c8c5-hzzw9" event={"ID":"ec616fc0-ba34-47b2-8423-93da5fe6e436","Type":"ContainerStarted","Data":"c2107aec3d973065c1b22b6fbdaa1d9a3a1a200a67a3e251696bf8e158c15fbc"} Jul 19 15:26:34.788571 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:26:34.788533 2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-44de5c81-openshift-default-74f765c8c5-hzzw9" Jul 19 15:26:34.793574 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:26:34.793550 2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-44de5c81-openshift-default-74f765c8c5-hzzw9" Jul 19 15:26:34.813780 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:26:34.813726 2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-44de5c81-openshift-default-74f765c8c5-hzzw9" podStartSLOduration=2.210917542 podStartE2EDuration="4.813710363s" totalImagesPullingTime="2.602792821s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-19 15:26:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-19 15:26:34.352961915 +0000 UTC m=+1735.274664641" watchObservedRunningTime="2026-07-19 15:26:34.813710363 +0000 UTC m=+1735.735413077" Jul 19 15:26:35.335969 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:26:35.335939 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-44de5c81-openshift-default-74f765c8c5-hzzw9" Jul 19 15:26:35.336837 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:26:35.336818 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-44de5c81-openshift-default-74f765c8c5-hzzw9" Jul 19 15:26:39.792470 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:26:39.792436 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-d4lll"] Jul 19 15:26:48.400081 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:26:48.400045 2558 generic.go:356] "Generic (PLEG): container finished" podID="ec616fc0-ba34-47b2-8423-93da5fe6e436" containerID="c2107aec3d973065c1b22b6fbdaa1d9a3a1a200a67a3e251696bf8e158c15fbc" exitCode=0 Jul 19 15:26:48.400490 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:26:48.400096 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-44de5c81-openshift-default-74f765c8c5-hzzw9" event={"ID":"ec616fc0-ba34-47b2-8423-93da5fe6e436","Type":"ContainerDied","Data":"c2107aec3d973065c1b22b6fbdaa1d9a3a1a200a67a3e251696bf8e158c15fbc"} Jul 19 15:26:48.400490 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:26:48.400435 2558 scope.go:117] "RemoveContainer" containerID="c2107aec3d973065c1b22b6fbdaa1d9a3a1a200a67a3e251696bf8e158c15fbc" Jul 19 15:26:48.401375 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:26:48.401343 2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 19 15:26:48.401483 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:26:48.401415 2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 19 15:26:48.401483 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:26:48.401465 2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 19 15:26:49.406415 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:26:49.406381 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-44de5c81-openshift-default-74f765c8c5-hzzw9" event={"ID":"ec616fc0-ba34-47b2-8423-93da5fe6e436","Type":"ContainerStarted","Data":"1bdf35f203f012389fa3fb28bfc114f68c441af24d77af13fc83de6316faa73c"} Jul 19 15:26:49.789334 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:26:49.789238 2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-44de5c81-openshift-default-74f765c8c5-hzzw9" Jul 19 15:26:49.790362 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:26:49.790336 2558 patch_prober.go:28] interesting pod/e2e-ait-44de5c81-openshift-default-74f765c8c5-hzzw9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.24:15021/healthz/ready\": dial tcp 10.133.0.24:15021: connect: connection refused" start-of-body= Jul 19 15:26:49.790471 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:26:49.790394 2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-44de5c81-openshift-default-74f765c8c5-hzzw9" podUID="ec616fc0-ba34-47b2-8423-93da5fe6e436" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.24:15021/healthz/ready\": dial tcp 10.133.0.24:15021: connect: connection refused" Jul 19 15:26:50.336786 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:26:50.336753 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-44de5c81-openshift-default-74f765c8c5-hzzw9" Jul 19 15:26:50.789195 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:26:50.789155 2558 patch_prober.go:28] interesting pod/e2e-ait-44de5c81-openshift-default-74f765c8c5-hzzw9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.24:15021/healthz/ready\": dial tcp 10.133.0.24:15021: connect: connection refused" start-of-body= Jul 19 15:26:50.789577 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:26:50.789209 2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-44de5c81-openshift-default-74f765c8c5-hzzw9" podUID="ec616fc0-ba34-47b2-8423-93da5fe6e436" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.24:15021/healthz/ready\": dial tcp 10.133.0.24:15021: connect: connection refused" Jul 19 15:26:51.789242 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:26:51.789206 2558 patch_prober.go:28] interesting pod/e2e-ait-44de5c81-openshift-default-74f765c8c5-hzzw9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.24:15021/healthz/ready\": dial tcp 10.133.0.24:15021: connect: connection refused" start-of-body= Jul 19 15:26:51.789645 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:26:51.789258 2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-44de5c81-openshift-default-74f765c8c5-hzzw9" podUID="ec616fc0-ba34-47b2-8423-93da5fe6e436" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.24:15021/healthz/ready\": dial tcp 10.133.0.24:15021: connect: connection refused" Jul 19 15:26:52.788501 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:26:52.788459 2558 patch_prober.go:28] interesting pod/e2e-ait-44de5c81-openshift-default-74f765c8c5-hzzw9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.24:15021/healthz/ready\": dial tcp 10.133.0.24:15021: connect: connection refused" start-of-body= Jul 19 15:26:52.788704 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:26:52.788522 2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-44de5c81-openshift-default-74f765c8c5-hzzw9" podUID="ec616fc0-ba34-47b2-8423-93da5fe6e436" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.24:15021/healthz/ready\": dial tcp 10.133.0.24:15021: connect: connection refused" Jul 19 15:26:53.792547 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:26:53.792519 2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-44de5c81-openshift-default-74f765c8c5-hzzw9" Jul 19 15:26:53.793635 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:26:53.793617 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-44de5c81-openshift-default-74f765c8c5-hzzw9" Jul 19 15:27:01.266859 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:27:01.266817 2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-44de5c81-jqhq9"] Jul 19 15:27:01.273153 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:27:01.273130 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-44de5c81-jqhq9" Jul 19 15:27:01.275594 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:27:01.275564 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 19 15:27:01.275594 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:27:01.275588 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qddbb\"" Jul 19 15:27:01.276638 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:27:01.276612 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-44de5c81-jqhq9"] Jul 19 15:27:01.320099 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:27:01.320074 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1fcf6682-0d35-462c-b92d-247f5244b4a9-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-44de5c81-jqhq9\" (UID: \"1fcf6682-0d35-462c-b92d-247f5244b4a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-44de5c81-jqhq9" Jul 19 15:27:01.420635 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:27:01.420597 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1fcf6682-0d35-462c-b92d-247f5244b4a9-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-44de5c81-jqhq9\" (UID: \"1fcf6682-0d35-462c-b92d-247f5244b4a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-44de5c81-jqhq9" Jul 19 15:27:01.421283 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:27:01.421264 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1fcf6682-0d35-462c-b92d-247f5244b4a9-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-44de5c81-jqhq9\" (UID: \"1fcf6682-0d35-462c-b92d-247f5244b4a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-44de5c81-jqhq9" Jul 19 15:27:01.584756 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:27:01.584680 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-44de5c81-jqhq9" Jul 19 15:27:01.711654 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:27:01.711628 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-44de5c81-jqhq9"] Jul 19 15:27:01.714645 ip-10-0-138-220 kubenswrapper[2558]: W0719 15:27:01.714610 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1fcf6682_0d35_462c_b92d_247f5244b4a9.slice/crio-5dc7f3b1b30cab64095baaad117824ff157f6b7aa24f5859ff8678b8323dfa5b WatchSource:0}: Error finding container 5dc7f3b1b30cab64095baaad117824ff157f6b7aa24f5859ff8678b8323dfa5b: Status 404 returned error can't find the container with id 5dc7f3b1b30cab64095baaad117824ff157f6b7aa24f5859ff8678b8323dfa5b Jul 19 15:27:02.462597 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:27:02.462567 2558 generic.go:356] "Generic (PLEG): container finished" podID="1fcf6682-0d35-462c-b92d-247f5244b4a9" containerID="ce5644aa1e75e49edc3bc2c423ee731231c8cb65ec734ec51c8e1c733573046a" exitCode=0 Jul 19 15:27:02.462939 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:27:02.462646 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-44de5c81-jqhq9" event={"ID":"1fcf6682-0d35-462c-b92d-247f5244b4a9","Type":"ContainerDied","Data":"ce5644aa1e75e49edc3bc2c423ee731231c8cb65ec734ec51c8e1c733573046a"} Jul 19 15:27:02.462939 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:27:02.462701 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-44de5c81-jqhq9" event={"ID":"1fcf6682-0d35-462c-b92d-247f5244b4a9","Type":"ContainerStarted","Data":"5dc7f3b1b30cab64095baaad117824ff157f6b7aa24f5859ff8678b8323dfa5b"} Jul 19 15:27:03.595923 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:27:03.595900 2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-44de5c81-jqhq9" Jul 19 15:27:03.738302 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:27:03.738220 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1fcf6682-0d35-462c-b92d-247f5244b4a9-maas-api-service-ca\") pod \"1fcf6682-0d35-462c-b92d-247f5244b4a9\" (UID: \"1fcf6682-0d35-462c-b92d-247f5244b4a9\") " Jul 19 15:27:03.738578 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:27:03.738554 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1fcf6682-0d35-462c-b92d-247f5244b4a9-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "1fcf6682-0d35-462c-b92d-247f5244b4a9" (UID: "1fcf6682-0d35-462c-b92d-247f5244b4a9"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 19 15:27:03.838944 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:27:03.838909 2558 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1fcf6682-0d35-462c-b92d-247f5244b4a9-maas-api-service-ca\") on node \"ip-10-0-138-220.ec2.internal\" DevicePath \"\"" Jul 19 15:27:04.472180 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:27:04.472143 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-44de5c81-jqhq9" event={"ID":"1fcf6682-0d35-462c-b92d-247f5244b4a9","Type":"ContainerDied","Data":"5dc7f3b1b30cab64095baaad117824ff157f6b7aa24f5859ff8678b8323dfa5b"} Jul 19 15:27:04.472344 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:27:04.472192 2558 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5dc7f3b1b30cab64095baaad117824ff157f6b7aa24f5859ff8678b8323dfa5b" Jul 19 15:27:04.472344 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:27:04.472158 2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-44de5c81-jqhq9" Jul 19 15:27:14.882962 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:27:14.882925 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-d4lll"] Jul 19 15:27:16.350166 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:27:16.350129 2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-44de5c81-jqhq9"] Jul 19 15:27:16.352998 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:27:16.352973 2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-44de5c81-jqhq9"] Jul 19 15:27:16.479356 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:27:16.479324 2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-44de5c81-openshift-default-74f765c8c5-hzzw9"] Jul 19 15:27:16.479630 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:27:16.479604 2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-44de5c81-openshift-default-74f765c8c5-hzzw9" podUID="ec616fc0-ba34-47b2-8423-93da5fe6e436" containerName="istio-proxy" containerID="cri-o://1bdf35f203f012389fa3fb28bfc114f68c441af24d77af13fc83de6316faa73c" gracePeriod=30 Jul 19 15:27:17.517734 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:27:17.517701 2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-migrate-431757e3-gw-openshift-default-7df98f6f75-x2kzr"] Jul 19 15:27:17.518207 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:27:17.518136 2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1fcf6682-0d35-462c-b92d-247f5244b4a9" containerName="revoke-keys" Jul 19 15:27:17.518207 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:27:17.518153 2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="1fcf6682-0d35-462c-b92d-247f5244b4a9" containerName="revoke-keys" Jul 19 15:27:17.518319 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:27:17.518242 2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="1fcf6682-0d35-462c-b92d-247f5244b4a9" containerName="revoke-keys" Jul 19 15:27:17.521055 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:27:17.521032 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-431757e3-gw-openshift-default-7df98f6f75-x2kzr" Jul 19 15:27:17.523251 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:27:17.523229 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-migrate-431757e3-gw-openshift-default-dockercfg-px74d\"" Jul 19 15:27:17.534541 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:27:17.534520 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-431757e3-gw-openshift-default-7df98f6f75-x2kzr"] Jul 19 15:27:17.565088 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:27:17.565059 2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1fcf6682-0d35-462c-b92d-247f5244b4a9" path="/var/lib/kubelet/pods/1fcf6682-0d35-462c-b92d-247f5244b4a9/volumes" Jul 19 15:27:17.641914 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:27:17.641877 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c3a47b97-4b42-4be7-a5ef-c33bd3f0e27c-istio-podinfo\") pod \"e2e-migrate-431757e3-gw-openshift-default-7df98f6f75-x2kzr\" (UID: \"c3a47b97-4b42-4be7-a5ef-c33bd3f0e27c\") " pod="openshift-ingress/e2e-migrate-431757e3-gw-openshift-default-7df98f6f75-x2kzr" Jul 19 15:27:17.641914 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:27:17.641916 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c3a47b97-4b42-4be7-a5ef-c33bd3f0e27c-istio-data\") pod \"e2e-migrate-431757e3-gw-openshift-default-7df98f6f75-x2kzr\" (UID: \"c3a47b97-4b42-4be7-a5ef-c33bd3f0e27c\") " pod="openshift-ingress/e2e-migrate-431757e3-gw-openshift-default-7df98f6f75-x2kzr" Jul 19 15:27:17.642123 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:27:17.641933 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c3a47b97-4b42-4be7-a5ef-c33bd3f0e27c-istiod-ca-cert\") pod \"e2e-migrate-431757e3-gw-openshift-default-7df98f6f75-x2kzr\" (UID: \"c3a47b97-4b42-4be7-a5ef-c33bd3f0e27c\") " pod="openshift-ingress/e2e-migrate-431757e3-gw-openshift-default-7df98f6f75-x2kzr" Jul 19 15:27:17.642123 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:27:17.642012 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c3a47b97-4b42-4be7-a5ef-c33bd3f0e27c-istio-token\") pod \"e2e-migrate-431757e3-gw-openshift-default-7df98f6f75-x2kzr\" (UID: \"c3a47b97-4b42-4be7-a5ef-c33bd3f0e27c\") " pod="openshift-ingress/e2e-migrate-431757e3-gw-openshift-default-7df98f6f75-x2kzr" Jul 19 15:27:17.642123 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:27:17.642065 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3a47b97-4b42-4be7-a5ef-c33bd3f0e27c-credential-socket\") pod \"e2e-migrate-431757e3-gw-openshift-default-7df98f6f75-x2kzr\" (UID: \"c3a47b97-4b42-4be7-a5ef-c33bd3f0e27c\") " pod="openshift-ingress/e2e-migrate-431757e3-gw-openshift-default-7df98f6f75-x2kzr" Jul 19 15:27:17.642123 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:27:17.642095 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c3a47b97-4b42-4be7-a5ef-c33bd3f0e27c-workload-certs\") pod \"e2e-migrate-431757e3-gw-openshift-default-7df98f6f75-x2kzr\" (UID: \"c3a47b97-4b42-4be7-a5ef-c33bd3f0e27c\") " pod="openshift-ingress/e2e-migrate-431757e3-gw-openshift-default-7df98f6f75-x2kzr" Jul 19 15:27:17.642247 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:27:17.642145 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c3a47b97-4b42-4be7-a5ef-c33bd3f0e27c-istio-envoy\") pod \"e2e-migrate-431757e3-gw-openshift-default-7df98f6f75-x2kzr\" (UID: \"c3a47b97-4b42-4be7-a5ef-c33bd3f0e27c\") " pod="openshift-ingress/e2e-migrate-431757e3-gw-openshift-default-7df98f6f75-x2kzr" Jul 19 15:27:17.642279 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:27:17.642247 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3a47b97-4b42-4be7-a5ef-c33bd3f0e27c-workload-socket\") pod \"e2e-migrate-431757e3-gw-openshift-default-7df98f6f75-x2kzr\" (UID: \"c3a47b97-4b42-4be7-a5ef-c33bd3f0e27c\") " pod="openshift-ingress/e2e-migrate-431757e3-gw-openshift-default-7df98f6f75-x2kzr" Jul 19 15:27:17.642311 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:27:17.642275 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-77ktn\" (UniqueName: \"kubernetes.io/projected/c3a47b97-4b42-4be7-a5ef-c33bd3f0e27c-kube-api-access-77ktn\") pod \"e2e-migrate-431757e3-gw-openshift-default-7df98f6f75-x2kzr\" (UID: \"c3a47b97-4b42-4be7-a5ef-c33bd3f0e27c\") " pod="openshift-ingress/e2e-migrate-431757e3-gw-openshift-default-7df98f6f75-x2kzr" Jul 19 15:27:17.742645 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:27:17.742609 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c3a47b97-4b42-4be7-a5ef-c33bd3f0e27c-workload-certs\") pod \"e2e-migrate-431757e3-gw-openshift-default-7df98f6f75-x2kzr\" (UID: \"c3a47b97-4b42-4be7-a5ef-c33bd3f0e27c\") " pod="openshift-ingress/e2e-migrate-431757e3-gw-openshift-default-7df98f6f75-x2kzr" Jul 19 15:27:17.742645 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:27:17.742646 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c3a47b97-4b42-4be7-a5ef-c33bd3f0e27c-istio-envoy\") pod \"e2e-migrate-431757e3-gw-openshift-default-7df98f6f75-x2kzr\" (UID: \"c3a47b97-4b42-4be7-a5ef-c33bd3f0e27c\") " pod="openshift-ingress/e2e-migrate-431757e3-gw-openshift-default-7df98f6f75-x2kzr" Jul 19 15:27:17.742840 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:27:17.742675 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3a47b97-4b42-4be7-a5ef-c33bd3f0e27c-workload-socket\") pod \"e2e-migrate-431757e3-gw-openshift-default-7df98f6f75-x2kzr\" (UID: \"c3a47b97-4b42-4be7-a5ef-c33bd3f0e27c\") " pod="openshift-ingress/e2e-migrate-431757e3-gw-openshift-default-7df98f6f75-x2kzr" Jul 19 15:27:17.742840 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:27:17.742691 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-77ktn\" (UniqueName: \"kubernetes.io/projected/c3a47b97-4b42-4be7-a5ef-c33bd3f0e27c-kube-api-access-77ktn\") pod \"e2e-migrate-431757e3-gw-openshift-default-7df98f6f75-x2kzr\" (UID: \"c3a47b97-4b42-4be7-a5ef-c33bd3f0e27c\") " pod="openshift-ingress/e2e-migrate-431757e3-gw-openshift-default-7df98f6f75-x2kzr" Jul 19 15:27:17.742840 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:27:17.742729 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c3a47b97-4b42-4be7-a5ef-c33bd3f0e27c-istio-podinfo\") pod \"e2e-migrate-431757e3-gw-openshift-default-7df98f6f75-x2kzr\" (UID: \"c3a47b97-4b42-4be7-a5ef-c33bd3f0e27c\") " pod="openshift-ingress/e2e-migrate-431757e3-gw-openshift-default-7df98f6f75-x2kzr" Jul 19 15:27:17.742840 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:27:17.742758 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c3a47b97-4b42-4be7-a5ef-c33bd3f0e27c-istio-data\") pod \"e2e-migrate-431757e3-gw-openshift-default-7df98f6f75-x2kzr\" (UID: \"c3a47b97-4b42-4be7-a5ef-c33bd3f0e27c\") " pod="openshift-ingress/e2e-migrate-431757e3-gw-openshift-default-7df98f6f75-x2kzr" Jul 19 15:27:17.742840 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:27:17.742779 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c3a47b97-4b42-4be7-a5ef-c33bd3f0e27c-istiod-ca-cert\") pod \"e2e-migrate-431757e3-gw-openshift-default-7df98f6f75-x2kzr\" (UID: \"c3a47b97-4b42-4be7-a5ef-c33bd3f0e27c\") " pod="openshift-ingress/e2e-migrate-431757e3-gw-openshift-default-7df98f6f75-x2kzr" Jul 19 15:27:17.742840 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:27:17.742835 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c3a47b97-4b42-4be7-a5ef-c33bd3f0e27c-istio-token\") pod \"e2e-migrate-431757e3-gw-openshift-default-7df98f6f75-x2kzr\" (UID: \"c3a47b97-4b42-4be7-a5ef-c33bd3f0e27c\") " pod="openshift-ingress/e2e-migrate-431757e3-gw-openshift-default-7df98f6f75-x2kzr" Jul 19 15:27:17.743193 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:27:17.742869 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3a47b97-4b42-4be7-a5ef-c33bd3f0e27c-credential-socket\") pod \"e2e-migrate-431757e3-gw-openshift-default-7df98f6f75-x2kzr\" (UID: \"c3a47b97-4b42-4be7-a5ef-c33bd3f0e27c\") " pod="openshift-ingress/e2e-migrate-431757e3-gw-openshift-default-7df98f6f75-x2kzr" Jul 19 15:27:17.743193 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:27:17.743072 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c3a47b97-4b42-4be7-a5ef-c33bd3f0e27c-workload-certs\") pod \"e2e-migrate-431757e3-gw-openshift-default-7df98f6f75-x2kzr\" (UID: \"c3a47b97-4b42-4be7-a5ef-c33bd3f0e27c\") " pod="openshift-ingress/e2e-migrate-431757e3-gw-openshift-default-7df98f6f75-x2kzr" Jul 19 15:27:17.743296 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:27:17.743270 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c3a47b97-4b42-4be7-a5ef-c33bd3f0e27c-istio-data\") pod \"e2e-migrate-431757e3-gw-openshift-default-7df98f6f75-x2kzr\" (UID: \"c3a47b97-4b42-4be7-a5ef-c33bd3f0e27c\") " pod="openshift-ingress/e2e-migrate-431757e3-gw-openshift-default-7df98f6f75-x2kzr" Jul 19 15:27:17.743296 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:27:17.743282 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3a47b97-4b42-4be7-a5ef-c33bd3f0e27c-credential-socket\") pod \"e2e-migrate-431757e3-gw-openshift-default-7df98f6f75-x2kzr\" (UID: \"c3a47b97-4b42-4be7-a5ef-c33bd3f0e27c\") " pod="openshift-ingress/e2e-migrate-431757e3-gw-openshift-default-7df98f6f75-x2kzr" Jul 19 15:27:17.743398 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:27:17.743306 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3a47b97-4b42-4be7-a5ef-c33bd3f0e27c-workload-socket\") pod \"e2e-migrate-431757e3-gw-openshift-default-7df98f6f75-x2kzr\" (UID: \"c3a47b97-4b42-4be7-a5ef-c33bd3f0e27c\") " pod="openshift-ingress/e2e-migrate-431757e3-gw-openshift-default-7df98f6f75-x2kzr" Jul 19 15:27:17.743621 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:27:17.743601 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c3a47b97-4b42-4be7-a5ef-c33bd3f0e27c-istiod-ca-cert\") pod \"e2e-migrate-431757e3-gw-openshift-default-7df98f6f75-x2kzr\" (UID: \"c3a47b97-4b42-4be7-a5ef-c33bd3f0e27c\") " pod="openshift-ingress/e2e-migrate-431757e3-gw-openshift-default-7df98f6f75-x2kzr" Jul 19 15:27:17.745444 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:27:17.745421 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c3a47b97-4b42-4be7-a5ef-c33bd3f0e27c-istio-envoy\") pod \"e2e-migrate-431757e3-gw-openshift-default-7df98f6f75-x2kzr\" (UID: \"c3a47b97-4b42-4be7-a5ef-c33bd3f0e27c\") " pod="openshift-ingress/e2e-migrate-431757e3-gw-openshift-default-7df98f6f75-x2kzr" Jul 19 15:27:17.745636 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:27:17.745619 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c3a47b97-4b42-4be7-a5ef-c33bd3f0e27c-istio-podinfo\") pod \"e2e-migrate-431757e3-gw-openshift-default-7df98f6f75-x2kzr\" (UID: \"c3a47b97-4b42-4be7-a5ef-c33bd3f0e27c\") " pod="openshift-ingress/e2e-migrate-431757e3-gw-openshift-default-7df98f6f75-x2kzr" Jul 19 15:27:17.750901 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:27:17.750874 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c3a47b97-4b42-4be7-a5ef-c33bd3f0e27c-istio-token\") pod \"e2e-migrate-431757e3-gw-openshift-default-7df98f6f75-x2kzr\" (UID: \"c3a47b97-4b42-4be7-a5ef-c33bd3f0e27c\") " pod="openshift-ingress/e2e-migrate-431757e3-gw-openshift-default-7df98f6f75-x2kzr" Jul 19 15:27:17.750995 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:27:17.750948 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-77ktn\" (UniqueName: \"kubernetes.io/projected/c3a47b97-4b42-4be7-a5ef-c33bd3f0e27c-kube-api-access-77ktn\") pod \"e2e-migrate-431757e3-gw-openshift-default-7df98f6f75-x2kzr\" (UID: \"c3a47b97-4b42-4be7-a5ef-c33bd3f0e27c\") " pod="openshift-ingress/e2e-migrate-431757e3-gw-openshift-default-7df98f6f75-x2kzr" Jul 19 15:27:17.833613 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:27:17.833539 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-431757e3-gw-openshift-default-7df98f6f75-x2kzr" Jul 19 15:27:17.968126 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:27:17.968094 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-431757e3-gw-openshift-default-7df98f6f75-x2kzr"] Jul 19 15:27:17.971202 ip-10-0-138-220 kubenswrapper[2558]: W0719 15:27:17.971175 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc3a47b97_4b42_4be7_a5ef_c33bd3f0e27c.slice/crio-35fbb7a6c54fd2cf805e4d4e921ca5fcb4313eba2f1b16dfe9a5beb4cdf31a61 WatchSource:0}: Error finding container 35fbb7a6c54fd2cf805e4d4e921ca5fcb4313eba2f1b16dfe9a5beb4cdf31a61: Status 404 returned error can't find the container with id 35fbb7a6c54fd2cf805e4d4e921ca5fcb4313eba2f1b16dfe9a5beb4cdf31a61 Jul 19 15:27:17.973425 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:27:17.973390 2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 19 15:27:17.973520 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:27:17.973468 2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 19 15:27:17.973520 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:27:17.973512 2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 19 15:27:18.533365 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:27:18.533332 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-431757e3-gw-openshift-default-7df98f6f75-x2kzr" event={"ID":"c3a47b97-4b42-4be7-a5ef-c33bd3f0e27c","Type":"ContainerStarted","Data":"eda464ff1860b9218f1fd097159e639844a263fa43ad3b8f9bcd3ef260e13332"} Jul 19 15:27:18.533811 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:27:18.533382 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-431757e3-gw-openshift-default-7df98f6f75-x2kzr" event={"ID":"c3a47b97-4b42-4be7-a5ef-c33bd3f0e27c","Type":"ContainerStarted","Data":"35fbb7a6c54fd2cf805e4d4e921ca5fcb4313eba2f1b16dfe9a5beb4cdf31a61"} Jul 19 15:27:18.555471 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:27:18.555409 2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-migrate-431757e3-gw-openshift-default-7df98f6f75-x2kzr" podStartSLOduration=1.555391165 podStartE2EDuration="1.555391165s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-19 15:27:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-19 15:27:18.551084933 +0000 UTC m=+1779.472787646" watchObservedRunningTime="2026-07-19 15:27:18.555391165 +0000 UTC m=+1779.477093881" Jul 19 15:27:18.834168 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:27:18.834079 2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-431757e3-gw-openshift-default-7df98f6f75-x2kzr" Jul 19 15:27:19.838675 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:27:19.838644 2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-431757e3-gw-openshift-default-7df98f6f75-x2kzr" Jul 19 15:27:20.541732 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:27:20.541696 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-431757e3-gw-openshift-default-7df98f6f75-x2kzr" Jul 19 15:27:20.542836 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:27:20.542808 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-431757e3-gw-openshift-default-7df98f6f75-x2kzr" Jul 19 15:27:21.725315 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:27:21.725290 2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-44de5c81-openshift-default-74f765c8c5-hzzw9" Jul 19 15:27:21.877415 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:27:21.877349 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ec616fc0-ba34-47b2-8423-93da5fe6e436-istio-data\") pod \"ec616fc0-ba34-47b2-8423-93da5fe6e436\" (UID: \"ec616fc0-ba34-47b2-8423-93da5fe6e436\") " Jul 19 15:27:21.877415 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:27:21.877385 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec616fc0-ba34-47b2-8423-93da5fe6e436-workload-socket\") pod \"ec616fc0-ba34-47b2-8423-93da5fe6e436\" (UID: \"ec616fc0-ba34-47b2-8423-93da5fe6e436\") " Jul 19 15:27:21.877415 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:27:21.877410 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec616fc0-ba34-47b2-8423-93da5fe6e436-credential-socket\") pod \"ec616fc0-ba34-47b2-8423-93da5fe6e436\" (UID: \"ec616fc0-ba34-47b2-8423-93da5fe6e436\") " Jul 19 15:27:21.877620 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:27:21.877435 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ec616fc0-ba34-47b2-8423-93da5fe6e436-istio-token\") pod \"ec616fc0-ba34-47b2-8423-93da5fe6e436\" (UID: \"ec616fc0-ba34-47b2-8423-93da5fe6e436\") " Jul 19 15:27:21.877620 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:27:21.877458 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ec616fc0-ba34-47b2-8423-93da5fe6e436-istiod-ca-cert\") pod \"ec616fc0-ba34-47b2-8423-93da5fe6e436\" (UID: \"ec616fc0-ba34-47b2-8423-93da5fe6e436\") " Jul 19 15:27:21.877620 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:27:21.877491 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ec616fc0-ba34-47b2-8423-93da5fe6e436-istio-envoy\") pod \"ec616fc0-ba34-47b2-8423-93da5fe6e436\" (UID: \"ec616fc0-ba34-47b2-8423-93da5fe6e436\") " Jul 19 15:27:21.877620 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:27:21.877520 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ec616fc0-ba34-47b2-8423-93da5fe6e436-istio-podinfo\") pod \"ec616fc0-ba34-47b2-8423-93da5fe6e436\" (UID: \"ec616fc0-ba34-47b2-8423-93da5fe6e436\") " Jul 19 15:27:21.877620 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:27:21.877565 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zl64q\" (UniqueName: \"kubernetes.io/projected/ec616fc0-ba34-47b2-8423-93da5fe6e436-kube-api-access-zl64q\") pod \"ec616fc0-ba34-47b2-8423-93da5fe6e436\" (UID: \"ec616fc0-ba34-47b2-8423-93da5fe6e436\") " Jul 19 15:27:21.877842 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:27:21.877628 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ec616fc0-ba34-47b2-8423-93da5fe6e436-workload-certs\") pod \"ec616fc0-ba34-47b2-8423-93da5fe6e436\" (UID: \"ec616fc0-ba34-47b2-8423-93da5fe6e436\") " Jul 19 15:27:21.877842 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:27:21.877680 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ec616fc0-ba34-47b2-8423-93da5fe6e436-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "ec616fc0-ba34-47b2-8423-93da5fe6e436" (UID: "ec616fc0-ba34-47b2-8423-93da5fe6e436"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 19 15:27:21.877842 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:27:21.877712 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ec616fc0-ba34-47b2-8423-93da5fe6e436-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "ec616fc0-ba34-47b2-8423-93da5fe6e436" (UID: "ec616fc0-ba34-47b2-8423-93da5fe6e436"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 19 15:27:21.877986 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:27:21.877894 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ec616fc0-ba34-47b2-8423-93da5fe6e436-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "ec616fc0-ba34-47b2-8423-93da5fe6e436" (UID: "ec616fc0-ba34-47b2-8423-93da5fe6e436"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 19 15:27:21.877986 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:27:21.877940 2558 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec616fc0-ba34-47b2-8423-93da5fe6e436-workload-socket\") on node \"ip-10-0-138-220.ec2.internal\" DevicePath \"\"" Jul 19 15:27:21.877986 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:27:21.877952 2558 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec616fc0-ba34-47b2-8423-93da5fe6e436-credential-socket\") on node \"ip-10-0-138-220.ec2.internal\" DevicePath \"\"" Jul 19 15:27:21.878222 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:27:21.878199 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ec616fc0-ba34-47b2-8423-93da5fe6e436-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "ec616fc0-ba34-47b2-8423-93da5fe6e436" (UID: "ec616fc0-ba34-47b2-8423-93da5fe6e436"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 19 15:27:21.878287 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:27:21.878235 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ec616fc0-ba34-47b2-8423-93da5fe6e436-istio-data" (OuterVolumeSpecName: "istio-data") pod "ec616fc0-ba34-47b2-8423-93da5fe6e436" (UID: "ec616fc0-ba34-47b2-8423-93da5fe6e436"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 19 15:27:21.880005 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:27:21.879972 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ec616fc0-ba34-47b2-8423-93da5fe6e436-kube-api-access-zl64q" (OuterVolumeSpecName: "kube-api-access-zl64q") pod "ec616fc0-ba34-47b2-8423-93da5fe6e436" (UID: "ec616fc0-ba34-47b2-8423-93da5fe6e436"). InnerVolumeSpecName "kube-api-access-zl64q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 19 15:27:21.880005 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:27:21.879991 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ec616fc0-ba34-47b2-8423-93da5fe6e436-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "ec616fc0-ba34-47b2-8423-93da5fe6e436" (UID: "ec616fc0-ba34-47b2-8423-93da5fe6e436"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 19 15:27:21.880207 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:27:21.880187 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ec616fc0-ba34-47b2-8423-93da5fe6e436-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "ec616fc0-ba34-47b2-8423-93da5fe6e436" (UID: "ec616fc0-ba34-47b2-8423-93da5fe6e436"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 19 15:27:21.880264 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:27:21.880226 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ec616fc0-ba34-47b2-8423-93da5fe6e436-istio-token" (OuterVolumeSpecName: "istio-token") pod "ec616fc0-ba34-47b2-8423-93da5fe6e436" (UID: "ec616fc0-ba34-47b2-8423-93da5fe6e436"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 19 15:27:21.978607 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:27:21.978573 2558 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ec616fc0-ba34-47b2-8423-93da5fe6e436-istio-data\") on node \"ip-10-0-138-220.ec2.internal\" DevicePath \"\"" Jul 19 15:27:21.978607 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:27:21.978605 2558 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ec616fc0-ba34-47b2-8423-93da5fe6e436-istio-token\") on node \"ip-10-0-138-220.ec2.internal\" DevicePath \"\"" Jul 19 15:27:21.978607 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:27:21.978613 2558 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ec616fc0-ba34-47b2-8423-93da5fe6e436-istiod-ca-cert\") on node \"ip-10-0-138-220.ec2.internal\" DevicePath \"\"" Jul 19 15:27:21.978827 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:27:21.978629 2558 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ec616fc0-ba34-47b2-8423-93da5fe6e436-istio-envoy\") on node \"ip-10-0-138-220.ec2.internal\" DevicePath \"\"" Jul 19 15:27:21.978827 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:27:21.978636 2558 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ec616fc0-ba34-47b2-8423-93da5fe6e436-istio-podinfo\") on node \"ip-10-0-138-220.ec2.internal\" DevicePath \"\"" Jul 19 15:27:21.978827 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:27:21.978644 2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zl64q\" (UniqueName: \"kubernetes.io/projected/ec616fc0-ba34-47b2-8423-93da5fe6e436-kube-api-access-zl64q\") on node \"ip-10-0-138-220.ec2.internal\" DevicePath \"\"" Jul 19 15:27:21.978827 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:27:21.978671 2558 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ec616fc0-ba34-47b2-8423-93da5fe6e436-workload-certs\") on node \"ip-10-0-138-220.ec2.internal\" DevicePath \"\"" Jul 19 15:27:22.551136 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:27:22.551104 2558 generic.go:356] "Generic (PLEG): container finished" podID="ec616fc0-ba34-47b2-8423-93da5fe6e436" containerID="1bdf35f203f012389fa3fb28bfc114f68c441af24d77af13fc83de6316faa73c" exitCode=0 Jul 19 15:27:22.551286 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:27:22.551151 2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-44de5c81-openshift-default-74f765c8c5-hzzw9" Jul 19 15:27:22.551286 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:27:22.551180 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-44de5c81-openshift-default-74f765c8c5-hzzw9" event={"ID":"ec616fc0-ba34-47b2-8423-93da5fe6e436","Type":"ContainerDied","Data":"1bdf35f203f012389fa3fb28bfc114f68c441af24d77af13fc83de6316faa73c"} Jul 19 15:27:22.551286 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:27:22.551224 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-44de5c81-openshift-default-74f765c8c5-hzzw9" event={"ID":"ec616fc0-ba34-47b2-8423-93da5fe6e436","Type":"ContainerDied","Data":"619b4416d0a0e4703c419da08053586f9149a7e7747a5c919a07eedafbede4e7"} Jul 19 15:27:22.551286 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:27:22.551245 2558 scope.go:117] "RemoveContainer" containerID="1bdf35f203f012389fa3fb28bfc114f68c441af24d77af13fc83de6316faa73c" Jul 19 15:27:22.560097 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:27:22.560082 2558 scope.go:117] "RemoveContainer" containerID="c2107aec3d973065c1b22b6fbdaa1d9a3a1a200a67a3e251696bf8e158c15fbc" Jul 19 15:27:22.567856 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:27:22.567838 2558 scope.go:117] "RemoveContainer" containerID="1bdf35f203f012389fa3fb28bfc114f68c441af24d77af13fc83de6316faa73c" Jul 19 15:27:22.568187 ip-10-0-138-220 kubenswrapper[2558]: E0719 15:27:22.568168 2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1bdf35f203f012389fa3fb28bfc114f68c441af24d77af13fc83de6316faa73c\": container with ID starting with 1bdf35f203f012389fa3fb28bfc114f68c441af24d77af13fc83de6316faa73c not found: ID does not exist" containerID="1bdf35f203f012389fa3fb28bfc114f68c441af24d77af13fc83de6316faa73c" Jul 19 15:27:22.568249 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:27:22.568195 2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1bdf35f203f012389fa3fb28bfc114f68c441af24d77af13fc83de6316faa73c"} err="failed to get container status \"1bdf35f203f012389fa3fb28bfc114f68c441af24d77af13fc83de6316faa73c\": rpc error: code = NotFound desc = could not find container \"1bdf35f203f012389fa3fb28bfc114f68c441af24d77af13fc83de6316faa73c\": container with ID starting with 1bdf35f203f012389fa3fb28bfc114f68c441af24d77af13fc83de6316faa73c not found: ID does not exist" Jul 19 15:27:22.568249 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:27:22.568214 2558 scope.go:117] "RemoveContainer" containerID="c2107aec3d973065c1b22b6fbdaa1d9a3a1a200a67a3e251696bf8e158c15fbc" Jul 19 15:27:22.568460 ip-10-0-138-220 kubenswrapper[2558]: E0719 15:27:22.568448 2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c2107aec3d973065c1b22b6fbdaa1d9a3a1a200a67a3e251696bf8e158c15fbc\": container with ID starting with c2107aec3d973065c1b22b6fbdaa1d9a3a1a200a67a3e251696bf8e158c15fbc not found: ID does not exist" containerID="c2107aec3d973065c1b22b6fbdaa1d9a3a1a200a67a3e251696bf8e158c15fbc" Jul 19 15:27:22.568502 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:27:22.568463 2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c2107aec3d973065c1b22b6fbdaa1d9a3a1a200a67a3e251696bf8e158c15fbc"} err="failed to get container status \"c2107aec3d973065c1b22b6fbdaa1d9a3a1a200a67a3e251696bf8e158c15fbc\": rpc error: code = NotFound desc = could not find container \"c2107aec3d973065c1b22b6fbdaa1d9a3a1a200a67a3e251696bf8e158c15fbc\": container with ID starting with c2107aec3d973065c1b22b6fbdaa1d9a3a1a200a67a3e251696bf8e158c15fbc not found: ID does not exist" Jul 19 15:27:22.571598 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:27:22.571573 2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-44de5c81-openshift-default-74f765c8c5-hzzw9"] Jul 19 15:27:22.575752 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:27:22.575719 2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-44de5c81-openshift-default-74f765c8c5-hzzw9"] Jul 19 15:27:23.565803 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:27:23.565763 2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ec616fc0-ba34-47b2-8423-93da5fe6e436" path="/var/lib/kubelet/pods/ec616fc0-ba34-47b2-8423-93da5fe6e436/volumes" Jul 19 15:27:25.494644 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:27:25.494599 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-d4lll"] Jul 19 15:27:33.599148 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:27:33.599056 2558 generic.go:356] "Generic (PLEG): container finished" podID="c3a47b97-4b42-4be7-a5ef-c33bd3f0e27c" containerID="eda464ff1860b9218f1fd097159e639844a263fa43ad3b8f9bcd3ef260e13332" exitCode=0 Jul 19 15:27:33.599148 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:27:33.599135 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-431757e3-gw-openshift-default-7df98f6f75-x2kzr" event={"ID":"c3a47b97-4b42-4be7-a5ef-c33bd3f0e27c","Type":"ContainerDied","Data":"eda464ff1860b9218f1fd097159e639844a263fa43ad3b8f9bcd3ef260e13332"} Jul 19 15:27:33.599575 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:27:33.599476 2558 scope.go:117] "RemoveContainer" containerID="eda464ff1860b9218f1fd097159e639844a263fa43ad3b8f9bcd3ef260e13332" Jul 19 15:27:33.600551 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:27:33.600527 2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 19 15:27:33.600603 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:27:33.600590 2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 19 15:27:33.600633 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:27:33.600621 2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 19 15:27:34.605189 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:27:34.605157 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-431757e3-gw-openshift-default-7df98f6f75-x2kzr" event={"ID":"c3a47b97-4b42-4be7-a5ef-c33bd3f0e27c","Type":"ContainerStarted","Data":"3b2936eb3551af193519814fafbe73b3448b138982f9e21a7eb4af3fd8a7a92d"} Jul 19 15:27:34.833977 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:27:34.833938 2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-431757e3-gw-openshift-default-7df98f6f75-x2kzr" Jul 19 15:27:34.835104 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:27:34.835078 2558 patch_prober.go:28] interesting pod/e2e-migrate-431757e3-gw-openshift-default-7df98f6f75-x2kzr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.26:15021/healthz/ready\": dial tcp 10.133.0.26:15021: connect: connection refused" start-of-body= Jul 19 15:27:34.835234 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:27:34.835127 2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-431757e3-gw-openshift-default-7df98f6f75-x2kzr" podUID="c3a47b97-4b42-4be7-a5ef-c33bd3f0e27c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.26:15021/healthz/ready\": dial tcp 10.133.0.26:15021: connect: connection refused" Jul 19 15:27:35.542492 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:27:35.542458 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-431757e3-gw-openshift-default-7df98f6f75-x2kzr" Jul 19 15:27:35.834874 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:27:35.834792 2558 patch_prober.go:28] interesting pod/e2e-migrate-431757e3-gw-openshift-default-7df98f6f75-x2kzr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.26:15021/healthz/ready\": dial tcp 10.133.0.26:15021: connect: connection refused" start-of-body= Jul 19 15:27:35.834874 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:27:35.834847 2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-431757e3-gw-openshift-default-7df98f6f75-x2kzr" podUID="c3a47b97-4b42-4be7-a5ef-c33bd3f0e27c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.26:15021/healthz/ready\": dial tcp 10.133.0.26:15021: connect: connection refused" Jul 19 15:27:36.834329 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:27:36.834286 2558 patch_prober.go:28] interesting pod/e2e-migrate-431757e3-gw-openshift-default-7df98f6f75-x2kzr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.26:15021/healthz/ready\": dial tcp 10.133.0.26:15021: connect: connection refused" start-of-body= Jul 19 15:27:36.834528 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:27:36.834346 2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-431757e3-gw-openshift-default-7df98f6f75-x2kzr" podUID="c3a47b97-4b42-4be7-a5ef-c33bd3f0e27c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.26:15021/healthz/ready\": dial tcp 10.133.0.26:15021: connect: connection refused" Jul 19 15:27:37.835001 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:27:37.834960 2558 patch_prober.go:28] interesting pod/e2e-migrate-431757e3-gw-openshift-default-7df98f6f75-x2kzr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.26:15021/healthz/ready\": dial tcp 10.133.0.26:15021: connect: connection refused" start-of-body= Jul 19 15:27:37.835457 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:27:37.835051 2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-431757e3-gw-openshift-default-7df98f6f75-x2kzr" podUID="c3a47b97-4b42-4be7-a5ef-c33bd3f0e27c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.26:15021/healthz/ready\": dial tcp 10.133.0.26:15021: connect: connection refused" Jul 19 15:27:38.838383 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:27:38.838355 2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-431757e3-gw-openshift-default-7df98f6f75-x2kzr" Jul 19 15:27:38.839386 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:27:38.839370 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-431757e3-gw-openshift-default-7df98f6f75-x2kzr" Jul 19 15:27:39.620240 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:27:39.620214 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-5wfzs_5b2d3a07-86b2-4ba2-8a71-c2d89ada79c6/ovn-acl-logging/0.log" Jul 19 15:27:39.623844 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:27:39.623819 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-5wfzs_5b2d3a07-86b2-4ba2-8a71-c2d89ada79c6/ovn-acl-logging/0.log" Jul 19 15:27:45.024735 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:27:45.024704 2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-431757e3-7hgtg"] Jul 19 15:27:45.025118 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:27:45.024985 2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ec616fc0-ba34-47b2-8423-93da5fe6e436" containerName="istio-proxy" Jul 19 15:27:45.025118 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:27:45.024996 2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="ec616fc0-ba34-47b2-8423-93da5fe6e436" containerName="istio-proxy" Jul 19 15:27:45.025118 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:27:45.025071 2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="ec616fc0-ba34-47b2-8423-93da5fe6e436" containerName="istio-proxy" Jul 19 15:27:45.025118 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:27:45.025078 2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="ec616fc0-ba34-47b2-8423-93da5fe6e436" containerName="istio-proxy" Jul 19 15:27:45.027951 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:27:45.027934 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-431757e3-7hgtg" Jul 19 15:27:45.030059 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:27:45.030034 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qddbb\"" Jul 19 15:27:45.030193 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:27:45.030066 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 19 15:27:45.035054 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:27:45.034958 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-431757e3-7hgtg"] Jul 19 15:27:45.169867 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:27:45.169832 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2ff11bfa-490b-4225-86c5-0b24956b110d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-431757e3-7hgtg\" (UID: \"2ff11bfa-490b-4225-86c5-0b24956b110d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-431757e3-7hgtg" Jul 19 15:27:45.270963 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:27:45.270932 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2ff11bfa-490b-4225-86c5-0b24956b110d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-431757e3-7hgtg\" (UID: \"2ff11bfa-490b-4225-86c5-0b24956b110d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-431757e3-7hgtg" Jul 19 15:27:45.271545 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:27:45.271524 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2ff11bfa-490b-4225-86c5-0b24956b110d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-431757e3-7hgtg\" (UID: \"2ff11bfa-490b-4225-86c5-0b24956b110d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-431757e3-7hgtg" Jul 19 15:27:45.339079 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:27:45.338993 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-431757e3-7hgtg" Jul 19 15:27:45.462806 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:27:45.462775 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-431757e3-7hgtg"] Jul 19 15:27:45.465213 ip-10-0-138-220 kubenswrapper[2558]: W0719 15:27:45.465177 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2ff11bfa_490b_4225_86c5_0b24956b110d.slice/crio-0851b34453541a5ac7450c228b52fae6d6de50a736db510cbdf0b907463afecd WatchSource:0}: Error finding container 0851b34453541a5ac7450c228b52fae6d6de50a736db510cbdf0b907463afecd: Status 404 returned error can't find the container with id 0851b34453541a5ac7450c228b52fae6d6de50a736db510cbdf0b907463afecd Jul 19 15:27:45.649089 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:27:45.649058 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-431757e3-7hgtg" event={"ID":"2ff11bfa-490b-4225-86c5-0b24956b110d","Type":"ContainerStarted","Data":"0851b34453541a5ac7450c228b52fae6d6de50a736db510cbdf0b907463afecd"} Jul 19 15:27:46.655003 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:27:46.654971 2558 generic.go:356] "Generic (PLEG): container finished" podID="2ff11bfa-490b-4225-86c5-0b24956b110d" containerID="036839a8bec543a75b8036795264ceb1b8131f5c48b14257642f551d3ab12017" exitCode=0 Jul 19 15:27:46.655395 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:27:46.655062 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-431757e3-7hgtg" event={"ID":"2ff11bfa-490b-4225-86c5-0b24956b110d","Type":"ContainerDied","Data":"036839a8bec543a75b8036795264ceb1b8131f5c48b14257642f551d3ab12017"} Jul 19 15:27:47.783531 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:27:47.783508 2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-431757e3-7hgtg" Jul 19 15:27:47.790890 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:27:47.790872 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2ff11bfa-490b-4225-86c5-0b24956b110d-maas-api-service-ca\") pod \"2ff11bfa-490b-4225-86c5-0b24956b110d\" (UID: \"2ff11bfa-490b-4225-86c5-0b24956b110d\") " Jul 19 15:27:47.791232 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:27:47.791211 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2ff11bfa-490b-4225-86c5-0b24956b110d-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "2ff11bfa-490b-4225-86c5-0b24956b110d" (UID: "2ff11bfa-490b-4225-86c5-0b24956b110d"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 19 15:27:47.891689 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:27:47.891662 2558 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2ff11bfa-490b-4225-86c5-0b24956b110d-maas-api-service-ca\") on node \"ip-10-0-138-220.ec2.internal\" DevicePath \"\"" Jul 19 15:27:48.664341 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:27:48.664309 2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-431757e3-7hgtg" Jul 19 15:27:48.664341 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:27:48.664329 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-431757e3-7hgtg" event={"ID":"2ff11bfa-490b-4225-86c5-0b24956b110d","Type":"ContainerDied","Data":"0851b34453541a5ac7450c228b52fae6d6de50a736db510cbdf0b907463afecd"} Jul 19 15:27:48.664586 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:27:48.664372 2558 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0851b34453541a5ac7450c228b52fae6d6de50a736db510cbdf0b907463afecd" Jul 19 15:27:58.582406 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:27:58.582374 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-d4lll"] Jul 19 15:28:00.101225 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:28:00.101193 2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-431757e3-7hgtg"] Jul 19 15:28:00.104110 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:28:00.104088 2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-431757e3-7hgtg"] Jul 19 15:28:00.246156 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:28:00.246124 2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-431757e3-gw-openshift-default-7df98f6f75-x2kzr"] Jul 19 15:28:00.246431 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:28:00.246384 2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-migrate-431757e3-gw-openshift-default-7df98f6f75-x2kzr" podUID="c3a47b97-4b42-4be7-a5ef-c33bd3f0e27c" containerName="istio-proxy" containerID="cri-o://3b2936eb3551af193519814fafbe73b3448b138982f9e21a7eb4af3fd8a7a92d" gracePeriod=30 Jul 19 15:28:01.055313 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:28:01.055283 2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-a4b8edd0-openshift-default-5db5d69955-5fll7"] Jul 19 15:28:01.055652 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:28:01.055634 2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ec616fc0-ba34-47b2-8423-93da5fe6e436" containerName="istio-proxy" Jul 19 15:28:01.055742 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:28:01.055656 2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="ec616fc0-ba34-47b2-8423-93da5fe6e436" containerName="istio-proxy" Jul 19 15:28:01.055742 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:28:01.055680 2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2ff11bfa-490b-4225-86c5-0b24956b110d" containerName="revoke-keys" Jul 19 15:28:01.055742 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:28:01.055689 2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="2ff11bfa-490b-4225-86c5-0b24956b110d" containerName="revoke-keys" Jul 19 15:28:01.055889 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:28:01.055782 2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="2ff11bfa-490b-4225-86c5-0b24956b110d" containerName="revoke-keys" Jul 19 15:28:01.059336 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:28:01.059316 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-a4b8edd0-openshift-default-5db5d69955-5fll7" Jul 19 15:28:01.061443 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:28:01.061424 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-a4b8edd0-openshift-default-dockercfg-tms8j\"" Jul 19 15:28:01.070275 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:28:01.070250 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-a4b8edd0-openshift-default-5db5d69955-5fll7"] Jul 19 15:28:01.088647 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:28:01.088622 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7txww\" (UniqueName: \"kubernetes.io/projected/1cbf9895-d90f-49bb-96c8-af951e03ed63-kube-api-access-7txww\") pod \"e2e-ait-a4b8edd0-openshift-default-5db5d69955-5fll7\" (UID: \"1cbf9895-d90f-49bb-96c8-af951e03ed63\") " pod="openshift-ingress/e2e-ait-a4b8edd0-openshift-default-5db5d69955-5fll7" Jul 19 15:28:01.088747 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:28:01.088672 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1cbf9895-d90f-49bb-96c8-af951e03ed63-workload-socket\") pod \"e2e-ait-a4b8edd0-openshift-default-5db5d69955-5fll7\" (UID: \"1cbf9895-d90f-49bb-96c8-af951e03ed63\") " pod="openshift-ingress/e2e-ait-a4b8edd0-openshift-default-5db5d69955-5fll7" Jul 19 15:28:01.088805 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:28:01.088743 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1cbf9895-d90f-49bb-96c8-af951e03ed63-istio-data\") pod \"e2e-ait-a4b8edd0-openshift-default-5db5d69955-5fll7\" (UID: \"1cbf9895-d90f-49bb-96c8-af951e03ed63\") " pod="openshift-ingress/e2e-ait-a4b8edd0-openshift-default-5db5d69955-5fll7" Jul 19 15:28:01.088805 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:28:01.088767 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1cbf9895-d90f-49bb-96c8-af951e03ed63-istio-podinfo\") pod \"e2e-ait-a4b8edd0-openshift-default-5db5d69955-5fll7\" (UID: \"1cbf9895-d90f-49bb-96c8-af951e03ed63\") " pod="openshift-ingress/e2e-ait-a4b8edd0-openshift-default-5db5d69955-5fll7" Jul 19 15:28:01.088805 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:28:01.088794 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1cbf9895-d90f-49bb-96c8-af951e03ed63-istio-token\") pod \"e2e-ait-a4b8edd0-openshift-default-5db5d69955-5fll7\" (UID: \"1cbf9895-d90f-49bb-96c8-af951e03ed63\") " pod="openshift-ingress/e2e-ait-a4b8edd0-openshift-default-5db5d69955-5fll7" Jul 19 15:28:01.088940 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:28:01.088875 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1cbf9895-d90f-49bb-96c8-af951e03ed63-credential-socket\") pod \"e2e-ait-a4b8edd0-openshift-default-5db5d69955-5fll7\" (UID: \"1cbf9895-d90f-49bb-96c8-af951e03ed63\") " pod="openshift-ingress/e2e-ait-a4b8edd0-openshift-default-5db5d69955-5fll7" Jul 19 15:28:01.088940 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:28:01.088917 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1cbf9895-d90f-49bb-96c8-af951e03ed63-workload-certs\") pod \"e2e-ait-a4b8edd0-openshift-default-5db5d69955-5fll7\" (UID: \"1cbf9895-d90f-49bb-96c8-af951e03ed63\") " pod="openshift-ingress/e2e-ait-a4b8edd0-openshift-default-5db5d69955-5fll7" Jul 19 15:28:01.089070 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:28:01.088946 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1cbf9895-d90f-49bb-96c8-af951e03ed63-istio-envoy\") pod \"e2e-ait-a4b8edd0-openshift-default-5db5d69955-5fll7\" (UID: \"1cbf9895-d90f-49bb-96c8-af951e03ed63\") " pod="openshift-ingress/e2e-ait-a4b8edd0-openshift-default-5db5d69955-5fll7" Jul 19 15:28:01.089070 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:28:01.088969 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1cbf9895-d90f-49bb-96c8-af951e03ed63-istiod-ca-cert\") pod \"e2e-ait-a4b8edd0-openshift-default-5db5d69955-5fll7\" (UID: \"1cbf9895-d90f-49bb-96c8-af951e03ed63\") " pod="openshift-ingress/e2e-ait-a4b8edd0-openshift-default-5db5d69955-5fll7" Jul 19 15:28:01.190285 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:28:01.190249 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1cbf9895-d90f-49bb-96c8-af951e03ed63-istio-data\") pod \"e2e-ait-a4b8edd0-openshift-default-5db5d69955-5fll7\" (UID: \"1cbf9895-d90f-49bb-96c8-af951e03ed63\") " pod="openshift-ingress/e2e-ait-a4b8edd0-openshift-default-5db5d69955-5fll7" Jul 19 15:28:01.190787 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:28:01.190289 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1cbf9895-d90f-49bb-96c8-af951e03ed63-istio-podinfo\") pod \"e2e-ait-a4b8edd0-openshift-default-5db5d69955-5fll7\" (UID: \"1cbf9895-d90f-49bb-96c8-af951e03ed63\") " pod="openshift-ingress/e2e-ait-a4b8edd0-openshift-default-5db5d69955-5fll7" Jul 19 15:28:01.190787 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:28:01.190339 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1cbf9895-d90f-49bb-96c8-af951e03ed63-istio-token\") pod \"e2e-ait-a4b8edd0-openshift-default-5db5d69955-5fll7\" (UID: \"1cbf9895-d90f-49bb-96c8-af951e03ed63\") " pod="openshift-ingress/e2e-ait-a4b8edd0-openshift-default-5db5d69955-5fll7" Jul 19 15:28:01.190787 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:28:01.190418 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1cbf9895-d90f-49bb-96c8-af951e03ed63-credential-socket\") pod \"e2e-ait-a4b8edd0-openshift-default-5db5d69955-5fll7\" (UID: \"1cbf9895-d90f-49bb-96c8-af951e03ed63\") " pod="openshift-ingress/e2e-ait-a4b8edd0-openshift-default-5db5d69955-5fll7" Jul 19 15:28:01.190787 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:28:01.190465 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1cbf9895-d90f-49bb-96c8-af951e03ed63-workload-certs\") pod \"e2e-ait-a4b8edd0-openshift-default-5db5d69955-5fll7\" (UID: \"1cbf9895-d90f-49bb-96c8-af951e03ed63\") " pod="openshift-ingress/e2e-ait-a4b8edd0-openshift-default-5db5d69955-5fll7" Jul 19 15:28:01.190787 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:28:01.190488 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1cbf9895-d90f-49bb-96c8-af951e03ed63-istio-envoy\") pod \"e2e-ait-a4b8edd0-openshift-default-5db5d69955-5fll7\" (UID: \"1cbf9895-d90f-49bb-96c8-af951e03ed63\") " pod="openshift-ingress/e2e-ait-a4b8edd0-openshift-default-5db5d69955-5fll7" Jul 19 15:28:01.190787 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:28:01.190503 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1cbf9895-d90f-49bb-96c8-af951e03ed63-istiod-ca-cert\") pod \"e2e-ait-a4b8edd0-openshift-default-5db5d69955-5fll7\" (UID: \"1cbf9895-d90f-49bb-96c8-af951e03ed63\") " pod="openshift-ingress/e2e-ait-a4b8edd0-openshift-default-5db5d69955-5fll7" Jul 19 15:28:01.190787 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:28:01.190527 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7txww\" (UniqueName: \"kubernetes.io/projected/1cbf9895-d90f-49bb-96c8-af951e03ed63-kube-api-access-7txww\") pod \"e2e-ait-a4b8edd0-openshift-default-5db5d69955-5fll7\" (UID: \"1cbf9895-d90f-49bb-96c8-af951e03ed63\") " pod="openshift-ingress/e2e-ait-a4b8edd0-openshift-default-5db5d69955-5fll7" Jul 19 15:28:01.190787 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:28:01.190573 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1cbf9895-d90f-49bb-96c8-af951e03ed63-workload-socket\") pod \"e2e-ait-a4b8edd0-openshift-default-5db5d69955-5fll7\" (UID: \"1cbf9895-d90f-49bb-96c8-af951e03ed63\") " pod="openshift-ingress/e2e-ait-a4b8edd0-openshift-default-5db5d69955-5fll7" Jul 19 15:28:01.190787 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:28:01.190708 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1cbf9895-d90f-49bb-96c8-af951e03ed63-istio-data\") pod \"e2e-ait-a4b8edd0-openshift-default-5db5d69955-5fll7\" (UID: \"1cbf9895-d90f-49bb-96c8-af951e03ed63\") " pod="openshift-ingress/e2e-ait-a4b8edd0-openshift-default-5db5d69955-5fll7" Jul 19 15:28:01.191378 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:28:01.190804 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1cbf9895-d90f-49bb-96c8-af951e03ed63-workload-certs\") pod \"e2e-ait-a4b8edd0-openshift-default-5db5d69955-5fll7\" (UID: \"1cbf9895-d90f-49bb-96c8-af951e03ed63\") " pod="openshift-ingress/e2e-ait-a4b8edd0-openshift-default-5db5d69955-5fll7" Jul 19 15:28:01.191378 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:28:01.191009 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1cbf9895-d90f-49bb-96c8-af951e03ed63-credential-socket\") pod \"e2e-ait-a4b8edd0-openshift-default-5db5d69955-5fll7\" (UID: \"1cbf9895-d90f-49bb-96c8-af951e03ed63\") " pod="openshift-ingress/e2e-ait-a4b8edd0-openshift-default-5db5d69955-5fll7" Jul 19 15:28:01.191378 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:28:01.191015 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1cbf9895-d90f-49bb-96c8-af951e03ed63-workload-socket\") pod \"e2e-ait-a4b8edd0-openshift-default-5db5d69955-5fll7\" (UID: \"1cbf9895-d90f-49bb-96c8-af951e03ed63\") " pod="openshift-ingress/e2e-ait-a4b8edd0-openshift-default-5db5d69955-5fll7" Jul 19 15:28:01.191378 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:28:01.191290 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1cbf9895-d90f-49bb-96c8-af951e03ed63-istiod-ca-cert\") pod \"e2e-ait-a4b8edd0-openshift-default-5db5d69955-5fll7\" (UID: \"1cbf9895-d90f-49bb-96c8-af951e03ed63\") " pod="openshift-ingress/e2e-ait-a4b8edd0-openshift-default-5db5d69955-5fll7" Jul 19 15:28:01.192935 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:28:01.192916 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1cbf9895-d90f-49bb-96c8-af951e03ed63-istio-envoy\") pod \"e2e-ait-a4b8edd0-openshift-default-5db5d69955-5fll7\" (UID: \"1cbf9895-d90f-49bb-96c8-af951e03ed63\") " pod="openshift-ingress/e2e-ait-a4b8edd0-openshift-default-5db5d69955-5fll7" Jul 19 15:28:01.193289 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:28:01.193269 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1cbf9895-d90f-49bb-96c8-af951e03ed63-istio-podinfo\") pod \"e2e-ait-a4b8edd0-openshift-default-5db5d69955-5fll7\" (UID: \"1cbf9895-d90f-49bb-96c8-af951e03ed63\") " pod="openshift-ingress/e2e-ait-a4b8edd0-openshift-default-5db5d69955-5fll7" Jul 19 15:28:01.205227 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:28:01.205202 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1cbf9895-d90f-49bb-96c8-af951e03ed63-istio-token\") pod \"e2e-ait-a4b8edd0-openshift-default-5db5d69955-5fll7\" (UID: \"1cbf9895-d90f-49bb-96c8-af951e03ed63\") " pod="openshift-ingress/e2e-ait-a4b8edd0-openshift-default-5db5d69955-5fll7" Jul 19 15:28:01.205329 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:28:01.205214 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7txww\" (UniqueName: \"kubernetes.io/projected/1cbf9895-d90f-49bb-96c8-af951e03ed63-kube-api-access-7txww\") pod \"e2e-ait-a4b8edd0-openshift-default-5db5d69955-5fll7\" (UID: \"1cbf9895-d90f-49bb-96c8-af951e03ed63\") " pod="openshift-ingress/e2e-ait-a4b8edd0-openshift-default-5db5d69955-5fll7" Jul 19 15:28:01.371392 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:28:01.371308 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-a4b8edd0-openshift-default-5db5d69955-5fll7" Jul 19 15:28:01.511150 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:28:01.511125 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-a4b8edd0-openshift-default-5db5d69955-5fll7"] Jul 19 15:28:01.513495 ip-10-0-138-220 kubenswrapper[2558]: W0719 15:28:01.513473 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1cbf9895_d90f_49bb_96c8_af951e03ed63.slice/crio-76f69c5bbbb968841c38c66fc4487d7970790ad8d9d9fa6b6584655bb8a31477 WatchSource:0}: Error finding container 76f69c5bbbb968841c38c66fc4487d7970790ad8d9d9fa6b6584655bb8a31477: Status 404 returned error can't find the container with id 76f69c5bbbb968841c38c66fc4487d7970790ad8d9d9fa6b6584655bb8a31477 Jul 19 15:28:01.515527 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:28:01.515494 2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 19 15:28:01.515622 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:28:01.515559 2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 19 15:28:01.515622 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:28:01.515586 2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 19 15:28:01.566364 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:28:01.566333 2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2ff11bfa-490b-4225-86c5-0b24956b110d" path="/var/lib/kubelet/pods/2ff11bfa-490b-4225-86c5-0b24956b110d/volumes" Jul 19 15:28:01.720219 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:28:01.720172 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-a4b8edd0-openshift-default-5db5d69955-5fll7" event={"ID":"1cbf9895-d90f-49bb-96c8-af951e03ed63","Type":"ContainerStarted","Data":"ea532bf9034d0866789a27dd5d9b4dee57fe676be4b42cb63850ee03544c0289"} Jul 19 15:28:01.720367 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:28:01.720235 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-a4b8edd0-openshift-default-5db5d69955-5fll7" event={"ID":"1cbf9895-d90f-49bb-96c8-af951e03ed63","Type":"ContainerStarted","Data":"76f69c5bbbb968841c38c66fc4487d7970790ad8d9d9fa6b6584655bb8a31477"} Jul 19 15:28:01.741574 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:28:01.741528 2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-a4b8edd0-openshift-default-5db5d69955-5fll7" podStartSLOduration=0.74151264 podStartE2EDuration="741.51264ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-19 15:28:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-19 15:28:01.739492638 +0000 UTC m=+1822.661195365" watchObservedRunningTime="2026-07-19 15:28:01.74151264 +0000 UTC m=+1822.663215353" Jul 19 15:28:02.372285 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:28:02.372253 2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-a4b8edd0-openshift-default-5db5d69955-5fll7" Jul 19 15:28:02.376890 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:28:02.376864 2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-a4b8edd0-openshift-default-5db5d69955-5fll7" Jul 19 15:28:02.724580 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:28:02.724550 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-a4b8edd0-openshift-default-5db5d69955-5fll7" Jul 19 15:28:02.725452 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:28:02.725434 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-a4b8edd0-openshift-default-5db5d69955-5fll7" Jul 19 15:28:05.483623 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:28:05.483596 2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-431757e3-gw-openshift-default-7df98f6f75-x2kzr" Jul 19 15:28:05.528462 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:28:05.528390 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c3a47b97-4b42-4be7-a5ef-c33bd3f0e27c-istio-podinfo\") pod \"c3a47b97-4b42-4be7-a5ef-c33bd3f0e27c\" (UID: \"c3a47b97-4b42-4be7-a5ef-c33bd3f0e27c\") " Jul 19 15:28:05.528462 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:28:05.528426 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3a47b97-4b42-4be7-a5ef-c33bd3f0e27c-workload-socket\") pod \"c3a47b97-4b42-4be7-a5ef-c33bd3f0e27c\" (UID: \"c3a47b97-4b42-4be7-a5ef-c33bd3f0e27c\") " Jul 19 15:28:05.528462 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:28:05.528443 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-77ktn\" (UniqueName: \"kubernetes.io/projected/c3a47b97-4b42-4be7-a5ef-c33bd3f0e27c-kube-api-access-77ktn\") pod \"c3a47b97-4b42-4be7-a5ef-c33bd3f0e27c\" (UID: \"c3a47b97-4b42-4be7-a5ef-c33bd3f0e27c\") " Jul 19 15:28:05.528687 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:28:05.528507 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3a47b97-4b42-4be7-a5ef-c33bd3f0e27c-credential-socket\") pod \"c3a47b97-4b42-4be7-a5ef-c33bd3f0e27c\" (UID: \"c3a47b97-4b42-4be7-a5ef-c33bd3f0e27c\") " Jul 19 15:28:05.528687 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:28:05.528525 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c3a47b97-4b42-4be7-a5ef-c33bd3f0e27c-workload-certs\") pod \"c3a47b97-4b42-4be7-a5ef-c33bd3f0e27c\" (UID: \"c3a47b97-4b42-4be7-a5ef-c33bd3f0e27c\") " Jul 19 15:28:05.528687 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:28:05.528549 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c3a47b97-4b42-4be7-a5ef-c33bd3f0e27c-istio-envoy\") pod \"c3a47b97-4b42-4be7-a5ef-c33bd3f0e27c\" (UID: \"c3a47b97-4b42-4be7-a5ef-c33bd3f0e27c\") " Jul 19 15:28:05.528687 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:28:05.528586 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c3a47b97-4b42-4be7-a5ef-c33bd3f0e27c-istio-token\") pod \"c3a47b97-4b42-4be7-a5ef-c33bd3f0e27c\" (UID: \"c3a47b97-4b42-4be7-a5ef-c33bd3f0e27c\") " Jul 19 15:28:05.528687 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:28:05.528617 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c3a47b97-4b42-4be7-a5ef-c33bd3f0e27c-istiod-ca-cert\") pod \"c3a47b97-4b42-4be7-a5ef-c33bd3f0e27c\" (UID: \"c3a47b97-4b42-4be7-a5ef-c33bd3f0e27c\") " Jul 19 15:28:05.528687 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:28:05.528644 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c3a47b97-4b42-4be7-a5ef-c33bd3f0e27c-istio-data\") pod \"c3a47b97-4b42-4be7-a5ef-c33bd3f0e27c\" (UID: \"c3a47b97-4b42-4be7-a5ef-c33bd3f0e27c\") " Jul 19 15:28:05.528687 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:28:05.528673 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c3a47b97-4b42-4be7-a5ef-c33bd3f0e27c-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "c3a47b97-4b42-4be7-a5ef-c33bd3f0e27c" (UID: "c3a47b97-4b42-4be7-a5ef-c33bd3f0e27c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 19 15:28:05.529012 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:28:05.528866 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c3a47b97-4b42-4be7-a5ef-c33bd3f0e27c-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "c3a47b97-4b42-4be7-a5ef-c33bd3f0e27c" (UID: "c3a47b97-4b42-4be7-a5ef-c33bd3f0e27c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 19 15:28:05.529012 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:28:05.528922 2558 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3a47b97-4b42-4be7-a5ef-c33bd3f0e27c-workload-socket\") on node \"ip-10-0-138-220.ec2.internal\" DevicePath \"\"" Jul 19 15:28:05.529144 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:28:05.529058 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c3a47b97-4b42-4be7-a5ef-c33bd3f0e27c-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "c3a47b97-4b42-4be7-a5ef-c33bd3f0e27c" (UID: "c3a47b97-4b42-4be7-a5ef-c33bd3f0e27c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 19 15:28:05.529193 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:28:05.529162 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c3a47b97-4b42-4be7-a5ef-c33bd3f0e27c-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "c3a47b97-4b42-4be7-a5ef-c33bd3f0e27c" (UID: "c3a47b97-4b42-4be7-a5ef-c33bd3f0e27c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 19 15:28:05.530139 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:28:05.530115 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c3a47b97-4b42-4be7-a5ef-c33bd3f0e27c-istio-data" (OuterVolumeSpecName: "istio-data") pod "c3a47b97-4b42-4be7-a5ef-c33bd3f0e27c" (UID: "c3a47b97-4b42-4be7-a5ef-c33bd3f0e27c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 19 15:28:05.530899 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:28:05.530875 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c3a47b97-4b42-4be7-a5ef-c33bd3f0e27c-istio-token" (OuterVolumeSpecName: "istio-token") pod "c3a47b97-4b42-4be7-a5ef-c33bd3f0e27c" (UID: "c3a47b97-4b42-4be7-a5ef-c33bd3f0e27c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 19 15:28:05.530899 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:28:05.530889 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c3a47b97-4b42-4be7-a5ef-c33bd3f0e27c-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "c3a47b97-4b42-4be7-a5ef-c33bd3f0e27c" (UID: "c3a47b97-4b42-4be7-a5ef-c33bd3f0e27c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 19 15:28:05.531194 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:28:05.531169 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c3a47b97-4b42-4be7-a5ef-c33bd3f0e27c-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "c3a47b97-4b42-4be7-a5ef-c33bd3f0e27c" (UID: "c3a47b97-4b42-4be7-a5ef-c33bd3f0e27c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 19 15:28:05.531194 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:28:05.531180 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c3a47b97-4b42-4be7-a5ef-c33bd3f0e27c-kube-api-access-77ktn" (OuterVolumeSpecName: "kube-api-access-77ktn") pod "c3a47b97-4b42-4be7-a5ef-c33bd3f0e27c" (UID: "c3a47b97-4b42-4be7-a5ef-c33bd3f0e27c"). InnerVolumeSpecName "kube-api-access-77ktn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 19 15:28:05.629969 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:28:05.629945 2558 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c3a47b97-4b42-4be7-a5ef-c33bd3f0e27c-istio-token\") on node \"ip-10-0-138-220.ec2.internal\" DevicePath \"\"" Jul 19 15:28:05.629969 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:28:05.629968 2558 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c3a47b97-4b42-4be7-a5ef-c33bd3f0e27c-istiod-ca-cert\") on node \"ip-10-0-138-220.ec2.internal\" DevicePath \"\"" Jul 19 15:28:05.630133 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:28:05.629977 2558 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c3a47b97-4b42-4be7-a5ef-c33bd3f0e27c-istio-data\") on node \"ip-10-0-138-220.ec2.internal\" DevicePath \"\"" Jul 19 15:28:05.630133 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:28:05.629984 2558 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c3a47b97-4b42-4be7-a5ef-c33bd3f0e27c-istio-podinfo\") on node \"ip-10-0-138-220.ec2.internal\" DevicePath \"\"" Jul 19 15:28:05.630133 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:28:05.629992 2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-77ktn\" (UniqueName: \"kubernetes.io/projected/c3a47b97-4b42-4be7-a5ef-c33bd3f0e27c-kube-api-access-77ktn\") on node \"ip-10-0-138-220.ec2.internal\" DevicePath \"\"" Jul 19 15:28:05.630133 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:28:05.630001 2558 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3a47b97-4b42-4be7-a5ef-c33bd3f0e27c-credential-socket\") on node \"ip-10-0-138-220.ec2.internal\" DevicePath \"\"" Jul 19 15:28:05.630133 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:28:05.630009 2558 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c3a47b97-4b42-4be7-a5ef-c33bd3f0e27c-workload-certs\") on node \"ip-10-0-138-220.ec2.internal\" DevicePath \"\"" Jul 19 15:28:05.630133 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:28:05.630016 2558 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c3a47b97-4b42-4be7-a5ef-c33bd3f0e27c-istio-envoy\") on node \"ip-10-0-138-220.ec2.internal\" DevicePath \"\"" Jul 19 15:28:05.737835 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:28:05.737805 2558 generic.go:356] "Generic (PLEG): container finished" podID="c3a47b97-4b42-4be7-a5ef-c33bd3f0e27c" containerID="3b2936eb3551af193519814fafbe73b3448b138982f9e21a7eb4af3fd8a7a92d" exitCode=0 Jul 19 15:28:05.737987 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:28:05.737856 2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-431757e3-gw-openshift-default-7df98f6f75-x2kzr" Jul 19 15:28:05.737987 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:28:05.737864 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-431757e3-gw-openshift-default-7df98f6f75-x2kzr" event={"ID":"c3a47b97-4b42-4be7-a5ef-c33bd3f0e27c","Type":"ContainerDied","Data":"3b2936eb3551af193519814fafbe73b3448b138982f9e21a7eb4af3fd8a7a92d"} Jul 19 15:28:05.737987 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:28:05.737904 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-431757e3-gw-openshift-default-7df98f6f75-x2kzr" event={"ID":"c3a47b97-4b42-4be7-a5ef-c33bd3f0e27c","Type":"ContainerDied","Data":"35fbb7a6c54fd2cf805e4d4e921ca5fcb4313eba2f1b16dfe9a5beb4cdf31a61"} Jul 19 15:28:05.737987 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:28:05.737925 2558 scope.go:117] "RemoveContainer" containerID="3b2936eb3551af193519814fafbe73b3448b138982f9e21a7eb4af3fd8a7a92d" Jul 19 15:28:05.746123 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:28:05.746103 2558 scope.go:117] "RemoveContainer" containerID="eda464ff1860b9218f1fd097159e639844a263fa43ad3b8f9bcd3ef260e13332" Jul 19 15:28:05.753534 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:28:05.753516 2558 scope.go:117] "RemoveContainer" containerID="3b2936eb3551af193519814fafbe73b3448b138982f9e21a7eb4af3fd8a7a92d" Jul 19 15:28:05.753822 ip-10-0-138-220 kubenswrapper[2558]: E0719 15:28:05.753798 2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3b2936eb3551af193519814fafbe73b3448b138982f9e21a7eb4af3fd8a7a92d\": container with ID starting with 3b2936eb3551af193519814fafbe73b3448b138982f9e21a7eb4af3fd8a7a92d not found: ID does not exist" containerID="3b2936eb3551af193519814fafbe73b3448b138982f9e21a7eb4af3fd8a7a92d" Jul 19 15:28:05.753878 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:28:05.753835 2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3b2936eb3551af193519814fafbe73b3448b138982f9e21a7eb4af3fd8a7a92d"} err="failed to get container status \"3b2936eb3551af193519814fafbe73b3448b138982f9e21a7eb4af3fd8a7a92d\": rpc error: code = NotFound desc = could not find container \"3b2936eb3551af193519814fafbe73b3448b138982f9e21a7eb4af3fd8a7a92d\": container with ID starting with 3b2936eb3551af193519814fafbe73b3448b138982f9e21a7eb4af3fd8a7a92d not found: ID does not exist" Jul 19 15:28:05.753927 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:28:05.753860 2558 scope.go:117] "RemoveContainer" containerID="eda464ff1860b9218f1fd097159e639844a263fa43ad3b8f9bcd3ef260e13332" Jul 19 15:28:05.754357 ip-10-0-138-220 kubenswrapper[2558]: E0719 15:28:05.754255 2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eda464ff1860b9218f1fd097159e639844a263fa43ad3b8f9bcd3ef260e13332\": container with ID starting with eda464ff1860b9218f1fd097159e639844a263fa43ad3b8f9bcd3ef260e13332 not found: ID does not exist" containerID="eda464ff1860b9218f1fd097159e639844a263fa43ad3b8f9bcd3ef260e13332" Jul 19 15:28:05.754357 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:28:05.754288 2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eda464ff1860b9218f1fd097159e639844a263fa43ad3b8f9bcd3ef260e13332"} err="failed to get container status \"eda464ff1860b9218f1fd097159e639844a263fa43ad3b8f9bcd3ef260e13332\": rpc error: code = NotFound desc = could not find container \"eda464ff1860b9218f1fd097159e639844a263fa43ad3b8f9bcd3ef260e13332\": container with ID starting with eda464ff1860b9218f1fd097159e639844a263fa43ad3b8f9bcd3ef260e13332 not found: ID does not exist" Jul 19 15:28:05.756455 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:28:05.756398 2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-431757e3-gw-openshift-default-7df98f6f75-x2kzr"] Jul 19 15:28:05.758099 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:28:05.758077 2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-migrate-431757e3-gw-openshift-default-7df98f6f75-x2kzr"] Jul 19 15:28:07.564201 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:28:07.564172 2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c3a47b97-4b42-4be7-a5ef-c33bd3f0e27c" path="/var/lib/kubelet/pods/c3a47b97-4b42-4be7-a5ef-c33bd3f0e27c/volumes" Jul 19 15:28:09.482837 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:28:09.482801 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-d4lll"] Jul 19 15:28:16.790835 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:28:16.790809 2558 generic.go:356] "Generic (PLEG): container finished" podID="1cbf9895-d90f-49bb-96c8-af951e03ed63" containerID="ea532bf9034d0866789a27dd5d9b4dee57fe676be4b42cb63850ee03544c0289" exitCode=0 Jul 19 15:28:16.791206 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:28:16.790889 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-a4b8edd0-openshift-default-5db5d69955-5fll7" event={"ID":"1cbf9895-d90f-49bb-96c8-af951e03ed63","Type":"ContainerDied","Data":"ea532bf9034d0866789a27dd5d9b4dee57fe676be4b42cb63850ee03544c0289"} Jul 19 15:28:16.791282 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:28:16.791266 2558 scope.go:117] "RemoveContainer" containerID="ea532bf9034d0866789a27dd5d9b4dee57fe676be4b42cb63850ee03544c0289" Jul 19 15:28:16.792225 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:28:16.792194 2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 19 15:28:16.792323 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:28:16.792279 2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 19 15:28:16.792378 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:28:16.792328 2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 19 15:28:17.724661 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:28:17.724629 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-a4b8edd0-openshift-default-5db5d69955-5fll7" Jul 19 15:28:17.797268 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:28:17.797235 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-a4b8edd0-openshift-default-5db5d69955-5fll7" event={"ID":"1cbf9895-d90f-49bb-96c8-af951e03ed63","Type":"ContainerStarted","Data":"3cb2fd9d680fa6388c4cb11f5ca007b8687617cf7571b68c50209a72d968d299"} Jul 19 15:28:18.372066 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:28:18.371727 2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-a4b8edd0-openshift-default-5db5d69955-5fll7" Jul 19 15:28:18.372718 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:28:18.372692 2558 patch_prober.go:28] interesting pod/e2e-ait-a4b8edd0-openshift-default-5db5d69955-5fll7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.28:15021/healthz/ready\": dial tcp 10.133.0.28:15021: connect: connection refused" start-of-body= Jul 19 15:28:18.372777 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:28:18.372753 2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-a4b8edd0-openshift-default-5db5d69955-5fll7" podUID="1cbf9895-d90f-49bb-96c8-af951e03ed63" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.28:15021/healthz/ready\": dial tcp 10.133.0.28:15021: connect: connection refused" Jul 19 15:28:19.372317 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:28:19.372275 2558 patch_prober.go:28] interesting pod/e2e-ait-a4b8edd0-openshift-default-5db5d69955-5fll7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.28:15021/healthz/ready\": dial tcp 10.133.0.28:15021: connect: connection refused" start-of-body= Jul 19 15:28:19.372714 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:28:19.372338 2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-a4b8edd0-openshift-default-5db5d69955-5fll7" podUID="1cbf9895-d90f-49bb-96c8-af951e03ed63" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.28:15021/healthz/ready\": dial tcp 10.133.0.28:15021: connect: connection refused" Jul 19 15:28:20.372644 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:28:20.372606 2558 patch_prober.go:28] interesting pod/e2e-ait-a4b8edd0-openshift-default-5db5d69955-5fll7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.28:15021/healthz/ready\": dial tcp 10.133.0.28:15021: connect: connection refused" start-of-body= Jul 19 15:28:20.373049 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:28:20.372674 2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-a4b8edd0-openshift-default-5db5d69955-5fll7" podUID="1cbf9895-d90f-49bb-96c8-af951e03ed63" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.28:15021/healthz/ready\": dial tcp 10.133.0.28:15021: connect: connection refused" Jul 19 15:28:21.371816 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:28:21.371758 2558 patch_prober.go:28] interesting pod/e2e-ait-a4b8edd0-openshift-default-5db5d69955-5fll7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.28:15021/healthz/ready\": dial tcp 10.133.0.28:15021: connect: connection refused" start-of-body= Jul 19 15:28:21.372012 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:28:21.371833 2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-a4b8edd0-openshift-default-5db5d69955-5fll7" podUID="1cbf9895-d90f-49bb-96c8-af951e03ed63" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.28:15021/healthz/ready\": dial tcp 10.133.0.28:15021: connect: connection refused" Jul 19 15:28:22.375570 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:28:22.375537 2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-a4b8edd0-openshift-default-5db5d69955-5fll7" Jul 19 15:28:22.375993 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:28:22.375832 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-a4b8edd0-openshift-default-5db5d69955-5fll7" Jul 19 15:28:22.376599 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:28:22.376582 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-a4b8edd0-openshift-default-5db5d69955-5fll7" Jul 19 15:28:28.872383 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:28:28.872346 2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a4b8edd0-npsxf"] Jul 19 15:28:28.872865 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:28:28.872643 2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c3a47b97-4b42-4be7-a5ef-c33bd3f0e27c" containerName="istio-proxy" Jul 19 15:28:28.872865 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:28:28.872655 2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="c3a47b97-4b42-4be7-a5ef-c33bd3f0e27c" containerName="istio-proxy" Jul 19 15:28:28.872865 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:28:28.872667 2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c3a47b97-4b42-4be7-a5ef-c33bd3f0e27c" containerName="istio-proxy" Jul 19 15:28:28.872865 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:28:28.872672 2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="c3a47b97-4b42-4be7-a5ef-c33bd3f0e27c" containerName="istio-proxy" Jul 19 15:28:28.872865 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:28:28.872722 2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="c3a47b97-4b42-4be7-a5ef-c33bd3f0e27c" containerName="istio-proxy" Jul 19 15:28:28.872865 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:28:28.872729 2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="c3a47b97-4b42-4be7-a5ef-c33bd3f0e27c" containerName="istio-proxy" Jul 19 15:28:28.874606 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:28:28.874583 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a4b8edd0-npsxf" Jul 19 15:28:28.876858 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:28:28.876834 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 19 15:28:28.876962 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:28:28.876843 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qddbb\"" Jul 19 15:28:28.882799 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:28:28.882775 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a4b8edd0-npsxf"] Jul 19 15:28:29.029120 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:28:29.029088 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/19394fb6-98e4-4b72-8e03-cb41eea2a786-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-a4b8edd0-npsxf\" (UID: \"19394fb6-98e4-4b72-8e03-cb41eea2a786\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a4b8edd0-npsxf" Jul 19 15:28:29.130419 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:28:29.130344 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/19394fb6-98e4-4b72-8e03-cb41eea2a786-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-a4b8edd0-npsxf\" (UID: \"19394fb6-98e4-4b72-8e03-cb41eea2a786\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a4b8edd0-npsxf" Jul 19 15:28:29.130935 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:28:29.130917 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/19394fb6-98e4-4b72-8e03-cb41eea2a786-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-a4b8edd0-npsxf\" (UID: \"19394fb6-98e4-4b72-8e03-cb41eea2a786\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a4b8edd0-npsxf" Jul 19 15:28:29.186006 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:28:29.185975 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a4b8edd0-npsxf" Jul 19 15:28:29.316573 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:28:29.316544 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a4b8edd0-npsxf"] Jul 19 15:28:29.319753 ip-10-0-138-220 kubenswrapper[2558]: W0719 15:28:29.319727 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod19394fb6_98e4_4b72_8e03_cb41eea2a786.slice/crio-9ee722a9d51171363b681312d1074d9a66cc7155d5fbeb05aef2ca56e5123f09 WatchSource:0}: Error finding container 9ee722a9d51171363b681312d1074d9a66cc7155d5fbeb05aef2ca56e5123f09: Status 404 returned error can't find the container with id 9ee722a9d51171363b681312d1074d9a66cc7155d5fbeb05aef2ca56e5123f09 Jul 19 15:28:29.848618 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:28:29.848521 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a4b8edd0-npsxf" event={"ID":"19394fb6-98e4-4b72-8e03-cb41eea2a786","Type":"ContainerStarted","Data":"ab3f88024c01f778a6108901b485ab561fe9b43d8dd4b8e40193e2b6553dee83"} Jul 19 15:28:29.848618 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:28:29.848579 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a4b8edd0-npsxf" event={"ID":"19394fb6-98e4-4b72-8e03-cb41eea2a786","Type":"ContainerStarted","Data":"9ee722a9d51171363b681312d1074d9a66cc7155d5fbeb05aef2ca56e5123f09"} Jul 19 15:28:29.863203 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:28:29.863159 2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a4b8edd0-npsxf" podStartSLOduration=1.863145905 podStartE2EDuration="1.863145905s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-19 15:28:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-19 15:28:29.8620696 +0000 UTC m=+1850.783772313" watchObservedRunningTime="2026-07-19 15:28:29.863145905 +0000 UTC m=+1850.784848619" Jul 19 15:28:30.853966 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:28:30.853932 2558 generic.go:356] "Generic (PLEG): container finished" podID="19394fb6-98e4-4b72-8e03-cb41eea2a786" containerID="ab3f88024c01f778a6108901b485ab561fe9b43d8dd4b8e40193e2b6553dee83" exitCode=0 Jul 19 15:28:30.854372 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:28:30.854015 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a4b8edd0-npsxf" event={"ID":"19394fb6-98e4-4b72-8e03-cb41eea2a786","Type":"ContainerDied","Data":"ab3f88024c01f778a6108901b485ab561fe9b43d8dd4b8e40193e2b6553dee83"} Jul 19 15:28:31.990367 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:28:31.990342 2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a4b8edd0-npsxf" Jul 19 15:28:32.155777 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:28:32.155746 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/19394fb6-98e4-4b72-8e03-cb41eea2a786-maas-api-service-ca\") pod \"19394fb6-98e4-4b72-8e03-cb41eea2a786\" (UID: \"19394fb6-98e4-4b72-8e03-cb41eea2a786\") " Jul 19 15:28:32.156138 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:28:32.156114 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/19394fb6-98e4-4b72-8e03-cb41eea2a786-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "19394fb6-98e4-4b72-8e03-cb41eea2a786" (UID: "19394fb6-98e4-4b72-8e03-cb41eea2a786"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 19 15:28:32.257282 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:28:32.257243 2558 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/19394fb6-98e4-4b72-8e03-cb41eea2a786-maas-api-service-ca\") on node \"ip-10-0-138-220.ec2.internal\" DevicePath \"\"" Jul 19 15:28:32.862749 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:28:32.862713 2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a4b8edd0-npsxf" Jul 19 15:28:32.862749 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:28:32.862727 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a4b8edd0-npsxf" event={"ID":"19394fb6-98e4-4b72-8e03-cb41eea2a786","Type":"ContainerDied","Data":"9ee722a9d51171363b681312d1074d9a66cc7155d5fbeb05aef2ca56e5123f09"} Jul 19 15:28:32.863005 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:28:32.862772 2558 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9ee722a9d51171363b681312d1074d9a66cc7155d5fbeb05aef2ca56e5123f09" Jul 19 15:28:42.086830 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:28:42.086794 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-d4lll"] Jul 19 15:28:43.964893 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:28:43.964851 2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a4b8edd0-npsxf"] Jul 19 15:28:43.967315 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:28:43.967285 2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a4b8edd0-npsxf"] Jul 19 15:28:45.152017 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:28:45.151984 2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-a4b8edd0-openshift-default-5db5d69955-5fll7"] Jul 19 15:28:45.152457 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:28:45.152260 2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-a4b8edd0-openshift-default-5db5d69955-5fll7" podUID="1cbf9895-d90f-49bb-96c8-af951e03ed63" containerName="istio-proxy" containerID="cri-o://3cb2fd9d680fa6388c4cb11f5ca007b8687617cf7571b68c50209a72d968d299" gracePeriod=30 Jul 19 15:28:45.564224 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:28:45.564131 2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="19394fb6-98e4-4b72-8e03-cb41eea2a786" path="/var/lib/kubelet/pods/19394fb6-98e4-4b72-8e03-cb41eea2a786/volumes" Jul 19 15:28:45.903295 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:28:45.903260 2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-derive-e7bec00b-openshift-default-777c96ff67-v84vb"] Jul 19 15:28:45.903693 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:28:45.903679 2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="19394fb6-98e4-4b72-8e03-cb41eea2a786" containerName="revoke-keys" Jul 19 15:28:45.903749 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:28:45.903697 2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="19394fb6-98e4-4b72-8e03-cb41eea2a786" containerName="revoke-keys" Jul 19 15:28:45.903783 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:28:45.903759 2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="19394fb6-98e4-4b72-8e03-cb41eea2a786" containerName="revoke-keys" Jul 19 15:28:45.906967 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:28:45.906944 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-e7bec00b-openshift-default-777c96ff67-v84vb" Jul 19 15:28:45.909187 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:28:45.909166 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-derive-e7bec00b-openshift-default-dockercfg-62rmv\"" Jul 19 15:28:45.919071 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:28:45.919040 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-e7bec00b-openshift-default-777c96ff67-v84vb"] Jul 19 15:28:45.966728 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:28:45.966694 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mjd55\" (UniqueName: \"kubernetes.io/projected/6276272e-75e7-4d75-9671-33902770dd53-kube-api-access-mjd55\") pod \"e2e-derive-e7bec00b-openshift-default-777c96ff67-v84vb\" (UID: \"6276272e-75e7-4d75-9671-33902770dd53\") " pod="openshift-ingress/e2e-derive-e7bec00b-openshift-default-777c96ff67-v84vb" Jul 19 15:28:45.966886 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:28:45.966740 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6276272e-75e7-4d75-9671-33902770dd53-istio-podinfo\") pod \"e2e-derive-e7bec00b-openshift-default-777c96ff67-v84vb\" (UID: \"6276272e-75e7-4d75-9671-33902770dd53\") " pod="openshift-ingress/e2e-derive-e7bec00b-openshift-default-777c96ff67-v84vb" Jul 19 15:28:45.966886 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:28:45.966795 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6276272e-75e7-4d75-9671-33902770dd53-workload-socket\") pod \"e2e-derive-e7bec00b-openshift-default-777c96ff67-v84vb\" (UID: \"6276272e-75e7-4d75-9671-33902770dd53\") " pod="openshift-ingress/e2e-derive-e7bec00b-openshift-default-777c96ff67-v84vb" Jul 19 15:28:45.966886 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:28:45.966811 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6276272e-75e7-4d75-9671-33902770dd53-istio-token\") pod \"e2e-derive-e7bec00b-openshift-default-777c96ff67-v84vb\" (UID: \"6276272e-75e7-4d75-9671-33902770dd53\") " pod="openshift-ingress/e2e-derive-e7bec00b-openshift-default-777c96ff67-v84vb" Jul 19 15:28:45.966886 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:28:45.966844 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6276272e-75e7-4d75-9671-33902770dd53-istio-data\") pod \"e2e-derive-e7bec00b-openshift-default-777c96ff67-v84vb\" (UID: \"6276272e-75e7-4d75-9671-33902770dd53\") " pod="openshift-ingress/e2e-derive-e7bec00b-openshift-default-777c96ff67-v84vb" Jul 19 15:28:45.966886 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:28:45.966869 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6276272e-75e7-4d75-9671-33902770dd53-istiod-ca-cert\") pod \"e2e-derive-e7bec00b-openshift-default-777c96ff67-v84vb\" (UID: \"6276272e-75e7-4d75-9671-33902770dd53\") " pod="openshift-ingress/e2e-derive-e7bec00b-openshift-default-777c96ff67-v84vb" Jul 19 15:28:45.967119 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:28:45.966891 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6276272e-75e7-4d75-9671-33902770dd53-istio-envoy\") pod \"e2e-derive-e7bec00b-openshift-default-777c96ff67-v84vb\" (UID: \"6276272e-75e7-4d75-9671-33902770dd53\") " pod="openshift-ingress/e2e-derive-e7bec00b-openshift-default-777c96ff67-v84vb" Jul 19 15:28:45.967119 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:28:45.966975 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6276272e-75e7-4d75-9671-33902770dd53-credential-socket\") pod \"e2e-derive-e7bec00b-openshift-default-777c96ff67-v84vb\" (UID: \"6276272e-75e7-4d75-9671-33902770dd53\") " pod="openshift-ingress/e2e-derive-e7bec00b-openshift-default-777c96ff67-v84vb" Jul 19 15:28:45.967119 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:28:45.967005 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6276272e-75e7-4d75-9671-33902770dd53-workload-certs\") pod \"e2e-derive-e7bec00b-openshift-default-777c96ff67-v84vb\" (UID: \"6276272e-75e7-4d75-9671-33902770dd53\") " pod="openshift-ingress/e2e-derive-e7bec00b-openshift-default-777c96ff67-v84vb" Jul 19 15:28:46.067644 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:28:46.067613 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6276272e-75e7-4d75-9671-33902770dd53-workload-certs\") pod \"e2e-derive-e7bec00b-openshift-default-777c96ff67-v84vb\" (UID: \"6276272e-75e7-4d75-9671-33902770dd53\") " pod="openshift-ingress/e2e-derive-e7bec00b-openshift-default-777c96ff67-v84vb" Jul 19 15:28:46.067827 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:28:46.067651 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mjd55\" (UniqueName: \"kubernetes.io/projected/6276272e-75e7-4d75-9671-33902770dd53-kube-api-access-mjd55\") pod \"e2e-derive-e7bec00b-openshift-default-777c96ff67-v84vb\" (UID: \"6276272e-75e7-4d75-9671-33902770dd53\") " pod="openshift-ingress/e2e-derive-e7bec00b-openshift-default-777c96ff67-v84vb" Jul 19 15:28:46.067827 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:28:46.067676 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6276272e-75e7-4d75-9671-33902770dd53-istio-podinfo\") pod \"e2e-derive-e7bec00b-openshift-default-777c96ff67-v84vb\" (UID: \"6276272e-75e7-4d75-9671-33902770dd53\") " pod="openshift-ingress/e2e-derive-e7bec00b-openshift-default-777c96ff67-v84vb" Jul 19 15:28:46.067827 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:28:46.067699 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6276272e-75e7-4d75-9671-33902770dd53-workload-socket\") pod \"e2e-derive-e7bec00b-openshift-default-777c96ff67-v84vb\" (UID: \"6276272e-75e7-4d75-9671-33902770dd53\") " pod="openshift-ingress/e2e-derive-e7bec00b-openshift-default-777c96ff67-v84vb" Jul 19 15:28:46.067827 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:28:46.067725 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6276272e-75e7-4d75-9671-33902770dd53-istio-token\") pod \"e2e-derive-e7bec00b-openshift-default-777c96ff67-v84vb\" (UID: \"6276272e-75e7-4d75-9671-33902770dd53\") " pod="openshift-ingress/e2e-derive-e7bec00b-openshift-default-777c96ff67-v84vb" Jul 19 15:28:46.067827 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:28:46.067749 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6276272e-75e7-4d75-9671-33902770dd53-istio-data\") pod \"e2e-derive-e7bec00b-openshift-default-777c96ff67-v84vb\" (UID: \"6276272e-75e7-4d75-9671-33902770dd53\") " pod="openshift-ingress/e2e-derive-e7bec00b-openshift-default-777c96ff67-v84vb" Jul 19 15:28:46.067827 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:28:46.067769 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6276272e-75e7-4d75-9671-33902770dd53-istiod-ca-cert\") pod \"e2e-derive-e7bec00b-openshift-default-777c96ff67-v84vb\" (UID: \"6276272e-75e7-4d75-9671-33902770dd53\") " pod="openshift-ingress/e2e-derive-e7bec00b-openshift-default-777c96ff67-v84vb" Jul 19 15:28:46.067827 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:28:46.067795 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6276272e-75e7-4d75-9671-33902770dd53-istio-envoy\") pod \"e2e-derive-e7bec00b-openshift-default-777c96ff67-v84vb\" (UID: \"6276272e-75e7-4d75-9671-33902770dd53\") " pod="openshift-ingress/e2e-derive-e7bec00b-openshift-default-777c96ff67-v84vb" Jul 19 15:28:46.068240 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:28:46.067904 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6276272e-75e7-4d75-9671-33902770dd53-credential-socket\") pod \"e2e-derive-e7bec00b-openshift-default-777c96ff67-v84vb\" (UID: \"6276272e-75e7-4d75-9671-33902770dd53\") " pod="openshift-ingress/e2e-derive-e7bec00b-openshift-default-777c96ff67-v84vb" Jul 19 15:28:46.068240 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:28:46.068077 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6276272e-75e7-4d75-9671-33902770dd53-workload-certs\") pod \"e2e-derive-e7bec00b-openshift-default-777c96ff67-v84vb\" (UID: \"6276272e-75e7-4d75-9671-33902770dd53\") " pod="openshift-ingress/e2e-derive-e7bec00b-openshift-default-777c96ff67-v84vb" Jul 19 15:28:46.068240 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:28:46.068113 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6276272e-75e7-4d75-9671-33902770dd53-workload-socket\") pod \"e2e-derive-e7bec00b-openshift-default-777c96ff67-v84vb\" (UID: \"6276272e-75e7-4d75-9671-33902770dd53\") " pod="openshift-ingress/e2e-derive-e7bec00b-openshift-default-777c96ff67-v84vb" Jul 19 15:28:46.068240 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:28:46.068246 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6276272e-75e7-4d75-9671-33902770dd53-istio-data\") pod \"e2e-derive-e7bec00b-openshift-default-777c96ff67-v84vb\" (UID: \"6276272e-75e7-4d75-9671-33902770dd53\") " pod="openshift-ingress/e2e-derive-e7bec00b-openshift-default-777c96ff67-v84vb" Jul 19 15:28:46.068526 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:28:46.068473 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6276272e-75e7-4d75-9671-33902770dd53-credential-socket\") pod \"e2e-derive-e7bec00b-openshift-default-777c96ff67-v84vb\" (UID: \"6276272e-75e7-4d75-9671-33902770dd53\") " pod="openshift-ingress/e2e-derive-e7bec00b-openshift-default-777c96ff67-v84vb" Jul 19 15:28:46.068588 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:28:46.068559 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6276272e-75e7-4d75-9671-33902770dd53-istiod-ca-cert\") pod \"e2e-derive-e7bec00b-openshift-default-777c96ff67-v84vb\" (UID: \"6276272e-75e7-4d75-9671-33902770dd53\") " pod="openshift-ingress/e2e-derive-e7bec00b-openshift-default-777c96ff67-v84vb" Jul 19 15:28:46.070328 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:28:46.070294 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6276272e-75e7-4d75-9671-33902770dd53-istio-envoy\") pod \"e2e-derive-e7bec00b-openshift-default-777c96ff67-v84vb\" (UID: \"6276272e-75e7-4d75-9671-33902770dd53\") " pod="openshift-ingress/e2e-derive-e7bec00b-openshift-default-777c96ff67-v84vb" Jul 19 15:28:46.070873 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:28:46.070850 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6276272e-75e7-4d75-9671-33902770dd53-istio-podinfo\") pod \"e2e-derive-e7bec00b-openshift-default-777c96ff67-v84vb\" (UID: \"6276272e-75e7-4d75-9671-33902770dd53\") " pod="openshift-ingress/e2e-derive-e7bec00b-openshift-default-777c96ff67-v84vb" Jul 19 15:28:46.075535 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:28:46.075510 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6276272e-75e7-4d75-9671-33902770dd53-istio-token\") pod \"e2e-derive-e7bec00b-openshift-default-777c96ff67-v84vb\" (UID: \"6276272e-75e7-4d75-9671-33902770dd53\") " pod="openshift-ingress/e2e-derive-e7bec00b-openshift-default-777c96ff67-v84vb" Jul 19 15:28:46.075653 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:28:46.075609 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mjd55\" (UniqueName: \"kubernetes.io/projected/6276272e-75e7-4d75-9671-33902770dd53-kube-api-access-mjd55\") pod \"e2e-derive-e7bec00b-openshift-default-777c96ff67-v84vb\" (UID: \"6276272e-75e7-4d75-9671-33902770dd53\") " pod="openshift-ingress/e2e-derive-e7bec00b-openshift-default-777c96ff67-v84vb" Jul 19 15:28:46.218665 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:28:46.218575 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-e7bec00b-openshift-default-777c96ff67-v84vb" Jul 19 15:28:46.373480 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:28:46.373455 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-e7bec00b-openshift-default-777c96ff67-v84vb"] Jul 19 15:28:46.375785 ip-10-0-138-220 kubenswrapper[2558]: W0719 15:28:46.375754 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6276272e_75e7_4d75_9671_33902770dd53.slice/crio-b9486247caf2c8338ac38c93fbd7a52db96a51cb60e4100520aaa6c81ac90def WatchSource:0}: Error finding container b9486247caf2c8338ac38c93fbd7a52db96a51cb60e4100520aaa6c81ac90def: Status 404 returned error can't find the container with id b9486247caf2c8338ac38c93fbd7a52db96a51cb60e4100520aaa6c81ac90def Jul 19 15:28:46.377804 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:28:46.377774 2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 19 15:28:46.377881 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:28:46.377842 2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 19 15:28:46.377881 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:28:46.377870 2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 19 15:28:46.922187 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:28:46.922145 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-e7bec00b-openshift-default-777c96ff67-v84vb" event={"ID":"6276272e-75e7-4d75-9671-33902770dd53","Type":"ContainerStarted","Data":"9817f27b66c36940d3d23c8ffd46bc2dabc6ca83c0464494f2e2d8d1596bf087"} Jul 19 15:28:46.922357 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:28:46.922235 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-e7bec00b-openshift-default-777c96ff67-v84vb" event={"ID":"6276272e-75e7-4d75-9671-33902770dd53","Type":"ContainerStarted","Data":"b9486247caf2c8338ac38c93fbd7a52db96a51cb60e4100520aaa6c81ac90def"} Jul 19 15:28:46.941921 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:28:46.941874 2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-derive-e7bec00b-openshift-default-777c96ff67-v84vb" podStartSLOduration=1.941859555 podStartE2EDuration="1.941859555s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-19 15:28:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-19 15:28:46.940669418 +0000 UTC m=+1867.862372133" watchObservedRunningTime="2026-07-19 15:28:46.941859555 +0000 UTC m=+1867.863562269" Jul 19 15:28:47.219134 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:28:47.219014 2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-e7bec00b-openshift-default-777c96ff67-v84vb" Jul 19 15:28:47.223709 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:28:47.223684 2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-e7bec00b-openshift-default-777c96ff67-v84vb" Jul 19 15:28:47.926356 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:28:47.926326 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-e7bec00b-openshift-default-777c96ff67-v84vb" Jul 19 15:28:47.927417 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:28:47.927394 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-e7bec00b-openshift-default-777c96ff67-v84vb" Jul 19 15:28:50.392736 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:28:50.392711 2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-a4b8edd0-openshift-default-5db5d69955-5fll7" Jul 19 15:28:50.405696 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:28:50.405668 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1cbf9895-d90f-49bb-96c8-af951e03ed63-istio-token\") pod \"1cbf9895-d90f-49bb-96c8-af951e03ed63\" (UID: \"1cbf9895-d90f-49bb-96c8-af951e03ed63\") " Jul 19 15:28:50.405841 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:28:50.405709 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1cbf9895-d90f-49bb-96c8-af951e03ed63-istio-podinfo\") pod \"1cbf9895-d90f-49bb-96c8-af951e03ed63\" (UID: \"1cbf9895-d90f-49bb-96c8-af951e03ed63\") " Jul 19 15:28:50.405841 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:28:50.405736 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7txww\" (UniqueName: \"kubernetes.io/projected/1cbf9895-d90f-49bb-96c8-af951e03ed63-kube-api-access-7txww\") pod \"1cbf9895-d90f-49bb-96c8-af951e03ed63\" (UID: \"1cbf9895-d90f-49bb-96c8-af951e03ed63\") " Jul 19 15:28:50.405841 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:28:50.405758 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1cbf9895-d90f-49bb-96c8-af951e03ed63-istio-envoy\") pod \"1cbf9895-d90f-49bb-96c8-af951e03ed63\" (UID: \"1cbf9895-d90f-49bb-96c8-af951e03ed63\") " Jul 19 15:28:50.405975 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:28:50.405901 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1cbf9895-d90f-49bb-96c8-af951e03ed63-istiod-ca-cert\") pod \"1cbf9895-d90f-49bb-96c8-af951e03ed63\" (UID: \"1cbf9895-d90f-49bb-96c8-af951e03ed63\") " Jul 19 15:28:50.405975 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:28:50.405945 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1cbf9895-d90f-49bb-96c8-af951e03ed63-istio-data\") pod \"1cbf9895-d90f-49bb-96c8-af951e03ed63\" (UID: \"1cbf9895-d90f-49bb-96c8-af951e03ed63\") " Jul 19 15:28:50.406114 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:28:50.405979 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1cbf9895-d90f-49bb-96c8-af951e03ed63-workload-socket\") pod \"1cbf9895-d90f-49bb-96c8-af951e03ed63\" (UID: \"1cbf9895-d90f-49bb-96c8-af951e03ed63\") " Jul 19 15:28:50.406114 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:28:50.406058 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1cbf9895-d90f-49bb-96c8-af951e03ed63-credential-socket\") pod \"1cbf9895-d90f-49bb-96c8-af951e03ed63\" (UID: \"1cbf9895-d90f-49bb-96c8-af951e03ed63\") " Jul 19 15:28:50.406114 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:28:50.406087 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1cbf9895-d90f-49bb-96c8-af951e03ed63-workload-certs\") pod \"1cbf9895-d90f-49bb-96c8-af951e03ed63\" (UID: \"1cbf9895-d90f-49bb-96c8-af951e03ed63\") " Jul 19 15:28:50.406370 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:28:50.406333 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1cbf9895-d90f-49bb-96c8-af951e03ed63-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "1cbf9895-d90f-49bb-96c8-af951e03ed63" (UID: "1cbf9895-d90f-49bb-96c8-af951e03ed63"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 19 15:28:50.406370 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:28:50.406356 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1cbf9895-d90f-49bb-96c8-af951e03ed63-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "1cbf9895-d90f-49bb-96c8-af951e03ed63" (UID: "1cbf9895-d90f-49bb-96c8-af951e03ed63"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 19 15:28:50.406592 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:28:50.406549 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1cbf9895-d90f-49bb-96c8-af951e03ed63-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "1cbf9895-d90f-49bb-96c8-af951e03ed63" (UID: "1cbf9895-d90f-49bb-96c8-af951e03ed63"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 19 15:28:50.406729 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:28:50.406707 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1cbf9895-d90f-49bb-96c8-af951e03ed63-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "1cbf9895-d90f-49bb-96c8-af951e03ed63" (UID: "1cbf9895-d90f-49bb-96c8-af951e03ed63"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 19 15:28:50.407437 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:28:50.407412 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1cbf9895-d90f-49bb-96c8-af951e03ed63-istio-data" (OuterVolumeSpecName: "istio-data") pod "1cbf9895-d90f-49bb-96c8-af951e03ed63" (UID: "1cbf9895-d90f-49bb-96c8-af951e03ed63"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 19 15:28:50.408080 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:28:50.408056 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1cbf9895-d90f-49bb-96c8-af951e03ed63-istio-token" (OuterVolumeSpecName: "istio-token") pod "1cbf9895-d90f-49bb-96c8-af951e03ed63" (UID: "1cbf9895-d90f-49bb-96c8-af951e03ed63"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 19 15:28:50.408182 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:28:50.408146 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1cbf9895-d90f-49bb-96c8-af951e03ed63-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "1cbf9895-d90f-49bb-96c8-af951e03ed63" (UID: "1cbf9895-d90f-49bb-96c8-af951e03ed63"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 19 15:28:50.408709 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:28:50.408685 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1cbf9895-d90f-49bb-96c8-af951e03ed63-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "1cbf9895-d90f-49bb-96c8-af951e03ed63" (UID: "1cbf9895-d90f-49bb-96c8-af951e03ed63"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 19 15:28:50.408850 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:28:50.408836 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1cbf9895-d90f-49bb-96c8-af951e03ed63-kube-api-access-7txww" (OuterVolumeSpecName: "kube-api-access-7txww") pod "1cbf9895-d90f-49bb-96c8-af951e03ed63" (UID: "1cbf9895-d90f-49bb-96c8-af951e03ed63"). InnerVolumeSpecName "kube-api-access-7txww". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 19 15:28:50.507637 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:28:50.507574 2558 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1cbf9895-d90f-49bb-96c8-af951e03ed63-istio-token\") on node \"ip-10-0-138-220.ec2.internal\" DevicePath \"\"" Jul 19 15:28:50.507637 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:28:50.507598 2558 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1cbf9895-d90f-49bb-96c8-af951e03ed63-istio-podinfo\") on node \"ip-10-0-138-220.ec2.internal\" DevicePath \"\"" Jul 19 15:28:50.507637 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:28:50.507607 2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7txww\" (UniqueName: \"kubernetes.io/projected/1cbf9895-d90f-49bb-96c8-af951e03ed63-kube-api-access-7txww\") on node \"ip-10-0-138-220.ec2.internal\" DevicePath \"\"" Jul 19 15:28:50.507637 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:28:50.507633 2558 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1cbf9895-d90f-49bb-96c8-af951e03ed63-istio-envoy\") on node \"ip-10-0-138-220.ec2.internal\" DevicePath \"\"" Jul 19 15:28:50.507826 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:28:50.507643 2558 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1cbf9895-d90f-49bb-96c8-af951e03ed63-istiod-ca-cert\") on node \"ip-10-0-138-220.ec2.internal\" DevicePath \"\"" Jul 19 15:28:50.507826 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:28:50.507653 2558 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1cbf9895-d90f-49bb-96c8-af951e03ed63-istio-data\") on node \"ip-10-0-138-220.ec2.internal\" DevicePath \"\"" Jul 19 15:28:50.507826 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:28:50.507661 2558 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1cbf9895-d90f-49bb-96c8-af951e03ed63-workload-socket\") on node \"ip-10-0-138-220.ec2.internal\" DevicePath \"\"" Jul 19 15:28:50.507826 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:28:50.507669 2558 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1cbf9895-d90f-49bb-96c8-af951e03ed63-credential-socket\") on node \"ip-10-0-138-220.ec2.internal\" DevicePath \"\"" Jul 19 15:28:50.507826 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:28:50.507677 2558 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1cbf9895-d90f-49bb-96c8-af951e03ed63-workload-certs\") on node \"ip-10-0-138-220.ec2.internal\" DevicePath \"\"" Jul 19 15:28:50.939827 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:28:50.939793 2558 generic.go:356] "Generic (PLEG): container finished" podID="1cbf9895-d90f-49bb-96c8-af951e03ed63" containerID="3cb2fd9d680fa6388c4cb11f5ca007b8687617cf7571b68c50209a72d968d299" exitCode=0 Jul 19 15:28:50.939982 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:28:50.939844 2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-a4b8edd0-openshift-default-5db5d69955-5fll7" Jul 19 15:28:50.939982 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:28:50.939843 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-a4b8edd0-openshift-default-5db5d69955-5fll7" event={"ID":"1cbf9895-d90f-49bb-96c8-af951e03ed63","Type":"ContainerDied","Data":"3cb2fd9d680fa6388c4cb11f5ca007b8687617cf7571b68c50209a72d968d299"} Jul 19 15:28:50.939982 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:28:50.939970 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-a4b8edd0-openshift-default-5db5d69955-5fll7" event={"ID":"1cbf9895-d90f-49bb-96c8-af951e03ed63","Type":"ContainerDied","Data":"76f69c5bbbb968841c38c66fc4487d7970790ad8d9d9fa6b6584655bb8a31477"} Jul 19 15:28:50.940114 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:28:50.939999 2558 scope.go:117] "RemoveContainer" containerID="3cb2fd9d680fa6388c4cb11f5ca007b8687617cf7571b68c50209a72d968d299" Jul 19 15:28:50.949153 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:28:50.949138 2558 scope.go:117] "RemoveContainer" containerID="ea532bf9034d0866789a27dd5d9b4dee57fe676be4b42cb63850ee03544c0289" Jul 19 15:28:50.956493 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:28:50.956475 2558 scope.go:117] "RemoveContainer" containerID="3cb2fd9d680fa6388c4cb11f5ca007b8687617cf7571b68c50209a72d968d299" Jul 19 15:28:50.956711 ip-10-0-138-220 kubenswrapper[2558]: E0719 15:28:50.956690 2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3cb2fd9d680fa6388c4cb11f5ca007b8687617cf7571b68c50209a72d968d299\": container with ID starting with 3cb2fd9d680fa6388c4cb11f5ca007b8687617cf7571b68c50209a72d968d299 not found: ID does not exist" containerID="3cb2fd9d680fa6388c4cb11f5ca007b8687617cf7571b68c50209a72d968d299" Jul 19 15:28:50.956754 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:28:50.956720 2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3cb2fd9d680fa6388c4cb11f5ca007b8687617cf7571b68c50209a72d968d299"} err="failed to get container status \"3cb2fd9d680fa6388c4cb11f5ca007b8687617cf7571b68c50209a72d968d299\": rpc error: code = NotFound desc = could not find container \"3cb2fd9d680fa6388c4cb11f5ca007b8687617cf7571b68c50209a72d968d299\": container with ID starting with 3cb2fd9d680fa6388c4cb11f5ca007b8687617cf7571b68c50209a72d968d299 not found: ID does not exist" Jul 19 15:28:50.956754 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:28:50.956738 2558 scope.go:117] "RemoveContainer" containerID="ea532bf9034d0866789a27dd5d9b4dee57fe676be4b42cb63850ee03544c0289" Jul 19 15:28:50.956983 ip-10-0-138-220 kubenswrapper[2558]: E0719 15:28:50.956966 2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ea532bf9034d0866789a27dd5d9b4dee57fe676be4b42cb63850ee03544c0289\": container with ID starting with ea532bf9034d0866789a27dd5d9b4dee57fe676be4b42cb63850ee03544c0289 not found: ID does not exist" containerID="ea532bf9034d0866789a27dd5d9b4dee57fe676be4b42cb63850ee03544c0289" Jul 19 15:28:50.957065 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:28:50.956987 2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ea532bf9034d0866789a27dd5d9b4dee57fe676be4b42cb63850ee03544c0289"} err="failed to get container status \"ea532bf9034d0866789a27dd5d9b4dee57fe676be4b42cb63850ee03544c0289\": rpc error: code = NotFound desc = could not find container \"ea532bf9034d0866789a27dd5d9b4dee57fe676be4b42cb63850ee03544c0289\": container with ID starting with ea532bf9034d0866789a27dd5d9b4dee57fe676be4b42cb63850ee03544c0289 not found: ID does not exist" Jul 19 15:28:50.960052 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:28:50.960017 2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-a4b8edd0-openshift-default-5db5d69955-5fll7"] Jul 19 15:28:50.963808 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:28:50.963788 2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-a4b8edd0-openshift-default-5db5d69955-5fll7"] Jul 19 15:28:51.564612 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:28:51.564577 2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1cbf9895-d90f-49bb-96c8-af951e03ed63" path="/var/lib/kubelet/pods/1cbf9895-d90f-49bb-96c8-af951e03ed63/volumes" Jul 19 15:28:53.095892 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:28:53.095859 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-d4lll"] Jul 19 15:29:00.989045 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:29:00.988990 2558 generic.go:356] "Generic (PLEG): container finished" podID="6276272e-75e7-4d75-9671-33902770dd53" containerID="9817f27b66c36940d3d23c8ffd46bc2dabc6ca83c0464494f2e2d8d1596bf087" exitCode=0 Jul 19 15:29:00.989411 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:29:00.989061 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-e7bec00b-openshift-default-777c96ff67-v84vb" event={"ID":"6276272e-75e7-4d75-9671-33902770dd53","Type":"ContainerDied","Data":"9817f27b66c36940d3d23c8ffd46bc2dabc6ca83c0464494f2e2d8d1596bf087"} Jul 19 15:29:00.989411 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:29:00.989401 2558 scope.go:117] "RemoveContainer" containerID="9817f27b66c36940d3d23c8ffd46bc2dabc6ca83c0464494f2e2d8d1596bf087" Jul 19 15:29:00.992836 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:29:00.992802 2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 19 15:29:00.992941 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:29:00.992897 2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 19 15:29:00.992941 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:29:00.992935 2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 19 15:29:02.001015 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:29:02.000980 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-e7bec00b-openshift-default-777c96ff67-v84vb" event={"ID":"6276272e-75e7-4d75-9671-33902770dd53","Type":"ContainerStarted","Data":"2e8a8a5c05488d6668267054155701bcff80cea115add94147f408685adb4bf9"} Jul 19 15:29:02.219007 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:29:02.218971 2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-e7bec00b-openshift-default-777c96ff67-v84vb" Jul 19 15:29:02.220281 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:29:02.220253 2558 patch_prober.go:28] interesting pod/e2e-derive-e7bec00b-openshift-default-777c96ff67-v84vb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.30:15021/healthz/ready\": dial tcp 10.133.0.30:15021: connect: connection refused" start-of-body= Jul 19 15:29:02.220413 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:29:02.220307 2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-e7bec00b-openshift-default-777c96ff67-v84vb" podUID="6276272e-75e7-4d75-9671-33902770dd53" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.30:15021/healthz/ready\": dial tcp 10.133.0.30:15021: connect: connection refused" Jul 19 15:29:02.927090 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:29:02.927047 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-e7bec00b-openshift-default-777c96ff67-v84vb" Jul 19 15:29:03.219175 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:29:03.219091 2558 patch_prober.go:28] interesting pod/e2e-derive-e7bec00b-openshift-default-777c96ff67-v84vb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.30:15021/healthz/ready\": dial tcp 10.133.0.30:15021: connect: connection refused" start-of-body= Jul 19 15:29:03.219175 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:29:03.219151 2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-e7bec00b-openshift-default-777c96ff67-v84vb" podUID="6276272e-75e7-4d75-9671-33902770dd53" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.30:15021/healthz/ready\": dial tcp 10.133.0.30:15021: connect: connection refused" Jul 19 15:29:04.219270 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:29:04.219229 2558 patch_prober.go:28] interesting pod/e2e-derive-e7bec00b-openshift-default-777c96ff67-v84vb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.30:15021/healthz/ready\": dial tcp 10.133.0.30:15021: connect: connection refused" start-of-body= Jul 19 15:29:04.219661 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:29:04.219298 2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-e7bec00b-openshift-default-777c96ff67-v84vb" podUID="6276272e-75e7-4d75-9671-33902770dd53" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.30:15021/healthz/ready\": dial tcp 10.133.0.30:15021: connect: connection refused" Jul 19 15:29:05.219837 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:29:05.219792 2558 patch_prober.go:28] interesting pod/e2e-derive-e7bec00b-openshift-default-777c96ff67-v84vb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.30:15021/healthz/ready\": dial tcp 10.133.0.30:15021: connect: connection refused" start-of-body= Jul 19 15:29:05.220264 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:29:05.219856 2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-e7bec00b-openshift-default-777c96ff67-v84vb" podUID="6276272e-75e7-4d75-9671-33902770dd53" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.30:15021/healthz/ready\": dial tcp 10.133.0.30:15021: connect: connection refused" Jul 19 15:29:06.223145 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:29:06.223109 2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-e7bec00b-openshift-default-777c96ff67-v84vb" Jul 19 15:29:06.223988 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:29:06.223971 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-e7bec00b-openshift-default-777c96ff67-v84vb" Jul 19 15:29:13.139513 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:29:13.139471 2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-e7bec00b-rl7k8"] Jul 19 15:29:13.139973 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:29:13.139886 2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1cbf9895-d90f-49bb-96c8-af951e03ed63" containerName="istio-proxy" Jul 19 15:29:13.139973 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:29:13.139906 2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="1cbf9895-d90f-49bb-96c8-af951e03ed63" containerName="istio-proxy" Jul 19 15:29:13.140062 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:29:13.140002 2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="1cbf9895-d90f-49bb-96c8-af951e03ed63" containerName="istio-proxy" Jul 19 15:29:13.143158 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:29:13.143139 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-e7bec00b-rl7k8" Jul 19 15:29:13.145584 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:29:13.145563 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 19 15:29:13.145711 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:29:13.145644 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qddbb\"" Jul 19 15:29:13.148724 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:29:13.148702 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-e7bec00b-rl7k8"] Jul 19 15:29:13.215282 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:29:13.215247 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1bbf2b8e-a4a7-412e-b027-4764ffc10712-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-e7bec00b-rl7k8\" (UID: \"1bbf2b8e-a4a7-412e-b027-4764ffc10712\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-e7bec00b-rl7k8" Jul 19 15:29:13.316425 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:29:13.316393 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1bbf2b8e-a4a7-412e-b027-4764ffc10712-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-e7bec00b-rl7k8\" (UID: \"1bbf2b8e-a4a7-412e-b027-4764ffc10712\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-e7bec00b-rl7k8" Jul 19 15:29:13.317012 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:29:13.316989 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1bbf2b8e-a4a7-412e-b027-4764ffc10712-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-e7bec00b-rl7k8\" (UID: \"1bbf2b8e-a4a7-412e-b027-4764ffc10712\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-e7bec00b-rl7k8" Jul 19 15:29:13.454981 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:29:13.454951 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-e7bec00b-rl7k8" Jul 19 15:29:13.592052 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:29:13.591978 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-e7bec00b-rl7k8"] Jul 19 15:29:13.594897 ip-10-0-138-220 kubenswrapper[2558]: W0719 15:29:13.594870 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1bbf2b8e_a4a7_412e_b027_4764ffc10712.slice/crio-dc122ecc034fe81e7fddad3029255cbb7583f38546bbfd36bb6a3a9cf2b55a23 WatchSource:0}: Error finding container dc122ecc034fe81e7fddad3029255cbb7583f38546bbfd36bb6a3a9cf2b55a23: Status 404 returned error can't find the container with id dc122ecc034fe81e7fddad3029255cbb7583f38546bbfd36bb6a3a9cf2b55a23 Jul 19 15:29:14.061059 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:29:14.060946 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-e7bec00b-rl7k8" event={"ID":"1bbf2b8e-a4a7-412e-b027-4764ffc10712","Type":"ContainerStarted","Data":"f0e462a2ee6bf3e00269022e661974728de4aa01565bb2bdd6529440bfb7d6d8"} Jul 19 15:29:14.061059 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:29:14.061052 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-e7bec00b-rl7k8" event={"ID":"1bbf2b8e-a4a7-412e-b027-4764ffc10712","Type":"ContainerStarted","Data":"dc122ecc034fe81e7fddad3029255cbb7583f38546bbfd36bb6a3a9cf2b55a23"} Jul 19 15:29:14.075492 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:29:14.075448 2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-e7bec00b-rl7k8" podStartSLOduration=1.075433487 podStartE2EDuration="1.075433487s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-19 15:29:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-19 15:29:14.074414289 +0000 UTC m=+1894.996117003" watchObservedRunningTime="2026-07-19 15:29:14.075433487 +0000 UTC m=+1894.997136210" Jul 19 15:29:15.066579 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:29:15.066545 2558 generic.go:356] "Generic (PLEG): container finished" podID="1bbf2b8e-a4a7-412e-b027-4764ffc10712" containerID="f0e462a2ee6bf3e00269022e661974728de4aa01565bb2bdd6529440bfb7d6d8" exitCode=0 Jul 19 15:29:15.066991 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:29:15.066644 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-e7bec00b-rl7k8" event={"ID":"1bbf2b8e-a4a7-412e-b027-4764ffc10712","Type":"ContainerDied","Data":"f0e462a2ee6bf3e00269022e661974728de4aa01565bb2bdd6529440bfb7d6d8"} Jul 19 15:29:16.195567 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:29:16.195541 2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-e7bec00b-rl7k8" Jul 19 15:29:16.339918 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:29:16.339812 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1bbf2b8e-a4a7-412e-b027-4764ffc10712-maas-api-service-ca\") pod \"1bbf2b8e-a4a7-412e-b027-4764ffc10712\" (UID: \"1bbf2b8e-a4a7-412e-b027-4764ffc10712\") " Jul 19 15:29:16.340199 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:29:16.340177 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1bbf2b8e-a4a7-412e-b027-4764ffc10712-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "1bbf2b8e-a4a7-412e-b027-4764ffc10712" (UID: "1bbf2b8e-a4a7-412e-b027-4764ffc10712"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 19 15:29:16.441288 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:29:16.441252 2558 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1bbf2b8e-a4a7-412e-b027-4764ffc10712-maas-api-service-ca\") on node \"ip-10-0-138-220.ec2.internal\" DevicePath \"\"" Jul 19 15:29:17.076567 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:29:17.076540 2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-e7bec00b-rl7k8" Jul 19 15:29:17.076706 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:29:17.076569 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-e7bec00b-rl7k8" event={"ID":"1bbf2b8e-a4a7-412e-b027-4764ffc10712","Type":"ContainerDied","Data":"dc122ecc034fe81e7fddad3029255cbb7583f38546bbfd36bb6a3a9cf2b55a23"} Jul 19 15:29:17.076706 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:29:17.076611 2558 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="dc122ecc034fe81e7fddad3029255cbb7583f38546bbfd36bb6a3a9cf2b55a23" Jul 19 15:29:26.388269 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:29:26.388229 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-d4lll"] Jul 19 15:29:28.208926 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:29:28.208890 2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-e7bec00b-rl7k8"] Jul 19 15:29:28.210926 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:29:28.210901 2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-e7bec00b-rl7k8"] Jul 19 15:29:28.347998 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:29:28.347963 2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-e7bec00b-openshift-default-777c96ff67-v84vb"] Jul 19 15:29:28.348254 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:29:28.348229 2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-derive-e7bec00b-openshift-default-777c96ff67-v84vb" podUID="6276272e-75e7-4d75-9671-33902770dd53" containerName="istio-proxy" containerID="cri-o://2e8a8a5c05488d6668267054155701bcff80cea115add94147f408685adb4bf9" gracePeriod=30 Jul 19 15:29:29.565392 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:29:29.565361 2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1bbf2b8e-a4a7-412e-b027-4764ffc10712" path="/var/lib/kubelet/pods/1bbf2b8e-a4a7-412e-b027-4764ffc10712/volumes" Jul 19 15:29:33.601114 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:29:33.601090 2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-e7bec00b-openshift-default-777c96ff67-v84vb" Jul 19 15:29:33.683336 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:29:33.683296 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6276272e-75e7-4d75-9671-33902770dd53-workload-socket\") pod \"6276272e-75e7-4d75-9671-33902770dd53\" (UID: \"6276272e-75e7-4d75-9671-33902770dd53\") " Jul 19 15:29:33.683336 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:29:33.683336 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6276272e-75e7-4d75-9671-33902770dd53-workload-certs\") pod \"6276272e-75e7-4d75-9671-33902770dd53\" (UID: \"6276272e-75e7-4d75-9671-33902770dd53\") " Jul 19 15:29:33.683598 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:29:33.683368 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6276272e-75e7-4d75-9671-33902770dd53-istio-podinfo\") pod \"6276272e-75e7-4d75-9671-33902770dd53\" (UID: \"6276272e-75e7-4d75-9671-33902770dd53\") " Jul 19 15:29:33.683598 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:29:33.683399 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mjd55\" (UniqueName: \"kubernetes.io/projected/6276272e-75e7-4d75-9671-33902770dd53-kube-api-access-mjd55\") pod \"6276272e-75e7-4d75-9671-33902770dd53\" (UID: \"6276272e-75e7-4d75-9671-33902770dd53\") " Jul 19 15:29:33.683598 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:29:33.683540 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6276272e-75e7-4d75-9671-33902770dd53-istio-envoy\") pod \"6276272e-75e7-4d75-9671-33902770dd53\" (UID: \"6276272e-75e7-4d75-9671-33902770dd53\") " Jul 19 15:29:33.683749 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:29:33.683607 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6276272e-75e7-4d75-9671-33902770dd53-istio-data\") pod \"6276272e-75e7-4d75-9671-33902770dd53\" (UID: \"6276272e-75e7-4d75-9671-33902770dd53\") " Jul 19 15:29:33.683749 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:29:33.683633 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6276272e-75e7-4d75-9671-33902770dd53-credential-socket\") pod \"6276272e-75e7-4d75-9671-33902770dd53\" (UID: \"6276272e-75e7-4d75-9671-33902770dd53\") " Jul 19 15:29:33.683749 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:29:33.683635 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6276272e-75e7-4d75-9671-33902770dd53-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "6276272e-75e7-4d75-9671-33902770dd53" (UID: "6276272e-75e7-4d75-9671-33902770dd53"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 19 15:29:33.683749 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:29:33.683662 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6276272e-75e7-4d75-9671-33902770dd53-istio-token\") pod \"6276272e-75e7-4d75-9671-33902770dd53\" (UID: \"6276272e-75e7-4d75-9671-33902770dd53\") " Jul 19 15:29:33.683749 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:29:33.683687 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6276272e-75e7-4d75-9671-33902770dd53-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "6276272e-75e7-4d75-9671-33902770dd53" (UID: "6276272e-75e7-4d75-9671-33902770dd53"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 19 15:29:33.683749 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:29:33.683718 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6276272e-75e7-4d75-9671-33902770dd53-istiod-ca-cert\") pod \"6276272e-75e7-4d75-9671-33902770dd53\" (UID: \"6276272e-75e7-4d75-9671-33902770dd53\") " Jul 19 15:29:33.684075 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:29:33.683877 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6276272e-75e7-4d75-9671-33902770dd53-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "6276272e-75e7-4d75-9671-33902770dd53" (UID: "6276272e-75e7-4d75-9671-33902770dd53"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 19 15:29:33.684075 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:29:33.683969 2558 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6276272e-75e7-4d75-9671-33902770dd53-credential-socket\") on node \"ip-10-0-138-220.ec2.internal\" DevicePath \"\"" Jul 19 15:29:33.684075 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:29:33.683987 2558 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6276272e-75e7-4d75-9671-33902770dd53-workload-socket\") on node \"ip-10-0-138-220.ec2.internal\" DevicePath \"\"" Jul 19 15:29:33.684075 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:29:33.684003 2558 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6276272e-75e7-4d75-9671-33902770dd53-workload-certs\") on node \"ip-10-0-138-220.ec2.internal\" DevicePath \"\"" Jul 19 15:29:33.684228 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:29:33.684112 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6276272e-75e7-4d75-9671-33902770dd53-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "6276272e-75e7-4d75-9671-33902770dd53" (UID: "6276272e-75e7-4d75-9671-33902770dd53"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 19 15:29:33.684748 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:29:33.684719 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6276272e-75e7-4d75-9671-33902770dd53-istio-data" (OuterVolumeSpecName: "istio-data") pod "6276272e-75e7-4d75-9671-33902770dd53" (UID: "6276272e-75e7-4d75-9671-33902770dd53"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 19 15:29:33.685846 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:29:33.685817 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6276272e-75e7-4d75-9671-33902770dd53-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "6276272e-75e7-4d75-9671-33902770dd53" (UID: "6276272e-75e7-4d75-9671-33902770dd53"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 19 15:29:33.685945 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:29:33.685913 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6276272e-75e7-4d75-9671-33902770dd53-kube-api-access-mjd55" (OuterVolumeSpecName: "kube-api-access-mjd55") pod "6276272e-75e7-4d75-9671-33902770dd53" (UID: "6276272e-75e7-4d75-9671-33902770dd53"). InnerVolumeSpecName "kube-api-access-mjd55". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 19 15:29:33.686246 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:29:33.686219 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6276272e-75e7-4d75-9671-33902770dd53-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "6276272e-75e7-4d75-9671-33902770dd53" (UID: "6276272e-75e7-4d75-9671-33902770dd53"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 19 15:29:33.686246 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:29:33.686231 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6276272e-75e7-4d75-9671-33902770dd53-istio-token" (OuterVolumeSpecName: "istio-token") pod "6276272e-75e7-4d75-9671-33902770dd53" (UID: "6276272e-75e7-4d75-9671-33902770dd53"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 19 15:29:33.785076 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:29:33.785016 2558 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6276272e-75e7-4d75-9671-33902770dd53-istio-token\") on node \"ip-10-0-138-220.ec2.internal\" DevicePath \"\"" Jul 19 15:29:33.785076 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:29:33.785070 2558 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6276272e-75e7-4d75-9671-33902770dd53-istiod-ca-cert\") on node \"ip-10-0-138-220.ec2.internal\" DevicePath \"\"" Jul 19 15:29:33.785076 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:29:33.785083 2558 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6276272e-75e7-4d75-9671-33902770dd53-istio-podinfo\") on node \"ip-10-0-138-220.ec2.internal\" DevicePath \"\"" Jul 19 15:29:33.785307 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:29:33.785097 2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mjd55\" (UniqueName: \"kubernetes.io/projected/6276272e-75e7-4d75-9671-33902770dd53-kube-api-access-mjd55\") on node \"ip-10-0-138-220.ec2.internal\" DevicePath \"\"" Jul 19 15:29:33.785307 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:29:33.785110 2558 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6276272e-75e7-4d75-9671-33902770dd53-istio-envoy\") on node \"ip-10-0-138-220.ec2.internal\" DevicePath \"\"" Jul 19 15:29:33.785307 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:29:33.785123 2558 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6276272e-75e7-4d75-9671-33902770dd53-istio-data\") on node \"ip-10-0-138-220.ec2.internal\" DevicePath \"\"" Jul 19 15:29:34.151676 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:29:34.151646 2558 generic.go:356] "Generic (PLEG): container finished" podID="6276272e-75e7-4d75-9671-33902770dd53" containerID="2e8a8a5c05488d6668267054155701bcff80cea115add94147f408685adb4bf9" exitCode=0 Jul 19 15:29:34.151833 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:29:34.151711 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-e7bec00b-openshift-default-777c96ff67-v84vb" event={"ID":"6276272e-75e7-4d75-9671-33902770dd53","Type":"ContainerDied","Data":"2e8a8a5c05488d6668267054155701bcff80cea115add94147f408685adb4bf9"} Jul 19 15:29:34.151833 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:29:34.151747 2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-e7bec00b-openshift-default-777c96ff67-v84vb" Jul 19 15:29:34.151833 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:29:34.151785 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-e7bec00b-openshift-default-777c96ff67-v84vb" event={"ID":"6276272e-75e7-4d75-9671-33902770dd53","Type":"ContainerDied","Data":"b9486247caf2c8338ac38c93fbd7a52db96a51cb60e4100520aaa6c81ac90def"} Jul 19 15:29:34.151833 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:29:34.151814 2558 scope.go:117] "RemoveContainer" containerID="2e8a8a5c05488d6668267054155701bcff80cea115add94147f408685adb4bf9" Jul 19 15:29:34.161414 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:29:34.161394 2558 scope.go:117] "RemoveContainer" containerID="9817f27b66c36940d3d23c8ffd46bc2dabc6ca83c0464494f2e2d8d1596bf087" Jul 19 15:29:34.169989 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:29:34.169965 2558 scope.go:117] "RemoveContainer" containerID="2e8a8a5c05488d6668267054155701bcff80cea115add94147f408685adb4bf9" Jul 19 15:29:34.170346 ip-10-0-138-220 kubenswrapper[2558]: E0719 15:29:34.170326 2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2e8a8a5c05488d6668267054155701bcff80cea115add94147f408685adb4bf9\": container with ID starting with 2e8a8a5c05488d6668267054155701bcff80cea115add94147f408685adb4bf9 not found: ID does not exist" containerID="2e8a8a5c05488d6668267054155701bcff80cea115add94147f408685adb4bf9" Jul 19 15:29:34.170429 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:29:34.170354 2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2e8a8a5c05488d6668267054155701bcff80cea115add94147f408685adb4bf9"} err="failed to get container status \"2e8a8a5c05488d6668267054155701bcff80cea115add94147f408685adb4bf9\": rpc error: code = NotFound desc = could not find container \"2e8a8a5c05488d6668267054155701bcff80cea115add94147f408685adb4bf9\": container with ID starting with 2e8a8a5c05488d6668267054155701bcff80cea115add94147f408685adb4bf9 not found: ID does not exist" Jul 19 15:29:34.170429 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:29:34.170372 2558 scope.go:117] "RemoveContainer" containerID="9817f27b66c36940d3d23c8ffd46bc2dabc6ca83c0464494f2e2d8d1596bf087" Jul 19 15:29:34.170642 ip-10-0-138-220 kubenswrapper[2558]: E0719 15:29:34.170613 2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9817f27b66c36940d3d23c8ffd46bc2dabc6ca83c0464494f2e2d8d1596bf087\": container with ID starting with 9817f27b66c36940d3d23c8ffd46bc2dabc6ca83c0464494f2e2d8d1596bf087 not found: ID does not exist" containerID="9817f27b66c36940d3d23c8ffd46bc2dabc6ca83c0464494f2e2d8d1596bf087" Jul 19 15:29:34.170683 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:29:34.170649 2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9817f27b66c36940d3d23c8ffd46bc2dabc6ca83c0464494f2e2d8d1596bf087"} err="failed to get container status \"9817f27b66c36940d3d23c8ffd46bc2dabc6ca83c0464494f2e2d8d1596bf087\": rpc error: code = NotFound desc = could not find container \"9817f27b66c36940d3d23c8ffd46bc2dabc6ca83c0464494f2e2d8d1596bf087\": container with ID starting with 9817f27b66c36940d3d23c8ffd46bc2dabc6ca83c0464494f2e2d8d1596bf087 not found: ID does not exist" Jul 19 15:29:34.174207 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:29:34.174171 2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-e7bec00b-openshift-default-777c96ff67-v84vb"] Jul 19 15:29:34.178973 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:29:34.178940 2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-derive-e7bec00b-openshift-default-777c96ff67-v84vb"] Jul 19 15:29:35.565084 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:29:35.565053 2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6276272e-75e7-4d75-9671-33902770dd53" path="/var/lib/kubelet/pods/6276272e-75e7-4d75-9671-33902770dd53/volumes" Jul 19 15:30:00.127981 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:30:00.127943 2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29741250-j875z"] Jul 19 15:30:00.128519 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:30:00.128232 2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1bbf2b8e-a4a7-412e-b027-4764ffc10712" containerName="revoke-keys" Jul 19 15:30:00.128519 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:30:00.128245 2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="1bbf2b8e-a4a7-412e-b027-4764ffc10712" containerName="revoke-keys" Jul 19 15:30:00.128519 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:30:00.128260 2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6276272e-75e7-4d75-9671-33902770dd53" containerName="istio-proxy" Jul 19 15:30:00.128519 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:30:00.128267 2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="6276272e-75e7-4d75-9671-33902770dd53" containerName="istio-proxy" Jul 19 15:30:00.128519 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:30:00.128280 2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1cbf9895-d90f-49bb-96c8-af951e03ed63" containerName="istio-proxy" Jul 19 15:30:00.128519 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:30:00.128286 2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="1cbf9895-d90f-49bb-96c8-af951e03ed63" containerName="istio-proxy" Jul 19 15:30:00.128519 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:30:00.128330 2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="1bbf2b8e-a4a7-412e-b027-4764ffc10712" containerName="revoke-keys" Jul 19 15:30:00.128519 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:30:00.128338 2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="6276272e-75e7-4d75-9671-33902770dd53" containerName="istio-proxy" Jul 19 15:30:00.128519 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:30:00.128345 2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="1cbf9895-d90f-49bb-96c8-af951e03ed63" containerName="istio-proxy" Jul 19 15:30:00.128519 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:30:00.128350 2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="6276272e-75e7-4d75-9671-33902770dd53" containerName="istio-proxy" Jul 19 15:30:00.131252 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:30:00.131236 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29741250-j875z" Jul 19 15:30:00.133372 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:30:00.133354 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qddbb\"" Jul 19 15:30:00.139324 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:30:00.139299 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29741250-j875z"] Jul 19 15:30:00.141378 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:30:00.141358 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29741250-j875z" Jul 19 15:30:00.273999 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:30:00.273974 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29741250-j875z"] Jul 19 15:30:00.276298 ip-10-0-138-220 kubenswrapper[2558]: W0719 15:30:00.276259 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbce4499d_3ed2_4f05_8a56_a28653e37a2a.slice/crio-c8b2ddc36792517e02b5b3cf071b0742db3d957626284a602990ed3c458feece WatchSource:0}: Error finding container c8b2ddc36792517e02b5b3cf071b0742db3d957626284a602990ed3c458feece: Status 404 returned error can't find the container with id c8b2ddc36792517e02b5b3cf071b0742db3d957626284a602990ed3c458feece Jul 19 15:30:01.267969 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:30:01.267931 2558 generic.go:356] "Generic (PLEG): container finished" podID="bce4499d-3ed2-4f05-8a56-a28653e37a2a" containerID="9b6c055d4e2cd5492400ab08a43af6e4d8bfc023647b902ca96ec2db5516d9d6" exitCode=0 Jul 19 15:30:01.268386 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:30:01.268002 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29741250-j875z" event={"ID":"bce4499d-3ed2-4f05-8a56-a28653e37a2a","Type":"ContainerDied","Data":"9b6c055d4e2cd5492400ab08a43af6e4d8bfc023647b902ca96ec2db5516d9d6"} Jul 19 15:30:01.268386 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:30:01.268072 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29741250-j875z" event={"ID":"bce4499d-3ed2-4f05-8a56-a28653e37a2a","Type":"ContainerStarted","Data":"c8b2ddc36792517e02b5b3cf071b0742db3d957626284a602990ed3c458feece"} Jul 19 15:30:02.401204 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:30:02.401173 2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29741250-j875z" Jul 19 15:30:03.278171 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:30:03.278146 2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29741250-j875z" Jul 19 15:30:03.278343 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:30:03.278191 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29741250-j875z" event={"ID":"bce4499d-3ed2-4f05-8a56-a28653e37a2a","Type":"ContainerDied","Data":"c8b2ddc36792517e02b5b3cf071b0742db3d957626284a602990ed3c458feece"} Jul 19 15:30:03.278343 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:30:03.278233 2558 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c8b2ddc36792517e02b5b3cf071b0742db3d957626284a602990ed3c458feece" Jul 19 15:30:53.091218 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:30:53.091183 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-d4lll"] Jul 19 15:31:04.698681 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:31:04.698635 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-d4lll"] Jul 19 15:31:05.178709 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:31:05.178673 2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-fe155acd-openshift-default-794f59b879-wbh6q"] Jul 19 15:31:05.178952 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:31:05.178939 2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6276272e-75e7-4d75-9671-33902770dd53" containerName="istio-proxy" Jul 19 15:31:05.178952 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:31:05.178952 2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="6276272e-75e7-4d75-9671-33902770dd53" containerName="istio-proxy" Jul 19 15:31:05.179056 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:31:05.178961 2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bce4499d-3ed2-4f05-8a56-a28653e37a2a" containerName="cleanup" Jul 19 15:31:05.179056 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:31:05.178967 2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="bce4499d-3ed2-4f05-8a56-a28653e37a2a" containerName="cleanup" Jul 19 15:31:05.179056 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:31:05.179037 2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="bce4499d-3ed2-4f05-8a56-a28653e37a2a" containerName="cleanup" Jul 19 15:31:05.182066 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:31:05.182042 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-fe155acd-openshift-default-794f59b879-wbh6q" Jul 19 15:31:05.184157 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:31:05.184136 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-fe155acd-openshift-default-dockercfg-n9zj7\"" Jul 19 15:31:05.194045 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:31:05.193999 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-fe155acd-openshift-default-794f59b879-wbh6q"] Jul 19 15:31:05.205469 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:31:05.205430 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/45da29d4-a468-43a1-9ca2-7be168e13b00-workload-certs\") pod \"e2e-mt-fe155acd-openshift-default-794f59b879-wbh6q\" (UID: \"45da29d4-a468-43a1-9ca2-7be168e13b00\") " pod="openshift-ingress/e2e-mt-fe155acd-openshift-default-794f59b879-wbh6q" Jul 19 15:31:05.205626 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:31:05.205501 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/45da29d4-a468-43a1-9ca2-7be168e13b00-credential-socket\") pod \"e2e-mt-fe155acd-openshift-default-794f59b879-wbh6q\" (UID: \"45da29d4-a468-43a1-9ca2-7be168e13b00\") " pod="openshift-ingress/e2e-mt-fe155acd-openshift-default-794f59b879-wbh6q" Jul 19 15:31:05.205626 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:31:05.205527 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/45da29d4-a468-43a1-9ca2-7be168e13b00-istiod-ca-cert\") pod \"e2e-mt-fe155acd-openshift-default-794f59b879-wbh6q\" (UID: \"45da29d4-a468-43a1-9ca2-7be168e13b00\") " pod="openshift-ingress/e2e-mt-fe155acd-openshift-default-794f59b879-wbh6q" Jul 19 15:31:05.205626 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:31:05.205560 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/45da29d4-a468-43a1-9ca2-7be168e13b00-workload-socket\") pod \"e2e-mt-fe155acd-openshift-default-794f59b879-wbh6q\" (UID: \"45da29d4-a468-43a1-9ca2-7be168e13b00\") " pod="openshift-ingress/e2e-mt-fe155acd-openshift-default-794f59b879-wbh6q" Jul 19 15:31:05.205626 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:31:05.205601 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/45da29d4-a468-43a1-9ca2-7be168e13b00-istio-envoy\") pod \"e2e-mt-fe155acd-openshift-default-794f59b879-wbh6q\" (UID: \"45da29d4-a468-43a1-9ca2-7be168e13b00\") " pod="openshift-ingress/e2e-mt-fe155acd-openshift-default-794f59b879-wbh6q" Jul 19 15:31:05.205786 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:31:05.205668 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/45da29d4-a468-43a1-9ca2-7be168e13b00-istio-token\") pod \"e2e-mt-fe155acd-openshift-default-794f59b879-wbh6q\" (UID: \"45da29d4-a468-43a1-9ca2-7be168e13b00\") " pod="openshift-ingress/e2e-mt-fe155acd-openshift-default-794f59b879-wbh6q" Jul 19 15:31:05.205786 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:31:05.205688 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/45da29d4-a468-43a1-9ca2-7be168e13b00-istio-podinfo\") pod \"e2e-mt-fe155acd-openshift-default-794f59b879-wbh6q\" (UID: \"45da29d4-a468-43a1-9ca2-7be168e13b00\") " pod="openshift-ingress/e2e-mt-fe155acd-openshift-default-794f59b879-wbh6q" Jul 19 15:31:05.205786 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:31:05.205734 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bbjfj\" (UniqueName: \"kubernetes.io/projected/45da29d4-a468-43a1-9ca2-7be168e13b00-kube-api-access-bbjfj\") pod \"e2e-mt-fe155acd-openshift-default-794f59b879-wbh6q\" (UID: \"45da29d4-a468-43a1-9ca2-7be168e13b00\") " pod="openshift-ingress/e2e-mt-fe155acd-openshift-default-794f59b879-wbh6q" Jul 19 15:31:05.205892 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:31:05.205843 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/45da29d4-a468-43a1-9ca2-7be168e13b00-istio-data\") pod \"e2e-mt-fe155acd-openshift-default-794f59b879-wbh6q\" (UID: \"45da29d4-a468-43a1-9ca2-7be168e13b00\") " pod="openshift-ingress/e2e-mt-fe155acd-openshift-default-794f59b879-wbh6q" Jul 19 15:31:05.307173 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:31:05.307136 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bbjfj\" (UniqueName: \"kubernetes.io/projected/45da29d4-a468-43a1-9ca2-7be168e13b00-kube-api-access-bbjfj\") pod \"e2e-mt-fe155acd-openshift-default-794f59b879-wbh6q\" (UID: \"45da29d4-a468-43a1-9ca2-7be168e13b00\") " pod="openshift-ingress/e2e-mt-fe155acd-openshift-default-794f59b879-wbh6q" Jul 19 15:31:05.307173 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:31:05.307176 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/45da29d4-a468-43a1-9ca2-7be168e13b00-istio-data\") pod \"e2e-mt-fe155acd-openshift-default-794f59b879-wbh6q\" (UID: \"45da29d4-a468-43a1-9ca2-7be168e13b00\") " pod="openshift-ingress/e2e-mt-fe155acd-openshift-default-794f59b879-wbh6q" Jul 19 15:31:05.307481 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:31:05.307212 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/45da29d4-a468-43a1-9ca2-7be168e13b00-workload-certs\") pod \"e2e-mt-fe155acd-openshift-default-794f59b879-wbh6q\" (UID: \"45da29d4-a468-43a1-9ca2-7be168e13b00\") " pod="openshift-ingress/e2e-mt-fe155acd-openshift-default-794f59b879-wbh6q" Jul 19 15:31:05.307481 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:31:05.307252 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/45da29d4-a468-43a1-9ca2-7be168e13b00-credential-socket\") pod \"e2e-mt-fe155acd-openshift-default-794f59b879-wbh6q\" (UID: \"45da29d4-a468-43a1-9ca2-7be168e13b00\") " pod="openshift-ingress/e2e-mt-fe155acd-openshift-default-794f59b879-wbh6q" Jul 19 15:31:05.307481 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:31:05.307276 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/45da29d4-a468-43a1-9ca2-7be168e13b00-istiod-ca-cert\") pod \"e2e-mt-fe155acd-openshift-default-794f59b879-wbh6q\" (UID: \"45da29d4-a468-43a1-9ca2-7be168e13b00\") " pod="openshift-ingress/e2e-mt-fe155acd-openshift-default-794f59b879-wbh6q" Jul 19 15:31:05.307481 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:31:05.307306 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/45da29d4-a468-43a1-9ca2-7be168e13b00-workload-socket\") pod \"e2e-mt-fe155acd-openshift-default-794f59b879-wbh6q\" (UID: \"45da29d4-a468-43a1-9ca2-7be168e13b00\") " pod="openshift-ingress/e2e-mt-fe155acd-openshift-default-794f59b879-wbh6q" Jul 19 15:31:05.307481 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:31:05.307328 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/45da29d4-a468-43a1-9ca2-7be168e13b00-istio-envoy\") pod \"e2e-mt-fe155acd-openshift-default-794f59b879-wbh6q\" (UID: \"45da29d4-a468-43a1-9ca2-7be168e13b00\") " pod="openshift-ingress/e2e-mt-fe155acd-openshift-default-794f59b879-wbh6q" Jul 19 15:31:05.307481 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:31:05.307428 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/45da29d4-a468-43a1-9ca2-7be168e13b00-istio-token\") pod \"e2e-mt-fe155acd-openshift-default-794f59b879-wbh6q\" (UID: \"45da29d4-a468-43a1-9ca2-7be168e13b00\") " pod="openshift-ingress/e2e-mt-fe155acd-openshift-default-794f59b879-wbh6q" Jul 19 15:31:05.307481 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:31:05.307459 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/45da29d4-a468-43a1-9ca2-7be168e13b00-istio-podinfo\") pod \"e2e-mt-fe155acd-openshift-default-794f59b879-wbh6q\" (UID: \"45da29d4-a468-43a1-9ca2-7be168e13b00\") " pod="openshift-ingress/e2e-mt-fe155acd-openshift-default-794f59b879-wbh6q" Jul 19 15:31:05.307776 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:31:05.307626 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/45da29d4-a468-43a1-9ca2-7be168e13b00-istio-data\") pod \"e2e-mt-fe155acd-openshift-default-794f59b879-wbh6q\" (UID: \"45da29d4-a468-43a1-9ca2-7be168e13b00\") " pod="openshift-ingress/e2e-mt-fe155acd-openshift-default-794f59b879-wbh6q" Jul 19 15:31:05.307776 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:31:05.307671 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/45da29d4-a468-43a1-9ca2-7be168e13b00-credential-socket\") pod \"e2e-mt-fe155acd-openshift-default-794f59b879-wbh6q\" (UID: \"45da29d4-a468-43a1-9ca2-7be168e13b00\") " pod="openshift-ingress/e2e-mt-fe155acd-openshift-default-794f59b879-wbh6q" Jul 19 15:31:05.307884 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:31:05.307862 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/45da29d4-a468-43a1-9ca2-7be168e13b00-workload-socket\") pod \"e2e-mt-fe155acd-openshift-default-794f59b879-wbh6q\" (UID: \"45da29d4-a468-43a1-9ca2-7be168e13b00\") " pod="openshift-ingress/e2e-mt-fe155acd-openshift-default-794f59b879-wbh6q" Jul 19 15:31:05.307949 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:31:05.307897 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/45da29d4-a468-43a1-9ca2-7be168e13b00-workload-certs\") pod \"e2e-mt-fe155acd-openshift-default-794f59b879-wbh6q\" (UID: \"45da29d4-a468-43a1-9ca2-7be168e13b00\") " pod="openshift-ingress/e2e-mt-fe155acd-openshift-default-794f59b879-wbh6q" Jul 19 15:31:05.308097 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:31:05.308077 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/45da29d4-a468-43a1-9ca2-7be168e13b00-istiod-ca-cert\") pod \"e2e-mt-fe155acd-openshift-default-794f59b879-wbh6q\" (UID: \"45da29d4-a468-43a1-9ca2-7be168e13b00\") " pod="openshift-ingress/e2e-mt-fe155acd-openshift-default-794f59b879-wbh6q" Jul 19 15:31:05.309633 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:31:05.309615 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/45da29d4-a468-43a1-9ca2-7be168e13b00-istio-envoy\") pod \"e2e-mt-fe155acd-openshift-default-794f59b879-wbh6q\" (UID: \"45da29d4-a468-43a1-9ca2-7be168e13b00\") " pod="openshift-ingress/e2e-mt-fe155acd-openshift-default-794f59b879-wbh6q" Jul 19 15:31:05.309959 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:31:05.309941 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/45da29d4-a468-43a1-9ca2-7be168e13b00-istio-podinfo\") pod \"e2e-mt-fe155acd-openshift-default-794f59b879-wbh6q\" (UID: \"45da29d4-a468-43a1-9ca2-7be168e13b00\") " pod="openshift-ingress/e2e-mt-fe155acd-openshift-default-794f59b879-wbh6q" Jul 19 15:31:05.314864 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:31:05.314832 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/45da29d4-a468-43a1-9ca2-7be168e13b00-istio-token\") pod \"e2e-mt-fe155acd-openshift-default-794f59b879-wbh6q\" (UID: \"45da29d4-a468-43a1-9ca2-7be168e13b00\") " pod="openshift-ingress/e2e-mt-fe155acd-openshift-default-794f59b879-wbh6q" Jul 19 15:31:05.314968 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:31:05.314896 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bbjfj\" (UniqueName: \"kubernetes.io/projected/45da29d4-a468-43a1-9ca2-7be168e13b00-kube-api-access-bbjfj\") pod \"e2e-mt-fe155acd-openshift-default-794f59b879-wbh6q\" (UID: \"45da29d4-a468-43a1-9ca2-7be168e13b00\") " pod="openshift-ingress/e2e-mt-fe155acd-openshift-default-794f59b879-wbh6q" Jul 19 15:31:05.493709 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:31:05.493614 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-fe155acd-openshift-default-794f59b879-wbh6q" Jul 19 15:31:05.638891 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:31:05.638854 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-fe155acd-openshift-default-794f59b879-wbh6q"] Jul 19 15:31:05.643580 ip-10-0-138-220 kubenswrapper[2558]: W0719 15:31:05.643550 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod45da29d4_a468_43a1_9ca2_7be168e13b00.slice/crio-a5ab28659cf27d0505924473d48ee99f5497a19fca9a57d1a0320bee145bc566 WatchSource:0}: Error finding container a5ab28659cf27d0505924473d48ee99f5497a19fca9a57d1a0320bee145bc566: Status 404 returned error can't find the container with id a5ab28659cf27d0505924473d48ee99f5497a19fca9a57d1a0320bee145bc566 Jul 19 15:31:05.645807 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:31:05.645766 2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 19 15:31:05.645907 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:31:05.645857 2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 19 15:31:05.645907 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:31:05.645897 2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 19 15:31:06.544836 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:31:06.544798 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-fe155acd-openshift-default-794f59b879-wbh6q" event={"ID":"45da29d4-a468-43a1-9ca2-7be168e13b00","Type":"ContainerStarted","Data":"d5edd49cb8c7e2a8d7ab88507211d208bcb6270536b34e919b4a3f69452a6108"} Jul 19 15:31:06.545301 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:31:06.544855 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-fe155acd-openshift-default-794f59b879-wbh6q" event={"ID":"45da29d4-a468-43a1-9ca2-7be168e13b00","Type":"ContainerStarted","Data":"a5ab28659cf27d0505924473d48ee99f5497a19fca9a57d1a0320bee145bc566"} Jul 19 15:31:06.566139 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:31:06.566073 2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-fe155acd-openshift-default-794f59b879-wbh6q" podStartSLOduration=1.566056508 podStartE2EDuration="1.566056508s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-19 15:31:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-19 15:31:06.565508975 +0000 UTC m=+2007.487211715" watchObservedRunningTime="2026-07-19 15:31:06.566056508 +0000 UTC m=+2007.487759221" Jul 19 15:31:07.494113 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:31:07.494074 2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-fe155acd-openshift-default-794f59b879-wbh6q" Jul 19 15:31:07.499170 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:31:07.499147 2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-fe155acd-openshift-default-794f59b879-wbh6q" Jul 19 15:31:07.549951 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:31:07.549916 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-fe155acd-openshift-default-794f59b879-wbh6q" Jul 19 15:31:07.550889 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:31:07.550867 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-fe155acd-openshift-default-794f59b879-wbh6q" Jul 19 15:31:08.707847 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:31:08.707813 2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-fe155acd-786f7ccf45-d4hqq"] Jul 19 15:31:08.711065 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:31:08.711048 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-fe155acd-786f7ccf45-d4hqq" Jul 19 15:31:08.713281 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:31:08.713258 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Jul 19 15:31:08.713407 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:31:08.713309 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-ljt6w\"" Jul 19 15:31:08.713971 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:31:08.713949 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-fe155acd-serving-cert\"" Jul 19 15:31:08.714114 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:31:08.713970 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 19 15:31:08.720180 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:31:08.719674 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-fe155acd-786f7ccf45-d4hqq"] Jul 19 15:31:08.735791 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:31:08.735759 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5b12f068-243d-4825-b621-f1a9bb46d55f-openshift-service-ca\") pod \"maas-api-e2e-mt-fe155acd-786f7ccf45-d4hqq\" (UID: \"5b12f068-243d-4825-b621-f1a9bb46d55f\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-fe155acd-786f7ccf45-d4hqq" Jul 19 15:31:08.735951 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:31:08.735825 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5b12f068-243d-4825-b621-f1a9bb46d55f-maas-api-tls\") pod \"maas-api-e2e-mt-fe155acd-786f7ccf45-d4hqq\" (UID: \"5b12f068-243d-4825-b621-f1a9bb46d55f\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-fe155acd-786f7ccf45-d4hqq" Jul 19 15:31:08.735951 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:31:08.735856 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6x9kw\" (UniqueName: \"kubernetes.io/projected/5b12f068-243d-4825-b621-f1a9bb46d55f-kube-api-access-6x9kw\") pod \"maas-api-e2e-mt-fe155acd-786f7ccf45-d4hqq\" (UID: \"5b12f068-243d-4825-b621-f1a9bb46d55f\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-fe155acd-786f7ccf45-d4hqq" Jul 19 15:31:08.836980 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:31:08.836944 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5b12f068-243d-4825-b621-f1a9bb46d55f-openshift-service-ca\") pod \"maas-api-e2e-mt-fe155acd-786f7ccf45-d4hqq\" (UID: \"5b12f068-243d-4825-b621-f1a9bb46d55f\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-fe155acd-786f7ccf45-d4hqq" Jul 19 15:31:08.837188 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:31:08.836992 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5b12f068-243d-4825-b621-f1a9bb46d55f-maas-api-tls\") pod \"maas-api-e2e-mt-fe155acd-786f7ccf45-d4hqq\" (UID: \"5b12f068-243d-4825-b621-f1a9bb46d55f\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-fe155acd-786f7ccf45-d4hqq" Jul 19 15:31:08.837188 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:31:08.837010 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6x9kw\" (UniqueName: \"kubernetes.io/projected/5b12f068-243d-4825-b621-f1a9bb46d55f-kube-api-access-6x9kw\") pod \"maas-api-e2e-mt-fe155acd-786f7ccf45-d4hqq\" (UID: \"5b12f068-243d-4825-b621-f1a9bb46d55f\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-fe155acd-786f7ccf45-d4hqq" Jul 19 15:31:08.837620 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:31:08.837599 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5b12f068-243d-4825-b621-f1a9bb46d55f-openshift-service-ca\") pod \"maas-api-e2e-mt-fe155acd-786f7ccf45-d4hqq\" (UID: \"5b12f068-243d-4825-b621-f1a9bb46d55f\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-fe155acd-786f7ccf45-d4hqq" Jul 19 15:31:08.839617 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:31:08.839599 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5b12f068-243d-4825-b621-f1a9bb46d55f-maas-api-tls\") pod \"maas-api-e2e-mt-fe155acd-786f7ccf45-d4hqq\" (UID: \"5b12f068-243d-4825-b621-f1a9bb46d55f\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-fe155acd-786f7ccf45-d4hqq" Jul 19 15:31:08.844877 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:31:08.844851 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6x9kw\" (UniqueName: \"kubernetes.io/projected/5b12f068-243d-4825-b621-f1a9bb46d55f-kube-api-access-6x9kw\") pod \"maas-api-e2e-mt-fe155acd-786f7ccf45-d4hqq\" (UID: \"5b12f068-243d-4825-b621-f1a9bb46d55f\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-fe155acd-786f7ccf45-d4hqq" Jul 19 15:31:09.024898 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:31:09.024800 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-fe155acd-786f7ccf45-d4hqq" Jul 19 15:31:09.161099 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:31:09.161066 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-fe155acd-786f7ccf45-d4hqq"] Jul 19 15:31:09.164866 ip-10-0-138-220 kubenswrapper[2558]: W0719 15:31:09.164830 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5b12f068_243d_4825_b621_f1a9bb46d55f.slice/crio-a8bbf8ba196c544bd176d91f5d2dbe6a23a2711ca8a0d778612a4a5355142641 WatchSource:0}: Error finding container a8bbf8ba196c544bd176d91f5d2dbe6a23a2711ca8a0d778612a4a5355142641: Status 404 returned error can't find the container with id a8bbf8ba196c544bd176d91f5d2dbe6a23a2711ca8a0d778612a4a5355142641 Jul 19 15:31:09.166418 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:31:09.166399 2558 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 19 15:31:09.570728 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:31:09.570686 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-fe155acd-786f7ccf45-d4hqq" event={"ID":"5b12f068-243d-4825-b621-f1a9bb46d55f","Type":"ContainerStarted","Data":"a8bbf8ba196c544bd176d91f5d2dbe6a23a2711ca8a0d778612a4a5355142641"} Jul 19 15:31:11.569196 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:31:11.569154 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-fe155acd-786f7ccf45-d4hqq" event={"ID":"5b12f068-243d-4825-b621-f1a9bb46d55f","Type":"ContainerStarted","Data":"e5636642d3dca32295a66ab26cd5ca7ef13308f26d8eee8a5cf9293c6821423b"} Jul 19 15:31:11.569760 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:31:11.569277 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-fe155acd-786f7ccf45-d4hqq" Jul 19 15:31:11.584948 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:31:11.584899 2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-fe155acd-786f7ccf45-d4hqq" podStartSLOduration=2.202897204 podStartE2EDuration="3.584882775s" totalImagesPullingTime="1.381985571s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-19 15:31:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-19 15:31:11.582875161 +0000 UTC m=+2012.504577876" watchObservedRunningTime="2026-07-19 15:31:11.584882775 +0000 UTC m=+2012.506585488" Jul 19 15:31:17.581378 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:31:17.581348 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-fe155acd-786f7ccf45-d4hqq" Jul 19 15:31:17.697347 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:31:17.697314 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-d4lll"] Jul 19 15:31:19.604789 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:31:19.604755 2558 generic.go:356] "Generic (PLEG): container finished" podID="45da29d4-a468-43a1-9ca2-7be168e13b00" containerID="d5edd49cb8c7e2a8d7ab88507211d208bcb6270536b34e919b4a3f69452a6108" exitCode=0 Jul 19 15:31:19.605253 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:31:19.604830 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-fe155acd-openshift-default-794f59b879-wbh6q" event={"ID":"45da29d4-a468-43a1-9ca2-7be168e13b00","Type":"ContainerDied","Data":"d5edd49cb8c7e2a8d7ab88507211d208bcb6270536b34e919b4a3f69452a6108"} Jul 19 15:31:19.605253 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:31:19.605200 2558 scope.go:117] "RemoveContainer" containerID="d5edd49cb8c7e2a8d7ab88507211d208bcb6270536b34e919b4a3f69452a6108" Jul 19 15:31:19.606163 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:31:19.606140 2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 19 15:31:19.606219 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:31:19.606200 2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 19 15:31:19.606263 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:31:19.606230 2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 19 15:31:20.305807 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:31:20.305760 2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-fe155acd-dtl7b"] Jul 19 15:31:20.308949 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:31:20.308923 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-fe155acd-dtl7b" Jul 19 15:31:20.310981 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:31:20.310955 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qddbb\"" Jul 19 15:31:20.315847 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:31:20.315820 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-fe155acd-dtl7b"] Jul 19 15:31:20.437859 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:31:20.437818 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5eda7280-3e1c-44c4-947f-04b7370f4052-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-fe155acd-dtl7b\" (UID: \"5eda7280-3e1c-44c4-947f-04b7370f4052\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-fe155acd-dtl7b" Jul 19 15:31:20.538384 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:31:20.538350 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5eda7280-3e1c-44c4-947f-04b7370f4052-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-fe155acd-dtl7b\" (UID: \"5eda7280-3e1c-44c4-947f-04b7370f4052\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-fe155acd-dtl7b" Jul 19 15:31:20.538962 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:31:20.538942 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5eda7280-3e1c-44c4-947f-04b7370f4052-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-fe155acd-dtl7b\" (UID: \"5eda7280-3e1c-44c4-947f-04b7370f4052\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-fe155acd-dtl7b" Jul 19 15:31:20.611397 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:31:20.611310 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-fe155acd-openshift-default-794f59b879-wbh6q" event={"ID":"45da29d4-a468-43a1-9ca2-7be168e13b00","Type":"ContainerStarted","Data":"f69a37e362c2283e40e4db7165deb2779d430744b749f65e7656e559480f1f95"} Jul 19 15:31:20.621006 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:31:20.620977 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-fe155acd-dtl7b" Jul 19 15:31:20.803508 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:31:20.803477 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-fe155acd-dtl7b"] Jul 19 15:31:20.806625 ip-10-0-138-220 kubenswrapper[2558]: W0719 15:31:20.806598 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5eda7280_3e1c_44c4_947f_04b7370f4052.slice/crio-3f6fabd464f3da5b488ef81ccffa023633b893e08830af5cc89e56cd9f8d329e WatchSource:0}: Error finding container 3f6fabd464f3da5b488ef81ccffa023633b893e08830af5cc89e56cd9f8d329e: Status 404 returned error can't find the container with id 3f6fabd464f3da5b488ef81ccffa023633b893e08830af5cc89e56cd9f8d329e Jul 19 15:31:21.494044 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:31:21.493995 2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-fe155acd-openshift-default-794f59b879-wbh6q" Jul 19 15:31:21.495156 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:31:21.495126 2558 patch_prober.go:28] interesting pod/e2e-mt-fe155acd-openshift-default-794f59b879-wbh6q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused" start-of-body= Jul 19 15:31:21.495272 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:31:21.495201 2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-fe155acd-openshift-default-794f59b879-wbh6q" podUID="45da29d4-a468-43a1-9ca2-7be168e13b00" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused" Jul 19 15:31:21.616892 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:31:21.616853 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-fe155acd-dtl7b" event={"ID":"5eda7280-3e1c-44c4-947f-04b7370f4052","Type":"ContainerStarted","Data":"f96c3b20c174d13e6dddb72deb2474e73da78dd55e84f45880e57e4e1bb7ce33"} Jul 19 15:31:21.617338 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:31:21.616905 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-fe155acd-dtl7b" event={"ID":"5eda7280-3e1c-44c4-947f-04b7370f4052","Type":"ContainerStarted","Data":"3f6fabd464f3da5b488ef81ccffa023633b893e08830af5cc89e56cd9f8d329e"} Jul 19 15:31:21.631535 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:31:21.631484 2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-fe155acd-dtl7b" podStartSLOduration=1.63146785 podStartE2EDuration="1.63146785s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-19 15:31:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-19 15:31:21.629846903 +0000 UTC m=+2022.551549616" watchObservedRunningTime="2026-07-19 15:31:21.63146785 +0000 UTC m=+2022.553170563" Jul 19 15:31:22.494226 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:31:22.494193 2558 patch_prober.go:28] interesting pod/e2e-mt-fe155acd-openshift-default-794f59b879-wbh6q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused" start-of-body= Jul 19 15:31:22.494386 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:31:22.494249 2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-fe155acd-openshift-default-794f59b879-wbh6q" podUID="45da29d4-a468-43a1-9ca2-7be168e13b00" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused" Jul 19 15:31:22.550839 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:31:22.550803 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-fe155acd-openshift-default-794f59b879-wbh6q" Jul 19 15:31:22.623814 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:31:22.623783 2558 generic.go:356] "Generic (PLEG): container finished" podID="5eda7280-3e1c-44c4-947f-04b7370f4052" containerID="f96c3b20c174d13e6dddb72deb2474e73da78dd55e84f45880e57e4e1bb7ce33" exitCode=0 Jul 19 15:31:22.624228 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:31:22.623858 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-fe155acd-dtl7b" event={"ID":"5eda7280-3e1c-44c4-947f-04b7370f4052","Type":"ContainerDied","Data":"f96c3b20c174d13e6dddb72deb2474e73da78dd55e84f45880e57e4e1bb7ce33"} Jul 19 15:31:23.495074 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:31:23.495008 2558 patch_prober.go:28] interesting pod/e2e-mt-fe155acd-openshift-default-794f59b879-wbh6q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused" start-of-body= Jul 19 15:31:23.495256 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:31:23.495096 2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-fe155acd-openshift-default-794f59b879-wbh6q" podUID="45da29d4-a468-43a1-9ca2-7be168e13b00" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused" Jul 19 15:31:23.800660 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:31:23.800632 2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-fe155acd-dtl7b" Jul 19 15:31:23.873831 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:31:23.873796 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5eda7280-3e1c-44c4-947f-04b7370f4052-maas-api-service-ca\") pod \"5eda7280-3e1c-44c4-947f-04b7370f4052\" (UID: \"5eda7280-3e1c-44c4-947f-04b7370f4052\") " Jul 19 15:31:23.874181 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:31:23.874154 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5eda7280-3e1c-44c4-947f-04b7370f4052-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "5eda7280-3e1c-44c4-947f-04b7370f4052" (UID: "5eda7280-3e1c-44c4-947f-04b7370f4052"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 19 15:31:23.974876 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:31:23.974836 2558 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5eda7280-3e1c-44c4-947f-04b7370f4052-maas-api-service-ca\") on node \"ip-10-0-138-220.ec2.internal\" DevicePath \"\"" Jul 19 15:31:24.495049 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:31:24.494986 2558 patch_prober.go:28] interesting pod/e2e-mt-fe155acd-openshift-default-794f59b879-wbh6q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused" start-of-body= Jul 19 15:31:24.495240 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:31:24.495068 2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-fe155acd-openshift-default-794f59b879-wbh6q" podUID="45da29d4-a468-43a1-9ca2-7be168e13b00" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused" Jul 19 15:31:24.634149 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:31:24.634113 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-fe155acd-dtl7b" event={"ID":"5eda7280-3e1c-44c4-947f-04b7370f4052","Type":"ContainerDied","Data":"3f6fabd464f3da5b488ef81ccffa023633b893e08830af5cc89e56cd9f8d329e"} Jul 19 15:31:24.634149 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:31:24.634138 2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-fe155acd-dtl7b" Jul 19 15:31:24.634348 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:31:24.634166 2558 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3f6fabd464f3da5b488ef81ccffa023633b893e08830af5cc89e56cd9f8d329e" Jul 19 15:31:25.498901 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:31:25.498872 2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-fe155acd-openshift-default-794f59b879-wbh6q" Jul 19 15:31:25.499906 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:31:25.499888 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-fe155acd-openshift-default-794f59b879-wbh6q" Jul 19 15:31:30.676526 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:31:30.676490 2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-fe155acd-786f7ccf45-d4hqq"] Jul 19 15:31:30.676950 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:31:30.676675 2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-fe155acd-786f7ccf45-d4hqq" podUID="5b12f068-243d-4825-b621-f1a9bb46d55f" containerName="maas-api" containerID="cri-o://e5636642d3dca32295a66ab26cd5ca7ef13308f26d8eee8a5cf9293c6821423b" gracePeriod=30 Jul 19 15:31:30.919176 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:31:30.919152 2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-fe155acd-786f7ccf45-d4hqq" Jul 19 15:31:31.032065 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:31:31.031950 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5b12f068-243d-4825-b621-f1a9bb46d55f-maas-api-tls\") pod \"5b12f068-243d-4825-b621-f1a9bb46d55f\" (UID: \"5b12f068-243d-4825-b621-f1a9bb46d55f\") " Jul 19 15:31:31.032065 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:31:31.031994 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6x9kw\" (UniqueName: \"kubernetes.io/projected/5b12f068-243d-4825-b621-f1a9bb46d55f-kube-api-access-6x9kw\") pod \"5b12f068-243d-4825-b621-f1a9bb46d55f\" (UID: \"5b12f068-243d-4825-b621-f1a9bb46d55f\") " Jul 19 15:31:31.032275 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:31:31.032074 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5b12f068-243d-4825-b621-f1a9bb46d55f-openshift-service-ca\") pod \"5b12f068-243d-4825-b621-f1a9bb46d55f\" (UID: \"5b12f068-243d-4825-b621-f1a9bb46d55f\") " Jul 19 15:31:31.032471 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:31:31.032442 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5b12f068-243d-4825-b621-f1a9bb46d55f-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "5b12f068-243d-4825-b621-f1a9bb46d55f" (UID: "5b12f068-243d-4825-b621-f1a9bb46d55f"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 19 15:31:31.034316 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:31:31.034297 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5b12f068-243d-4825-b621-f1a9bb46d55f-kube-api-access-6x9kw" (OuterVolumeSpecName: "kube-api-access-6x9kw") pod "5b12f068-243d-4825-b621-f1a9bb46d55f" (UID: "5b12f068-243d-4825-b621-f1a9bb46d55f"). InnerVolumeSpecName "kube-api-access-6x9kw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 19 15:31:31.034413 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:31:31.034378 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5b12f068-243d-4825-b621-f1a9bb46d55f-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "5b12f068-243d-4825-b621-f1a9bb46d55f" (UID: "5b12f068-243d-4825-b621-f1a9bb46d55f"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 19 15:31:31.132943 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:31:31.132900 2558 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5b12f068-243d-4825-b621-f1a9bb46d55f-maas-api-tls\") on node \"ip-10-0-138-220.ec2.internal\" DevicePath \"\"" Jul 19 15:31:31.132943 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:31:31.132940 2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6x9kw\" (UniqueName: \"kubernetes.io/projected/5b12f068-243d-4825-b621-f1a9bb46d55f-kube-api-access-6x9kw\") on node \"ip-10-0-138-220.ec2.internal\" DevicePath \"\"" Jul 19 15:31:31.132943 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:31:31.132951 2558 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5b12f068-243d-4825-b621-f1a9bb46d55f-openshift-service-ca\") on node \"ip-10-0-138-220.ec2.internal\" DevicePath \"\"" Jul 19 15:31:31.666191 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:31:31.666156 2558 generic.go:356] "Generic (PLEG): container finished" podID="5b12f068-243d-4825-b621-f1a9bb46d55f" containerID="e5636642d3dca32295a66ab26cd5ca7ef13308f26d8eee8a5cf9293c6821423b" exitCode=0 Jul 19 15:31:31.666350 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:31:31.666205 2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-fe155acd-786f7ccf45-d4hqq" Jul 19 15:31:31.666350 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:31:31.666251 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-fe155acd-786f7ccf45-d4hqq" event={"ID":"5b12f068-243d-4825-b621-f1a9bb46d55f","Type":"ContainerDied","Data":"e5636642d3dca32295a66ab26cd5ca7ef13308f26d8eee8a5cf9293c6821423b"} Jul 19 15:31:31.666350 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:31:31.666311 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-fe155acd-786f7ccf45-d4hqq" event={"ID":"5b12f068-243d-4825-b621-f1a9bb46d55f","Type":"ContainerDied","Data":"a8bbf8ba196c544bd176d91f5d2dbe6a23a2711ca8a0d778612a4a5355142641"} Jul 19 15:31:31.666350 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:31:31.666345 2558 scope.go:117] "RemoveContainer" containerID="e5636642d3dca32295a66ab26cd5ca7ef13308f26d8eee8a5cf9293c6821423b" Jul 19 15:31:31.675202 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:31:31.675186 2558 scope.go:117] "RemoveContainer" containerID="e5636642d3dca32295a66ab26cd5ca7ef13308f26d8eee8a5cf9293c6821423b" Jul 19 15:31:31.675472 ip-10-0-138-220 kubenswrapper[2558]: E0719 15:31:31.675453 2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e5636642d3dca32295a66ab26cd5ca7ef13308f26d8eee8a5cf9293c6821423b\": container with ID starting with e5636642d3dca32295a66ab26cd5ca7ef13308f26d8eee8a5cf9293c6821423b not found: ID does not exist" containerID="e5636642d3dca32295a66ab26cd5ca7ef13308f26d8eee8a5cf9293c6821423b" Jul 19 15:31:31.675548 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:31:31.675478 2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e5636642d3dca32295a66ab26cd5ca7ef13308f26d8eee8a5cf9293c6821423b"} err="failed to get container status \"e5636642d3dca32295a66ab26cd5ca7ef13308f26d8eee8a5cf9293c6821423b\": rpc error: code = NotFound desc = could not find container \"e5636642d3dca32295a66ab26cd5ca7ef13308f26d8eee8a5cf9293c6821423b\": container with ID starting with e5636642d3dca32295a66ab26cd5ca7ef13308f26d8eee8a5cf9293c6821423b not found: ID does not exist" Jul 19 15:31:31.681138 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:31:31.681116 2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-fe155acd-786f7ccf45-d4hqq"] Jul 19 15:31:31.683096 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:31:31.683074 2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-fe155acd-786f7ccf45-d4hqq"] Jul 19 15:31:33.491573 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:31:33.491540 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-d4lll"] Jul 19 15:31:33.565141 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:31:33.565110 2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5b12f068-243d-4825-b621-f1a9bb46d55f" path="/var/lib/kubelet/pods/5b12f068-243d-4825-b621-f1a9bb46d55f/volumes" Jul 19 15:31:35.373545 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:31:35.373511 2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-fe155acd-dtl7b"] Jul 19 15:31:35.375463 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:31:35.375439 2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-fe155acd-dtl7b"] Jul 19 15:31:35.564980 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:31:35.564941 2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5eda7280-3e1c-44c4-947f-04b7370f4052" path="/var/lib/kubelet/pods/5eda7280-3e1c-44c4-947f-04b7370f4052/volumes" Jul 19 15:31:41.716803 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:31:41.716772 2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-fe155acd-openshift-default-794f59b879-wbh6q"] Jul 19 15:31:41.717271 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:31:41.716980 2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-fe155acd-openshift-default-794f59b879-wbh6q" podUID="45da29d4-a468-43a1-9ca2-7be168e13b00" containerName="istio-proxy" containerID="cri-o://f69a37e362c2283e40e4db7165deb2779d430744b749f65e7656e559480f1f95" gracePeriod=30 Jul 19 15:31:46.964945 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:31:46.964920 2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-fe155acd-openshift-default-794f59b879-wbh6q" Jul 19 15:31:47.065460 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:31:47.065382 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/45da29d4-a468-43a1-9ca2-7be168e13b00-credential-socket\") pod \"45da29d4-a468-43a1-9ca2-7be168e13b00\" (UID: \"45da29d4-a468-43a1-9ca2-7be168e13b00\") " Jul 19 15:31:47.065460 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:31:47.065432 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/45da29d4-a468-43a1-9ca2-7be168e13b00-istio-token\") pod \"45da29d4-a468-43a1-9ca2-7be168e13b00\" (UID: \"45da29d4-a468-43a1-9ca2-7be168e13b00\") " Jul 19 15:31:47.065671 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:31:47.065478 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/45da29d4-a468-43a1-9ca2-7be168e13b00-istiod-ca-cert\") pod \"45da29d4-a468-43a1-9ca2-7be168e13b00\" (UID: \"45da29d4-a468-43a1-9ca2-7be168e13b00\") " Jul 19 15:31:47.065671 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:31:47.065503 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/45da29d4-a468-43a1-9ca2-7be168e13b00-workload-certs\") pod \"45da29d4-a468-43a1-9ca2-7be168e13b00\" (UID: \"45da29d4-a468-43a1-9ca2-7be168e13b00\") " Jul 19 15:31:47.065671 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:31:47.065600 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bbjfj\" (UniqueName: \"kubernetes.io/projected/45da29d4-a468-43a1-9ca2-7be168e13b00-kube-api-access-bbjfj\") pod \"45da29d4-a468-43a1-9ca2-7be168e13b00\" (UID: \"45da29d4-a468-43a1-9ca2-7be168e13b00\") " Jul 19 15:31:47.065671 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:31:47.065647 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/45da29d4-a468-43a1-9ca2-7be168e13b00-istio-data\") pod \"45da29d4-a468-43a1-9ca2-7be168e13b00\" (UID: \"45da29d4-a468-43a1-9ca2-7be168e13b00\") " Jul 19 15:31:47.065873 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:31:47.065686 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/45da29d4-a468-43a1-9ca2-7be168e13b00-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "45da29d4-a468-43a1-9ca2-7be168e13b00" (UID: "45da29d4-a468-43a1-9ca2-7be168e13b00"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 19 15:31:47.065873 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:31:47.065690 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/45da29d4-a468-43a1-9ca2-7be168e13b00-istio-podinfo\") pod \"45da29d4-a468-43a1-9ca2-7be168e13b00\" (UID: \"45da29d4-a468-43a1-9ca2-7be168e13b00\") " Jul 19 15:31:47.065873 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:31:47.065779 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/45da29d4-a468-43a1-9ca2-7be168e13b00-istio-envoy\") pod \"45da29d4-a468-43a1-9ca2-7be168e13b00\" (UID: \"45da29d4-a468-43a1-9ca2-7be168e13b00\") " Jul 19 15:31:47.065873 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:31:47.065808 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/45da29d4-a468-43a1-9ca2-7be168e13b00-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "45da29d4-a468-43a1-9ca2-7be168e13b00" (UID: "45da29d4-a468-43a1-9ca2-7be168e13b00"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 19 15:31:47.065873 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:31:47.065825 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/45da29d4-a468-43a1-9ca2-7be168e13b00-workload-socket\") pod \"45da29d4-a468-43a1-9ca2-7be168e13b00\" (UID: \"45da29d4-a468-43a1-9ca2-7be168e13b00\") " Jul 19 15:31:47.066161 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:31:47.065872 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/45da29d4-a468-43a1-9ca2-7be168e13b00-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "45da29d4-a468-43a1-9ca2-7be168e13b00" (UID: "45da29d4-a468-43a1-9ca2-7be168e13b00"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 19 15:31:47.066161 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:31:47.066075 2558 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/45da29d4-a468-43a1-9ca2-7be168e13b00-credential-socket\") on node \"ip-10-0-138-220.ec2.internal\" DevicePath \"\"" Jul 19 15:31:47.066161 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:31:47.066092 2558 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/45da29d4-a468-43a1-9ca2-7be168e13b00-istiod-ca-cert\") on node \"ip-10-0-138-220.ec2.internal\" DevicePath \"\"" Jul 19 15:31:47.066161 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:31:47.066105 2558 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/45da29d4-a468-43a1-9ca2-7be168e13b00-workload-certs\") on node \"ip-10-0-138-220.ec2.internal\" DevicePath \"\"" Jul 19 15:31:47.066364 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:31:47.066266 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/45da29d4-a468-43a1-9ca2-7be168e13b00-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "45da29d4-a468-43a1-9ca2-7be168e13b00" (UID: "45da29d4-a468-43a1-9ca2-7be168e13b00"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 19 15:31:47.067209 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:31:47.067167 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/45da29d4-a468-43a1-9ca2-7be168e13b00-istio-data" (OuterVolumeSpecName: "istio-data") pod "45da29d4-a468-43a1-9ca2-7be168e13b00" (UID: "45da29d4-a468-43a1-9ca2-7be168e13b00"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 19 15:31:47.068084 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:31:47.068057 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/45da29d4-a468-43a1-9ca2-7be168e13b00-istio-token" (OuterVolumeSpecName: "istio-token") pod "45da29d4-a468-43a1-9ca2-7be168e13b00" (UID: "45da29d4-a468-43a1-9ca2-7be168e13b00"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 19 15:31:47.068192 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:31:47.068112 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/45da29d4-a468-43a1-9ca2-7be168e13b00-kube-api-access-bbjfj" (OuterVolumeSpecName: "kube-api-access-bbjfj") pod "45da29d4-a468-43a1-9ca2-7be168e13b00" (UID: "45da29d4-a468-43a1-9ca2-7be168e13b00"). InnerVolumeSpecName "kube-api-access-bbjfj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 19 15:31:47.068192 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:31:47.068180 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/45da29d4-a468-43a1-9ca2-7be168e13b00-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "45da29d4-a468-43a1-9ca2-7be168e13b00" (UID: "45da29d4-a468-43a1-9ca2-7be168e13b00"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 19 15:31:47.068278 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:31:47.068184 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/45da29d4-a468-43a1-9ca2-7be168e13b00-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "45da29d4-a468-43a1-9ca2-7be168e13b00" (UID: "45da29d4-a468-43a1-9ca2-7be168e13b00"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 19 15:31:47.167194 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:31:47.167173 2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bbjfj\" (UniqueName: \"kubernetes.io/projected/45da29d4-a468-43a1-9ca2-7be168e13b00-kube-api-access-bbjfj\") on node \"ip-10-0-138-220.ec2.internal\" DevicePath \"\"" Jul 19 15:31:47.167194 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:31:47.167192 2558 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/45da29d4-a468-43a1-9ca2-7be168e13b00-istio-data\") on node \"ip-10-0-138-220.ec2.internal\" DevicePath \"\"" Jul 19 15:31:47.167319 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:31:47.167203 2558 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/45da29d4-a468-43a1-9ca2-7be168e13b00-istio-podinfo\") on node \"ip-10-0-138-220.ec2.internal\" DevicePath \"\"" Jul 19 15:31:47.167319 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:31:47.167210 2558 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/45da29d4-a468-43a1-9ca2-7be168e13b00-istio-envoy\") on node \"ip-10-0-138-220.ec2.internal\" DevicePath \"\"" Jul 19 15:31:47.167319 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:31:47.167219 2558 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/45da29d4-a468-43a1-9ca2-7be168e13b00-workload-socket\") on node \"ip-10-0-138-220.ec2.internal\" DevicePath \"\"" Jul 19 15:31:47.167319 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:31:47.167228 2558 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/45da29d4-a468-43a1-9ca2-7be168e13b00-istio-token\") on node \"ip-10-0-138-220.ec2.internal\" DevicePath \"\"" Jul 19 15:31:47.737858 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:31:47.737824 2558 generic.go:356] "Generic (PLEG): container finished" podID="45da29d4-a468-43a1-9ca2-7be168e13b00" containerID="f69a37e362c2283e40e4db7165deb2779d430744b749f65e7656e559480f1f95" exitCode=0 Jul 19 15:31:47.738074 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:31:47.737877 2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-fe155acd-openshift-default-794f59b879-wbh6q" Jul 19 15:31:47.738074 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:31:47.737893 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-fe155acd-openshift-default-794f59b879-wbh6q" event={"ID":"45da29d4-a468-43a1-9ca2-7be168e13b00","Type":"ContainerDied","Data":"f69a37e362c2283e40e4db7165deb2779d430744b749f65e7656e559480f1f95"} Jul 19 15:31:47.738074 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:31:47.737936 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-fe155acd-openshift-default-794f59b879-wbh6q" event={"ID":"45da29d4-a468-43a1-9ca2-7be168e13b00","Type":"ContainerDied","Data":"a5ab28659cf27d0505924473d48ee99f5497a19fca9a57d1a0320bee145bc566"} Jul 19 15:31:47.738074 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:31:47.737958 2558 scope.go:117] "RemoveContainer" containerID="f69a37e362c2283e40e4db7165deb2779d430744b749f65e7656e559480f1f95" Jul 19 15:31:47.746200 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:31:47.746182 2558 scope.go:117] "RemoveContainer" containerID="d5edd49cb8c7e2a8d7ab88507211d208bcb6270536b34e919b4a3f69452a6108" Jul 19 15:31:47.753202 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:31:47.753178 2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-fe155acd-openshift-default-794f59b879-wbh6q"] Jul 19 15:31:47.753996 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:31:47.753945 2558 scope.go:117] "RemoveContainer" containerID="f69a37e362c2283e40e4db7165deb2779d430744b749f65e7656e559480f1f95" Jul 19 15:31:47.754396 ip-10-0-138-220 kubenswrapper[2558]: E0719 15:31:47.754358 2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f69a37e362c2283e40e4db7165deb2779d430744b749f65e7656e559480f1f95\": container with ID starting with f69a37e362c2283e40e4db7165deb2779d430744b749f65e7656e559480f1f95 not found: ID does not exist" containerID="f69a37e362c2283e40e4db7165deb2779d430744b749f65e7656e559480f1f95" Jul 19 15:31:47.754496 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:31:47.754393 2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f69a37e362c2283e40e4db7165deb2779d430744b749f65e7656e559480f1f95"} err="failed to get container status \"f69a37e362c2283e40e4db7165deb2779d430744b749f65e7656e559480f1f95\": rpc error: code = NotFound desc = could not find container \"f69a37e362c2283e40e4db7165deb2779d430744b749f65e7656e559480f1f95\": container with ID starting with f69a37e362c2283e40e4db7165deb2779d430744b749f65e7656e559480f1f95 not found: ID does not exist" Jul 19 15:31:47.754496 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:31:47.754412 2558 scope.go:117] "RemoveContainer" containerID="d5edd49cb8c7e2a8d7ab88507211d208bcb6270536b34e919b4a3f69452a6108" Jul 19 15:31:47.754692 ip-10-0-138-220 kubenswrapper[2558]: E0719 15:31:47.754674 2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d5edd49cb8c7e2a8d7ab88507211d208bcb6270536b34e919b4a3f69452a6108\": container with ID starting with d5edd49cb8c7e2a8d7ab88507211d208bcb6270536b34e919b4a3f69452a6108 not found: ID does not exist" containerID="d5edd49cb8c7e2a8d7ab88507211d208bcb6270536b34e919b4a3f69452a6108" Jul 19 15:31:47.754734 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:31:47.754698 2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d5edd49cb8c7e2a8d7ab88507211d208bcb6270536b34e919b4a3f69452a6108"} err="failed to get container status \"d5edd49cb8c7e2a8d7ab88507211d208bcb6270536b34e919b4a3f69452a6108\": rpc error: code = NotFound desc = could not find container \"d5edd49cb8c7e2a8d7ab88507211d208bcb6270536b34e919b4a3f69452a6108\": container with ID starting with d5edd49cb8c7e2a8d7ab88507211d208bcb6270536b34e919b4a3f69452a6108 not found: ID does not exist" Jul 19 15:31:47.755223 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:31:47.755202 2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-fe155acd-openshift-default-794f59b879-wbh6q"] Jul 19 15:31:49.564111 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:31:49.564079 2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="45da29d4-a468-43a1-9ca2-7be168e13b00" path="/var/lib/kubelet/pods/45da29d4-a468-43a1-9ca2-7be168e13b00/volumes" Jul 19 15:32:00.987649 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:32:00.987564 2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-cc1f323c-openshift-default-55d68dcf67-p748l"] Jul 19 15:32:00.988197 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:32:00.987976 2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="45da29d4-a468-43a1-9ca2-7be168e13b00" containerName="istio-proxy" Jul 19 15:32:00.988197 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:32:00.987994 2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="45da29d4-a468-43a1-9ca2-7be168e13b00" containerName="istio-proxy" Jul 19 15:32:00.988197 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:32:00.988008 2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5b12f068-243d-4825-b621-f1a9bb46d55f" containerName="maas-api" Jul 19 15:32:00.988197 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:32:00.988016 2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="5b12f068-243d-4825-b621-f1a9bb46d55f" containerName="maas-api" Jul 19 15:32:00.988197 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:32:00.988053 2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5eda7280-3e1c-44c4-947f-04b7370f4052" containerName="revoke-keys" Jul 19 15:32:00.988197 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:32:00.988062 2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="5eda7280-3e1c-44c4-947f-04b7370f4052" containerName="revoke-keys" Jul 19 15:32:00.988197 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:32:00.988076 2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="45da29d4-a468-43a1-9ca2-7be168e13b00" containerName="istio-proxy" Jul 19 15:32:00.988197 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:32:00.988084 2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="45da29d4-a468-43a1-9ca2-7be168e13b00" containerName="istio-proxy" Jul 19 15:32:00.988197 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:32:00.988151 2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="45da29d4-a468-43a1-9ca2-7be168e13b00" containerName="istio-proxy" Jul 19 15:32:00.988197 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:32:00.988162 2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="45da29d4-a468-43a1-9ca2-7be168e13b00" containerName="istio-proxy" Jul 19 15:32:00.988197 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:32:00.988172 2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="5b12f068-243d-4825-b621-f1a9bb46d55f" containerName="maas-api" Jul 19 15:32:00.988197 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:32:00.988179 2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="5eda7280-3e1c-44c4-947f-04b7370f4052" containerName="revoke-keys" Jul 19 15:32:00.992984 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:32:00.992960 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-cc1f323c-openshift-default-55d68dcf67-p748l" Jul 19 15:32:00.995186 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:32:00.995169 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-cc1f323c-openshift-default-dockercfg-4kqmx\"" Jul 19 15:32:01.002525 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:32:01.002501 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-cc1f323c-openshift-default-55d68dcf67-p748l"] Jul 19 15:32:01.071735 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:32:01.071709 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d1d23079-96b6-4021-9dbc-4f3795ed414e-istio-token\") pod \"e2e-mt-cc1f323c-openshift-default-55d68dcf67-p748l\" (UID: \"d1d23079-96b6-4021-9dbc-4f3795ed414e\") " pod="openshift-ingress/e2e-mt-cc1f323c-openshift-default-55d68dcf67-p748l" Jul 19 15:32:01.071837 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:32:01.071745 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vl7mc\" (UniqueName: \"kubernetes.io/projected/d1d23079-96b6-4021-9dbc-4f3795ed414e-kube-api-access-vl7mc\") pod \"e2e-mt-cc1f323c-openshift-default-55d68dcf67-p748l\" (UID: \"d1d23079-96b6-4021-9dbc-4f3795ed414e\") " pod="openshift-ingress/e2e-mt-cc1f323c-openshift-default-55d68dcf67-p748l" Jul 19 15:32:01.071837 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:32:01.071767 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d1d23079-96b6-4021-9dbc-4f3795ed414e-istio-envoy\") pod \"e2e-mt-cc1f323c-openshift-default-55d68dcf67-p748l\" (UID: \"d1d23079-96b6-4021-9dbc-4f3795ed414e\") " pod="openshift-ingress/e2e-mt-cc1f323c-openshift-default-55d68dcf67-p748l" Jul 19 15:32:01.071837 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:32:01.071787 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1d23079-96b6-4021-9dbc-4f3795ed414e-credential-socket\") pod \"e2e-mt-cc1f323c-openshift-default-55d68dcf67-p748l\" (UID: \"d1d23079-96b6-4021-9dbc-4f3795ed414e\") " pod="openshift-ingress/e2e-mt-cc1f323c-openshift-default-55d68dcf67-p748l" Jul 19 15:32:01.071837 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:32:01.071806 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d1d23079-96b6-4021-9dbc-4f3795ed414e-istiod-ca-cert\") pod \"e2e-mt-cc1f323c-openshift-default-55d68dcf67-p748l\" (UID: \"d1d23079-96b6-4021-9dbc-4f3795ed414e\") " pod="openshift-ingress/e2e-mt-cc1f323c-openshift-default-55d68dcf67-p748l" Jul 19 15:32:01.071989 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:32:01.071875 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d1d23079-96b6-4021-9dbc-4f3795ed414e-workload-certs\") pod \"e2e-mt-cc1f323c-openshift-default-55d68dcf67-p748l\" (UID: \"d1d23079-96b6-4021-9dbc-4f3795ed414e\") " pod="openshift-ingress/e2e-mt-cc1f323c-openshift-default-55d68dcf67-p748l" Jul 19 15:32:01.071989 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:32:01.071899 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d1d23079-96b6-4021-9dbc-4f3795ed414e-istio-podinfo\") pod \"e2e-mt-cc1f323c-openshift-default-55d68dcf67-p748l\" (UID: \"d1d23079-96b6-4021-9dbc-4f3795ed414e\") " pod="openshift-ingress/e2e-mt-cc1f323c-openshift-default-55d68dcf67-p748l" Jul 19 15:32:01.071989 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:32:01.071916 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1d23079-96b6-4021-9dbc-4f3795ed414e-workload-socket\") pod \"e2e-mt-cc1f323c-openshift-default-55d68dcf67-p748l\" (UID: \"d1d23079-96b6-4021-9dbc-4f3795ed414e\") " pod="openshift-ingress/e2e-mt-cc1f323c-openshift-default-55d68dcf67-p748l" Jul 19 15:32:01.071989 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:32:01.071932 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d1d23079-96b6-4021-9dbc-4f3795ed414e-istio-data\") pod \"e2e-mt-cc1f323c-openshift-default-55d68dcf67-p748l\" (UID: \"d1d23079-96b6-4021-9dbc-4f3795ed414e\") " pod="openshift-ingress/e2e-mt-cc1f323c-openshift-default-55d68dcf67-p748l" Jul 19 15:32:01.173175 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:32:01.173144 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d1d23079-96b6-4021-9dbc-4f3795ed414e-istio-envoy\") pod \"e2e-mt-cc1f323c-openshift-default-55d68dcf67-p748l\" (UID: \"d1d23079-96b6-4021-9dbc-4f3795ed414e\") " pod="openshift-ingress/e2e-mt-cc1f323c-openshift-default-55d68dcf67-p748l" Jul 19 15:32:01.173306 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:32:01.173179 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1d23079-96b6-4021-9dbc-4f3795ed414e-credential-socket\") pod \"e2e-mt-cc1f323c-openshift-default-55d68dcf67-p748l\" (UID: \"d1d23079-96b6-4021-9dbc-4f3795ed414e\") " pod="openshift-ingress/e2e-mt-cc1f323c-openshift-default-55d68dcf67-p748l" Jul 19 15:32:01.173306 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:32:01.173198 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d1d23079-96b6-4021-9dbc-4f3795ed414e-istiod-ca-cert\") pod \"e2e-mt-cc1f323c-openshift-default-55d68dcf67-p748l\" (UID: \"d1d23079-96b6-4021-9dbc-4f3795ed414e\") " pod="openshift-ingress/e2e-mt-cc1f323c-openshift-default-55d68dcf67-p748l" Jul 19 15:32:01.173306 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:32:01.173245 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d1d23079-96b6-4021-9dbc-4f3795ed414e-workload-certs\") pod \"e2e-mt-cc1f323c-openshift-default-55d68dcf67-p748l\" (UID: \"d1d23079-96b6-4021-9dbc-4f3795ed414e\") " pod="openshift-ingress/e2e-mt-cc1f323c-openshift-default-55d68dcf67-p748l" Jul 19 15:32:01.173306 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:32:01.173261 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d1d23079-96b6-4021-9dbc-4f3795ed414e-istio-podinfo\") pod \"e2e-mt-cc1f323c-openshift-default-55d68dcf67-p748l\" (UID: \"d1d23079-96b6-4021-9dbc-4f3795ed414e\") " pod="openshift-ingress/e2e-mt-cc1f323c-openshift-default-55d68dcf67-p748l" Jul 19 15:32:01.173306 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:32:01.173280 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1d23079-96b6-4021-9dbc-4f3795ed414e-workload-socket\") pod \"e2e-mt-cc1f323c-openshift-default-55d68dcf67-p748l\" (UID: \"d1d23079-96b6-4021-9dbc-4f3795ed414e\") " pod="openshift-ingress/e2e-mt-cc1f323c-openshift-default-55d68dcf67-p748l" Jul 19 15:32:01.173306 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:32:01.173296 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d1d23079-96b6-4021-9dbc-4f3795ed414e-istio-data\") pod \"e2e-mt-cc1f323c-openshift-default-55d68dcf67-p748l\" (UID: \"d1d23079-96b6-4021-9dbc-4f3795ed414e\") " pod="openshift-ingress/e2e-mt-cc1f323c-openshift-default-55d68dcf67-p748l" Jul 19 15:32:01.173613 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:32:01.173317 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d1d23079-96b6-4021-9dbc-4f3795ed414e-istio-token\") pod \"e2e-mt-cc1f323c-openshift-default-55d68dcf67-p748l\" (UID: \"d1d23079-96b6-4021-9dbc-4f3795ed414e\") " pod="openshift-ingress/e2e-mt-cc1f323c-openshift-default-55d68dcf67-p748l" Jul 19 15:32:01.173613 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:32:01.173349 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vl7mc\" (UniqueName: \"kubernetes.io/projected/d1d23079-96b6-4021-9dbc-4f3795ed414e-kube-api-access-vl7mc\") pod \"e2e-mt-cc1f323c-openshift-default-55d68dcf67-p748l\" (UID: \"d1d23079-96b6-4021-9dbc-4f3795ed414e\") " pod="openshift-ingress/e2e-mt-cc1f323c-openshift-default-55d68dcf67-p748l" Jul 19 15:32:01.173721 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:32:01.173652 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d1d23079-96b6-4021-9dbc-4f3795ed414e-workload-certs\") pod \"e2e-mt-cc1f323c-openshift-default-55d68dcf67-p748l\" (UID: \"d1d23079-96b6-4021-9dbc-4f3795ed414e\") " pod="openshift-ingress/e2e-mt-cc1f323c-openshift-default-55d68dcf67-p748l" Jul 19 15:32:01.173777 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:32:01.173718 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d1d23079-96b6-4021-9dbc-4f3795ed414e-istio-data\") pod \"e2e-mt-cc1f323c-openshift-default-55d68dcf67-p748l\" (UID: \"d1d23079-96b6-4021-9dbc-4f3795ed414e\") " pod="openshift-ingress/e2e-mt-cc1f323c-openshift-default-55d68dcf67-p748l" Jul 19 15:32:01.173954 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:32:01.173933 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1d23079-96b6-4021-9dbc-4f3795ed414e-workload-socket\") pod \"e2e-mt-cc1f323c-openshift-default-55d68dcf67-p748l\" (UID: \"d1d23079-96b6-4021-9dbc-4f3795ed414e\") " pod="openshift-ingress/e2e-mt-cc1f323c-openshift-default-55d68dcf67-p748l" Jul 19 15:32:01.174142 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:32:01.173990 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d1d23079-96b6-4021-9dbc-4f3795ed414e-istiod-ca-cert\") pod \"e2e-mt-cc1f323c-openshift-default-55d68dcf67-p748l\" (UID: \"d1d23079-96b6-4021-9dbc-4f3795ed414e\") " pod="openshift-ingress/e2e-mt-cc1f323c-openshift-default-55d68dcf67-p748l" Jul 19 15:32:01.174245 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:32:01.174220 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1d23079-96b6-4021-9dbc-4f3795ed414e-credential-socket\") pod \"e2e-mt-cc1f323c-openshift-default-55d68dcf67-p748l\" (UID: \"d1d23079-96b6-4021-9dbc-4f3795ed414e\") " pod="openshift-ingress/e2e-mt-cc1f323c-openshift-default-55d68dcf67-p748l" Jul 19 15:32:01.175563 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:32:01.175544 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d1d23079-96b6-4021-9dbc-4f3795ed414e-istio-envoy\") pod \"e2e-mt-cc1f323c-openshift-default-55d68dcf67-p748l\" (UID: \"d1d23079-96b6-4021-9dbc-4f3795ed414e\") " pod="openshift-ingress/e2e-mt-cc1f323c-openshift-default-55d68dcf67-p748l" Jul 19 15:32:01.175918 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:32:01.175901 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d1d23079-96b6-4021-9dbc-4f3795ed414e-istio-podinfo\") pod \"e2e-mt-cc1f323c-openshift-default-55d68dcf67-p748l\" (UID: \"d1d23079-96b6-4021-9dbc-4f3795ed414e\") " pod="openshift-ingress/e2e-mt-cc1f323c-openshift-default-55d68dcf67-p748l" Jul 19 15:32:01.180731 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:32:01.180700 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vl7mc\" (UniqueName: \"kubernetes.io/projected/d1d23079-96b6-4021-9dbc-4f3795ed414e-kube-api-access-vl7mc\") pod \"e2e-mt-cc1f323c-openshift-default-55d68dcf67-p748l\" (UID: \"d1d23079-96b6-4021-9dbc-4f3795ed414e\") " pod="openshift-ingress/e2e-mt-cc1f323c-openshift-default-55d68dcf67-p748l" Jul 19 15:32:01.180830 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:32:01.180808 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d1d23079-96b6-4021-9dbc-4f3795ed414e-istio-token\") pod \"e2e-mt-cc1f323c-openshift-default-55d68dcf67-p748l\" (UID: \"d1d23079-96b6-4021-9dbc-4f3795ed414e\") " pod="openshift-ingress/e2e-mt-cc1f323c-openshift-default-55d68dcf67-p748l" Jul 19 15:32:01.303726 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:32:01.303655 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-cc1f323c-openshift-default-55d68dcf67-p748l" Jul 19 15:32:01.436395 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:32:01.436359 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-cc1f323c-openshift-default-55d68dcf67-p748l"] Jul 19 15:32:01.444275 ip-10-0-138-220 kubenswrapper[2558]: W0719 15:32:01.444249 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd1d23079_96b6_4021_9dbc_4f3795ed414e.slice/crio-fb3d4280a33b0054ffedcfd79b80eb4287612edd256ea335c779ba951c39312b WatchSource:0}: Error finding container fb3d4280a33b0054ffedcfd79b80eb4287612edd256ea335c779ba951c39312b: Status 404 returned error can't find the container with id fb3d4280a33b0054ffedcfd79b80eb4287612edd256ea335c779ba951c39312b Jul 19 15:32:01.446428 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:32:01.446390 2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 19 15:32:01.446628 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:32:01.446478 2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 19 15:32:01.446628 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:32:01.446515 2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 19 15:32:01.800294 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:32:01.800264 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-cc1f323c-openshift-default-55d68dcf67-p748l" event={"ID":"d1d23079-96b6-4021-9dbc-4f3795ed414e","Type":"ContainerStarted","Data":"6351f38ef2840bdd15214ff73ea4d82554291a59a232383b1e06260ea662c8de"} Jul 19 15:32:01.800485 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:32:01.800313 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-cc1f323c-openshift-default-55d68dcf67-p748l" event={"ID":"d1d23079-96b6-4021-9dbc-4f3795ed414e","Type":"ContainerStarted","Data":"fb3d4280a33b0054ffedcfd79b80eb4287612edd256ea335c779ba951c39312b"} Jul 19 15:32:01.818258 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:32:01.818211 2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-cc1f323c-openshift-default-55d68dcf67-p748l" podStartSLOduration=1.818195403 podStartE2EDuration="1.818195403s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-19 15:32:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-19 15:32:01.817150002 +0000 UTC m=+2062.738852717" watchObservedRunningTime="2026-07-19 15:32:01.818195403 +0000 UTC m=+2062.739898117" Jul 19 15:32:02.304660 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:32:02.304626 2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-cc1f323c-openshift-default-55d68dcf67-p748l" Jul 19 15:32:02.309174 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:32:02.309146 2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-cc1f323c-openshift-default-55d68dcf67-p748l" Jul 19 15:32:02.804666 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:32:02.804622 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-cc1f323c-openshift-default-55d68dcf67-p748l" Jul 19 15:32:02.805594 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:32:02.805570 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-cc1f323c-openshift-default-55d68dcf67-p748l" Jul 19 15:32:03.906837 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:32:03.906805 2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-cc1f323c-8466db7d65-pglmv"] Jul 19 15:32:03.910143 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:32:03.910125 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-cc1f323c-8466db7d65-pglmv" Jul 19 15:32:03.912183 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:32:03.912154 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-cc1f323c-serving-cert\"" Jul 19 15:32:03.912322 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:32:03.912303 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-ljt6w\"" Jul 19 15:32:03.912395 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:32:03.912310 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 19 15:32:03.912807 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:32:03.912788 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Jul 19 15:32:03.917958 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:32:03.917542 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-cc1f323c-8466db7d65-pglmv"] Jul 19 15:32:03.996651 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:32:03.996626 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q7srq\" (UniqueName: \"kubernetes.io/projected/d7b634f7-a402-49cb-8b2c-6363e6ac28b0-kube-api-access-q7srq\") pod \"maas-api-e2e-mt-cc1f323c-8466db7d65-pglmv\" (UID: \"d7b634f7-a402-49cb-8b2c-6363e6ac28b0\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-cc1f323c-8466db7d65-pglmv" Jul 19 15:32:03.996767 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:32:03.996675 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d7b634f7-a402-49cb-8b2c-6363e6ac28b0-openshift-service-ca\") pod \"maas-api-e2e-mt-cc1f323c-8466db7d65-pglmv\" (UID: \"d7b634f7-a402-49cb-8b2c-6363e6ac28b0\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-cc1f323c-8466db7d65-pglmv" Jul 19 15:32:03.996809 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:32:03.996765 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d7b634f7-a402-49cb-8b2c-6363e6ac28b0-maas-api-tls\") pod \"maas-api-e2e-mt-cc1f323c-8466db7d65-pglmv\" (UID: \"d7b634f7-a402-49cb-8b2c-6363e6ac28b0\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-cc1f323c-8466db7d65-pglmv" Jul 19 15:32:04.097529 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:32:04.097501 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d7b634f7-a402-49cb-8b2c-6363e6ac28b0-openshift-service-ca\") pod \"maas-api-e2e-mt-cc1f323c-8466db7d65-pglmv\" (UID: \"d7b634f7-a402-49cb-8b2c-6363e6ac28b0\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-cc1f323c-8466db7d65-pglmv" Jul 19 15:32:04.097669 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:32:04.097551 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d7b634f7-a402-49cb-8b2c-6363e6ac28b0-maas-api-tls\") pod \"maas-api-e2e-mt-cc1f323c-8466db7d65-pglmv\" (UID: \"d7b634f7-a402-49cb-8b2c-6363e6ac28b0\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-cc1f323c-8466db7d65-pglmv" Jul 19 15:32:04.097669 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:32:04.097582 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q7srq\" (UniqueName: \"kubernetes.io/projected/d7b634f7-a402-49cb-8b2c-6363e6ac28b0-kube-api-access-q7srq\") pod \"maas-api-e2e-mt-cc1f323c-8466db7d65-pglmv\" (UID: \"d7b634f7-a402-49cb-8b2c-6363e6ac28b0\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-cc1f323c-8466db7d65-pglmv" Jul 19 15:32:04.097746 ip-10-0-138-220 kubenswrapper[2558]: E0719 15:32:04.097680 2558 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-cc1f323c-serving-cert: secret "maas-api-e2e-mt-cc1f323c-serving-cert" not found Jul 19 15:32:04.097780 ip-10-0-138-220 kubenswrapper[2558]: E0719 15:32:04.097754 2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d7b634f7-a402-49cb-8b2c-6363e6ac28b0-maas-api-tls podName:d7b634f7-a402-49cb-8b2c-6363e6ac28b0 nodeName:}" failed. No retries permitted until 2026-07-19 15:32:04.59773371 +0000 UTC m=+2065.519436413 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/d7b634f7-a402-49cb-8b2c-6363e6ac28b0-maas-api-tls") pod "maas-api-e2e-mt-cc1f323c-8466db7d65-pglmv" (UID: "d7b634f7-a402-49cb-8b2c-6363e6ac28b0") : secret "maas-api-e2e-mt-cc1f323c-serving-cert" not found Jul 19 15:32:04.098162 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:32:04.098146 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d7b634f7-a402-49cb-8b2c-6363e6ac28b0-openshift-service-ca\") pod \"maas-api-e2e-mt-cc1f323c-8466db7d65-pglmv\" (UID: \"d7b634f7-a402-49cb-8b2c-6363e6ac28b0\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-cc1f323c-8466db7d65-pglmv" Jul 19 15:32:04.106504 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:32:04.106479 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q7srq\" (UniqueName: \"kubernetes.io/projected/d7b634f7-a402-49cb-8b2c-6363e6ac28b0-kube-api-access-q7srq\") pod \"maas-api-e2e-mt-cc1f323c-8466db7d65-pglmv\" (UID: \"d7b634f7-a402-49cb-8b2c-6363e6ac28b0\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-cc1f323c-8466db7d65-pglmv" Jul 19 15:32:04.602330 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:32:04.602288 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d7b634f7-a402-49cb-8b2c-6363e6ac28b0-maas-api-tls\") pod \"maas-api-e2e-mt-cc1f323c-8466db7d65-pglmv\" (UID: \"d7b634f7-a402-49cb-8b2c-6363e6ac28b0\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-cc1f323c-8466db7d65-pglmv" Jul 19 15:32:04.604945 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:32:04.604919 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d7b634f7-a402-49cb-8b2c-6363e6ac28b0-maas-api-tls\") pod \"maas-api-e2e-mt-cc1f323c-8466db7d65-pglmv\" (UID: \"d7b634f7-a402-49cb-8b2c-6363e6ac28b0\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-cc1f323c-8466db7d65-pglmv" Jul 19 15:32:04.823068 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:32:04.823007 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-cc1f323c-8466db7d65-pglmv" Jul 19 15:32:04.957534 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:32:04.957498 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-cc1f323c-8466db7d65-pglmv"] Jul 19 15:32:04.960231 ip-10-0-138-220 kubenswrapper[2558]: W0719 15:32:04.960206 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd7b634f7_a402_49cb_8b2c_6363e6ac28b0.slice/crio-cabbc3dc6407c068e320c29157672d5bd27faf499a981dd1098739f82213e151 WatchSource:0}: Error finding container cabbc3dc6407c068e320c29157672d5bd27faf499a981dd1098739f82213e151: Status 404 returned error can't find the container with id cabbc3dc6407c068e320c29157672d5bd27faf499a981dd1098739f82213e151 Jul 19 15:32:05.817820 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:32:05.817782 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-cc1f323c-8466db7d65-pglmv" event={"ID":"d7b634f7-a402-49cb-8b2c-6363e6ac28b0","Type":"ContainerStarted","Data":"bc0b5028e62d45c85d0d71c5a1745a3fb85a20c29edb7a5a7b5a5c156f0d41a4"} Jul 19 15:32:05.817994 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:32:05.817835 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-cc1f323c-8466db7d65-pglmv" event={"ID":"d7b634f7-a402-49cb-8b2c-6363e6ac28b0","Type":"ContainerStarted","Data":"cabbc3dc6407c068e320c29157672d5bd27faf499a981dd1098739f82213e151"} Jul 19 15:32:05.817994 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:32:05.817916 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-cc1f323c-8466db7d65-pglmv" Jul 19 15:32:05.836454 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:32:05.836403 2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-cc1f323c-8466db7d65-pglmv" podStartSLOduration=2.492823234 podStartE2EDuration="2.836390573s" totalImagesPullingTime="343.567339ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-19 15:32:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-19 15:32:05.834236122 +0000 UTC m=+2066.755938837" watchObservedRunningTime="2026-07-19 15:32:05.836390573 +0000 UTC m=+2066.758093286" Jul 19 15:32:08.685675 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:32:08.685640 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-d4lll"] Jul 19 15:32:11.827953 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:32:11.827922 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-cc1f323c-8466db7d65-pglmv" Jul 19 15:32:16.873732 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:32:16.873702 2558 generic.go:356] "Generic (PLEG): container finished" podID="d1d23079-96b6-4021-9dbc-4f3795ed414e" containerID="6351f38ef2840bdd15214ff73ea4d82554291a59a232383b1e06260ea662c8de" exitCode=0 Jul 19 15:32:16.874152 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:32:16.873776 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-cc1f323c-openshift-default-55d68dcf67-p748l" event={"ID":"d1d23079-96b6-4021-9dbc-4f3795ed414e","Type":"ContainerDied","Data":"6351f38ef2840bdd15214ff73ea4d82554291a59a232383b1e06260ea662c8de"} Jul 19 15:32:16.874211 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:32:16.874171 2558 scope.go:117] "RemoveContainer" containerID="6351f38ef2840bdd15214ff73ea4d82554291a59a232383b1e06260ea662c8de" Jul 19 15:32:16.875147 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:32:16.875117 2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 19 15:32:16.875229 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:32:16.875215 2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 19 15:32:16.875269 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:32:16.875251 2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 19 15:32:17.805071 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:32:17.805013 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-cc1f323c-openshift-default-55d68dcf67-p748l" Jul 19 15:32:17.881293 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:32:17.881254 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-cc1f323c-openshift-default-55d68dcf67-p748l" event={"ID":"d1d23079-96b6-4021-9dbc-4f3795ed414e","Type":"ContainerStarted","Data":"87c129b90ea7917f13086f7f3a890c93b08e3909edbba4e809c4e31dc260af00"} Jul 19 15:32:18.304857 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:32:18.304813 2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-cc1f323c-openshift-default-55d68dcf67-p748l" Jul 19 15:32:18.305963 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:32:18.305940 2558 patch_prober.go:28] interesting pod/e2e-mt-cc1f323c-openshift-default-55d68dcf67-p748l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused" start-of-body= Jul 19 15:32:18.306050 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:32:18.305988 2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-cc1f323c-openshift-default-55d68dcf67-p748l" podUID="d1d23079-96b6-4021-9dbc-4f3795ed414e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused" Jul 19 15:32:19.304583 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:32:19.304546 2558 patch_prober.go:28] interesting pod/e2e-mt-cc1f323c-openshift-default-55d68dcf67-p748l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused" start-of-body= Jul 19 15:32:19.304981 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:32:19.304609 2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-cc1f323c-openshift-default-55d68dcf67-p748l" podUID="d1d23079-96b6-4021-9dbc-4f3795ed414e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused" Jul 19 15:32:19.689413 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:32:19.689384 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-d4lll"] Jul 19 15:32:20.304319 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:32:20.304282 2558 patch_prober.go:28] interesting pod/e2e-mt-cc1f323c-openshift-default-55d68dcf67-p748l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused" start-of-body= Jul 19 15:32:20.304508 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:32:20.304337 2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-cc1f323c-openshift-default-55d68dcf67-p748l" podUID="d1d23079-96b6-4021-9dbc-4f3795ed414e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused" Jul 19 15:32:21.304573 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:32:21.304532 2558 patch_prober.go:28] interesting pod/e2e-mt-cc1f323c-openshift-default-55d68dcf67-p748l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused" start-of-body= Jul 19 15:32:21.304972 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:32:21.304586 2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-cc1f323c-openshift-default-55d68dcf67-p748l" podUID="d1d23079-96b6-4021-9dbc-4f3795ed414e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused" Jul 19 15:32:22.307998 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:32:22.307969 2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-cc1f323c-openshift-default-55d68dcf67-p748l" Jul 19 15:32:22.308397 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:32:22.308379 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-cc1f323c-openshift-default-55d68dcf67-p748l" Jul 19 15:32:22.308949 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:32:22.308925 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-cc1f323c-openshift-default-55d68dcf67-p748l" Jul 19 15:32:39.643827 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:32:39.643788 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-5wfzs_5b2d3a07-86b2-4ba2-8a71-c2d89ada79c6/ovn-acl-logging/0.log" Jul 19 15:32:39.649057 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:32:39.649037 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-5wfzs_5b2d3a07-86b2-4ba2-8a71-c2d89ada79c6/ovn-acl-logging/0.log" Jul 19 15:32:40.329157 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:32:40.329125 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-d4lll"] Jul 19 15:33:00.391268 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:33:00.391234 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-d4lll"] Jul 19 15:33:02.094534 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:33:02.094503 2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-cc1f323c-8466db7d65-pglmv"] Jul 19 15:33:02.094976 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:33:02.094759 2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-cc1f323c-8466db7d65-pglmv" podUID="d7b634f7-a402-49cb-8b2c-6363e6ac28b0" containerName="maas-api" containerID="cri-o://bc0b5028e62d45c85d0d71c5a1745a3fb85a20c29edb7a5a7b5a5c156f0d41a4" gracePeriod=30 Jul 19 15:33:02.333323 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:33:02.333296 2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-cc1f323c-8466db7d65-pglmv" Jul 19 15:33:02.460136 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:33:02.460109 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d7b634f7-a402-49cb-8b2c-6363e6ac28b0-openshift-service-ca\") pod \"d7b634f7-a402-49cb-8b2c-6363e6ac28b0\" (UID: \"d7b634f7-a402-49cb-8b2c-6363e6ac28b0\") " Jul 19 15:33:02.460273 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:33:02.460153 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d7b634f7-a402-49cb-8b2c-6363e6ac28b0-maas-api-tls\") pod \"d7b634f7-a402-49cb-8b2c-6363e6ac28b0\" (UID: \"d7b634f7-a402-49cb-8b2c-6363e6ac28b0\") " Jul 19 15:33:02.460273 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:33:02.460195 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-q7srq\" (UniqueName: \"kubernetes.io/projected/d7b634f7-a402-49cb-8b2c-6363e6ac28b0-kube-api-access-q7srq\") pod \"d7b634f7-a402-49cb-8b2c-6363e6ac28b0\" (UID: \"d7b634f7-a402-49cb-8b2c-6363e6ac28b0\") " Jul 19 15:33:02.460507 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:33:02.460488 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d7b634f7-a402-49cb-8b2c-6363e6ac28b0-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "d7b634f7-a402-49cb-8b2c-6363e6ac28b0" (UID: "d7b634f7-a402-49cb-8b2c-6363e6ac28b0"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 19 15:33:02.462300 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:33:02.462264 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d7b634f7-a402-49cb-8b2c-6363e6ac28b0-kube-api-access-q7srq" (OuterVolumeSpecName: "kube-api-access-q7srq") pod "d7b634f7-a402-49cb-8b2c-6363e6ac28b0" (UID: "d7b634f7-a402-49cb-8b2c-6363e6ac28b0"). InnerVolumeSpecName "kube-api-access-q7srq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 19 15:33:02.462405 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:33:02.462315 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d7b634f7-a402-49cb-8b2c-6363e6ac28b0-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "d7b634f7-a402-49cb-8b2c-6363e6ac28b0" (UID: "d7b634f7-a402-49cb-8b2c-6363e6ac28b0"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 19 15:33:02.561102 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:33:02.561079 2558 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d7b634f7-a402-49cb-8b2c-6363e6ac28b0-openshift-service-ca\") on node \"ip-10-0-138-220.ec2.internal\" DevicePath \"\"" Jul 19 15:33:02.561102 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:33:02.561100 2558 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d7b634f7-a402-49cb-8b2c-6363e6ac28b0-maas-api-tls\") on node \"ip-10-0-138-220.ec2.internal\" DevicePath \"\"" Jul 19 15:33:02.561238 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:33:02.561108 2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q7srq\" (UniqueName: \"kubernetes.io/projected/d7b634f7-a402-49cb-8b2c-6363e6ac28b0-kube-api-access-q7srq\") on node \"ip-10-0-138-220.ec2.internal\" DevicePath \"\"" Jul 19 15:33:03.084966 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:33:03.084934 2558 generic.go:356] "Generic (PLEG): container finished" podID="d7b634f7-a402-49cb-8b2c-6363e6ac28b0" containerID="bc0b5028e62d45c85d0d71c5a1745a3fb85a20c29edb7a5a7b5a5c156f0d41a4" exitCode=0 Jul 19 15:33:03.085154 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:33:03.084988 2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-cc1f323c-8466db7d65-pglmv" Jul 19 15:33:03.085154 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:33:03.084989 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-cc1f323c-8466db7d65-pglmv" event={"ID":"d7b634f7-a402-49cb-8b2c-6363e6ac28b0","Type":"ContainerDied","Data":"bc0b5028e62d45c85d0d71c5a1745a3fb85a20c29edb7a5a7b5a5c156f0d41a4"} Jul 19 15:33:03.085243 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:33:03.085152 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-cc1f323c-8466db7d65-pglmv" event={"ID":"d7b634f7-a402-49cb-8b2c-6363e6ac28b0","Type":"ContainerDied","Data":"cabbc3dc6407c068e320c29157672d5bd27faf499a981dd1098739f82213e151"} Jul 19 15:33:03.085243 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:33:03.085191 2558 scope.go:117] "RemoveContainer" containerID="bc0b5028e62d45c85d0d71c5a1745a3fb85a20c29edb7a5a7b5a5c156f0d41a4" Jul 19 15:33:03.094426 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:33:03.094410 2558 scope.go:117] "RemoveContainer" containerID="bc0b5028e62d45c85d0d71c5a1745a3fb85a20c29edb7a5a7b5a5c156f0d41a4" Jul 19 15:33:03.094660 ip-10-0-138-220 kubenswrapper[2558]: E0719 15:33:03.094641 2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bc0b5028e62d45c85d0d71c5a1745a3fb85a20c29edb7a5a7b5a5c156f0d41a4\": container with ID starting with bc0b5028e62d45c85d0d71c5a1745a3fb85a20c29edb7a5a7b5a5c156f0d41a4 not found: ID does not exist" containerID="bc0b5028e62d45c85d0d71c5a1745a3fb85a20c29edb7a5a7b5a5c156f0d41a4" Jul 19 15:33:03.094925 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:33:03.094668 2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bc0b5028e62d45c85d0d71c5a1745a3fb85a20c29edb7a5a7b5a5c156f0d41a4"} err="failed to get container status \"bc0b5028e62d45c85d0d71c5a1745a3fb85a20c29edb7a5a7b5a5c156f0d41a4\": rpc error: code = NotFound desc = could not find container \"bc0b5028e62d45c85d0d71c5a1745a3fb85a20c29edb7a5a7b5a5c156f0d41a4\": container with ID starting with bc0b5028e62d45c85d0d71c5a1745a3fb85a20c29edb7a5a7b5a5c156f0d41a4 not found: ID does not exist" Jul 19 15:33:03.110327 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:33:03.110306 2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-cc1f323c-8466db7d65-pglmv"] Jul 19 15:33:03.113703 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:33:03.113683 2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-cc1f323c-8466db7d65-pglmv"] Jul 19 15:33:03.565045 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:33:03.564998 2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d7b634f7-a402-49cb-8b2c-6363e6ac28b0" path="/var/lib/kubelet/pods/d7b634f7-a402-49cb-8b2c-6363e6ac28b0/volumes" Jul 19 15:33:11.194170 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:33:11.194132 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-d4lll"] Jul 19 15:33:39.721794 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:33:39.721708 2558 scope.go:117] "RemoveContainer" containerID="ce5644aa1e75e49edc3bc2c423ee731231c8cb65ec734ec51c8e1c733573046a" Jul 19 15:33:52.960961 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:33:52.960926 2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-cc1f323c-openshift-default-55d68dcf67-p748l"] Jul 19 15:33:52.961572 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:33:52.961216 2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-cc1f323c-openshift-default-55d68dcf67-p748l" podUID="d1d23079-96b6-4021-9dbc-4f3795ed414e" containerName="istio-proxy" containerID="cri-o://87c129b90ea7917f13086f7f3a890c93b08e3909edbba4e809c4e31dc260af00" gracePeriod=30 Jul 19 15:33:56.993481 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:33:56.993446 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-d4lll"] Jul 19 15:33:58.203189 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:33:58.203164 2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-cc1f323c-openshift-default-55d68dcf67-p748l" Jul 19 15:33:58.297228 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:33:58.297156 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d1d23079-96b6-4021-9dbc-4f3795ed414e-istio-podinfo\") pod \"d1d23079-96b6-4021-9dbc-4f3795ed414e\" (UID: \"d1d23079-96b6-4021-9dbc-4f3795ed414e\") " Jul 19 15:33:58.297228 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:33:58.297190 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d1d23079-96b6-4021-9dbc-4f3795ed414e-istio-data\") pod \"d1d23079-96b6-4021-9dbc-4f3795ed414e\" (UID: \"d1d23079-96b6-4021-9dbc-4f3795ed414e\") " Jul 19 15:33:58.297228 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:33:58.297213 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d1d23079-96b6-4021-9dbc-4f3795ed414e-istio-envoy\") pod \"d1d23079-96b6-4021-9dbc-4f3795ed414e\" (UID: \"d1d23079-96b6-4021-9dbc-4f3795ed414e\") " Jul 19 15:33:58.297453 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:33:58.297239 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1d23079-96b6-4021-9dbc-4f3795ed414e-workload-socket\") pod \"d1d23079-96b6-4021-9dbc-4f3795ed414e\" (UID: \"d1d23079-96b6-4021-9dbc-4f3795ed414e\") " Jul 19 15:33:58.297453 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:33:58.297270 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vl7mc\" (UniqueName: \"kubernetes.io/projected/d1d23079-96b6-4021-9dbc-4f3795ed414e-kube-api-access-vl7mc\") pod \"d1d23079-96b6-4021-9dbc-4f3795ed414e\" (UID: \"d1d23079-96b6-4021-9dbc-4f3795ed414e\") " Jul 19 15:33:58.297453 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:33:58.297303 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d1d23079-96b6-4021-9dbc-4f3795ed414e-istio-token\") pod \"d1d23079-96b6-4021-9dbc-4f3795ed414e\" (UID: \"d1d23079-96b6-4021-9dbc-4f3795ed414e\") " Jul 19 15:33:58.297453 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:33:58.297351 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d1d23079-96b6-4021-9dbc-4f3795ed414e-workload-certs\") pod \"d1d23079-96b6-4021-9dbc-4f3795ed414e\" (UID: \"d1d23079-96b6-4021-9dbc-4f3795ed414e\") " Jul 19 15:33:58.297669 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:33:58.297520 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1d23079-96b6-4021-9dbc-4f3795ed414e-credential-socket\") pod \"d1d23079-96b6-4021-9dbc-4f3795ed414e\" (UID: \"d1d23079-96b6-4021-9dbc-4f3795ed414e\") " Jul 19 15:33:58.297669 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:33:58.297565 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d1d23079-96b6-4021-9dbc-4f3795ed414e-istiod-ca-cert\") pod \"d1d23079-96b6-4021-9dbc-4f3795ed414e\" (UID: \"d1d23079-96b6-4021-9dbc-4f3795ed414e\") " Jul 19 15:33:58.297669 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:33:58.297634 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d1d23079-96b6-4021-9dbc-4f3795ed414e-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "d1d23079-96b6-4021-9dbc-4f3795ed414e" (UID: "d1d23079-96b6-4021-9dbc-4f3795ed414e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 19 15:33:58.297669 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:33:58.297648 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d1d23079-96b6-4021-9dbc-4f3795ed414e-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "d1d23079-96b6-4021-9dbc-4f3795ed414e" (UID: "d1d23079-96b6-4021-9dbc-4f3795ed414e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 19 15:33:58.297901 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:33:58.297824 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d1d23079-96b6-4021-9dbc-4f3795ed414e-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "d1d23079-96b6-4021-9dbc-4f3795ed414e" (UID: "d1d23079-96b6-4021-9dbc-4f3795ed414e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 19 15:33:58.297901 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:33:58.297837 2558 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1d23079-96b6-4021-9dbc-4f3795ed414e-workload-socket\") on node \"ip-10-0-138-220.ec2.internal\" DevicePath \"\"" Jul 19 15:33:58.298058 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:33:58.297967 2558 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d1d23079-96b6-4021-9dbc-4f3795ed414e-workload-certs\") on node \"ip-10-0-138-220.ec2.internal\" DevicePath \"\"" Jul 19 15:33:58.298293 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:33:58.298036 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d1d23079-96b6-4021-9dbc-4f3795ed414e-istio-data" (OuterVolumeSpecName: "istio-data") pod "d1d23079-96b6-4021-9dbc-4f3795ed414e" (UID: "d1d23079-96b6-4021-9dbc-4f3795ed414e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 19 15:33:58.298293 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:33:58.298073 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d1d23079-96b6-4021-9dbc-4f3795ed414e-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "d1d23079-96b6-4021-9dbc-4f3795ed414e" (UID: "d1d23079-96b6-4021-9dbc-4f3795ed414e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 19 15:33:58.299514 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:33:58.299481 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d1d23079-96b6-4021-9dbc-4f3795ed414e-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "d1d23079-96b6-4021-9dbc-4f3795ed414e" (UID: "d1d23079-96b6-4021-9dbc-4f3795ed414e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 19 15:33:58.299672 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:33:58.299650 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d1d23079-96b6-4021-9dbc-4f3795ed414e-kube-api-access-vl7mc" (OuterVolumeSpecName: "kube-api-access-vl7mc") pod "d1d23079-96b6-4021-9dbc-4f3795ed414e" (UID: "d1d23079-96b6-4021-9dbc-4f3795ed414e"). InnerVolumeSpecName "kube-api-access-vl7mc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 19 15:33:58.299770 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:33:58.299744 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d1d23079-96b6-4021-9dbc-4f3795ed414e-istio-token" (OuterVolumeSpecName: "istio-token") pod "d1d23079-96b6-4021-9dbc-4f3795ed414e" (UID: "d1d23079-96b6-4021-9dbc-4f3795ed414e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 19 15:33:58.299988 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:33:58.299968 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d1d23079-96b6-4021-9dbc-4f3795ed414e-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "d1d23079-96b6-4021-9dbc-4f3795ed414e" (UID: "d1d23079-96b6-4021-9dbc-4f3795ed414e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 19 15:33:58.321658 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:33:58.321627 2558 generic.go:356] "Generic (PLEG): container finished" podID="d1d23079-96b6-4021-9dbc-4f3795ed414e" containerID="87c129b90ea7917f13086f7f3a890c93b08e3909edbba4e809c4e31dc260af00" exitCode=0 Jul 19 15:33:58.321772 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:33:58.321662 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-cc1f323c-openshift-default-55d68dcf67-p748l" event={"ID":"d1d23079-96b6-4021-9dbc-4f3795ed414e","Type":"ContainerDied","Data":"87c129b90ea7917f13086f7f3a890c93b08e3909edbba4e809c4e31dc260af00"} Jul 19 15:33:58.321772 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:33:58.321688 2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-cc1f323c-openshift-default-55d68dcf67-p748l" Jul 19 15:33:58.321772 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:33:58.321704 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-cc1f323c-openshift-default-55d68dcf67-p748l" event={"ID":"d1d23079-96b6-4021-9dbc-4f3795ed414e","Type":"ContainerDied","Data":"fb3d4280a33b0054ffedcfd79b80eb4287612edd256ea335c779ba951c39312b"} Jul 19 15:33:58.321772 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:33:58.321727 2558 scope.go:117] "RemoveContainer" containerID="87c129b90ea7917f13086f7f3a890c93b08e3909edbba4e809c4e31dc260af00" Jul 19 15:33:58.330514 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:33:58.330491 2558 scope.go:117] "RemoveContainer" containerID="6351f38ef2840bdd15214ff73ea4d82554291a59a232383b1e06260ea662c8de" Jul 19 15:33:58.338223 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:33:58.338207 2558 scope.go:117] "RemoveContainer" containerID="87c129b90ea7917f13086f7f3a890c93b08e3909edbba4e809c4e31dc260af00" Jul 19 15:33:58.338485 ip-10-0-138-220 kubenswrapper[2558]: E0719 15:33:58.338467 2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"87c129b90ea7917f13086f7f3a890c93b08e3909edbba4e809c4e31dc260af00\": container with ID starting with 87c129b90ea7917f13086f7f3a890c93b08e3909edbba4e809c4e31dc260af00 not found: ID does not exist" containerID="87c129b90ea7917f13086f7f3a890c93b08e3909edbba4e809c4e31dc260af00" Jul 19 15:33:58.338525 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:33:58.338494 2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"87c129b90ea7917f13086f7f3a890c93b08e3909edbba4e809c4e31dc260af00"} err="failed to get container status \"87c129b90ea7917f13086f7f3a890c93b08e3909edbba4e809c4e31dc260af00\": rpc error: code = NotFound desc = could not find container \"87c129b90ea7917f13086f7f3a890c93b08e3909edbba4e809c4e31dc260af00\": container with ID starting with 87c129b90ea7917f13086f7f3a890c93b08e3909edbba4e809c4e31dc260af00 not found: ID does not exist" Jul 19 15:33:58.338525 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:33:58.338512 2558 scope.go:117] "RemoveContainer" containerID="6351f38ef2840bdd15214ff73ea4d82554291a59a232383b1e06260ea662c8de" Jul 19 15:33:58.338740 ip-10-0-138-220 kubenswrapper[2558]: E0719 15:33:58.338726 2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6351f38ef2840bdd15214ff73ea4d82554291a59a232383b1e06260ea662c8de\": container with ID starting with 6351f38ef2840bdd15214ff73ea4d82554291a59a232383b1e06260ea662c8de not found: ID does not exist" containerID="6351f38ef2840bdd15214ff73ea4d82554291a59a232383b1e06260ea662c8de" Jul 19 15:33:58.338779 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:33:58.338745 2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6351f38ef2840bdd15214ff73ea4d82554291a59a232383b1e06260ea662c8de"} err="failed to get container status \"6351f38ef2840bdd15214ff73ea4d82554291a59a232383b1e06260ea662c8de\": rpc error: code = NotFound desc = could not find container \"6351f38ef2840bdd15214ff73ea4d82554291a59a232383b1e06260ea662c8de\": container with ID starting with 6351f38ef2840bdd15214ff73ea4d82554291a59a232383b1e06260ea662c8de not found: ID does not exist" Jul 19 15:33:58.342257 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:33:58.342236 2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-cc1f323c-openshift-default-55d68dcf67-p748l"] Jul 19 15:33:58.348861 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:33:58.348840 2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-cc1f323c-openshift-default-55d68dcf67-p748l"] Jul 19 15:33:58.398640 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:33:58.398607 2558 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1d23079-96b6-4021-9dbc-4f3795ed414e-credential-socket\") on node \"ip-10-0-138-220.ec2.internal\" DevicePath \"\"" Jul 19 15:33:58.398640 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:33:58.398630 2558 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d1d23079-96b6-4021-9dbc-4f3795ed414e-istiod-ca-cert\") on node \"ip-10-0-138-220.ec2.internal\" DevicePath \"\"" Jul 19 15:33:58.398775 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:33:58.398650 2558 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d1d23079-96b6-4021-9dbc-4f3795ed414e-istio-podinfo\") on node \"ip-10-0-138-220.ec2.internal\" DevicePath \"\"" Jul 19 15:33:58.398775 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:33:58.398660 2558 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d1d23079-96b6-4021-9dbc-4f3795ed414e-istio-data\") on node \"ip-10-0-138-220.ec2.internal\" DevicePath \"\"" Jul 19 15:33:58.398775 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:33:58.398670 2558 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d1d23079-96b6-4021-9dbc-4f3795ed414e-istio-envoy\") on node \"ip-10-0-138-220.ec2.internal\" DevicePath \"\"" Jul 19 15:33:58.398775 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:33:58.398677 2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vl7mc\" (UniqueName: \"kubernetes.io/projected/d1d23079-96b6-4021-9dbc-4f3795ed414e-kube-api-access-vl7mc\") on node \"ip-10-0-138-220.ec2.internal\" DevicePath \"\"" Jul 19 15:33:58.398775 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:33:58.398687 2558 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d1d23079-96b6-4021-9dbc-4f3795ed414e-istio-token\") on node \"ip-10-0-138-220.ec2.internal\" DevicePath \"\"" Jul 19 15:33:59.564064 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:33:59.564004 2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d1d23079-96b6-4021-9dbc-4f3795ed414e" path="/var/lib/kubelet/pods/d1d23079-96b6-4021-9dbc-4f3795ed414e/volumes" Jul 19 15:34:09.505777 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:34:09.505743 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-d4lll"] Jul 19 15:34:39.742678 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:34:39.742646 2558 scope.go:117] "RemoveContainer" containerID="036839a8bec543a75b8036795264ceb1b8131f5c48b14257642f551d3ab12017" Jul 19 15:34:39.750397 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:34:39.750377 2558 scope.go:117] "RemoveContainer" containerID="ab3f88024c01f778a6108901b485ab561fe9b43d8dd4b8e40193e2b6553dee83" Jul 19 15:34:47.903909 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:34:47.903874 2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-1dc017-openshift-default-7489c84645-xzls8"] Jul 19 15:34:47.904385 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:34:47.904295 2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d7b634f7-a402-49cb-8b2c-6363e6ac28b0" containerName="maas-api" Jul 19 15:34:47.904385 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:34:47.904314 2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="d7b634f7-a402-49cb-8b2c-6363e6ac28b0" containerName="maas-api" Jul 19 15:34:47.904385 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:34:47.904329 2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d1d23079-96b6-4021-9dbc-4f3795ed414e" containerName="istio-proxy" Jul 19 15:34:47.904385 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:34:47.904337 2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="d1d23079-96b6-4021-9dbc-4f3795ed414e" containerName="istio-proxy" Jul 19 15:34:47.904597 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:34:47.904421 2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="d1d23079-96b6-4021-9dbc-4f3795ed414e" containerName="istio-proxy" Jul 19 15:34:47.904597 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:34:47.904432 2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="d1d23079-96b6-4021-9dbc-4f3795ed414e" containerName="istio-proxy" Jul 19 15:34:47.904597 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:34:47.904443 2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="d7b634f7-a402-49cb-8b2c-6363e6ac28b0" containerName="maas-api" Jul 19 15:34:47.907327 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:34:47.907306 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-1dc017-openshift-default-7489c84645-xzls8" Jul 19 15:34:47.909639 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:34:47.909616 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-1dc017-openshift-default-dockercfg-cjtjv\"" Jul 19 15:34:47.918249 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:34:47.918227 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-1dc017-openshift-default-7489c84645-xzls8"] Jul 19 15:34:47.947174 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:34:47.947151 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7cd49b06-890b-43ff-9d71-ae63ec00e193-istiod-ca-cert\") pod \"e2e-inf-a-1dc017-openshift-default-7489c84645-xzls8\" (UID: \"7cd49b06-890b-43ff-9d71-ae63ec00e193\") " pod="openshift-ingress/e2e-inf-a-1dc017-openshift-default-7489c84645-xzls8" Jul 19 15:34:47.947272 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:34:47.947190 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7cd49b06-890b-43ff-9d71-ae63ec00e193-workload-socket\") pod \"e2e-inf-a-1dc017-openshift-default-7489c84645-xzls8\" (UID: \"7cd49b06-890b-43ff-9d71-ae63ec00e193\") " pod="openshift-ingress/e2e-inf-a-1dc017-openshift-default-7489c84645-xzls8" Jul 19 15:34:47.947313 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:34:47.947280 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hv4rd\" (UniqueName: \"kubernetes.io/projected/7cd49b06-890b-43ff-9d71-ae63ec00e193-kube-api-access-hv4rd\") pod \"e2e-inf-a-1dc017-openshift-default-7489c84645-xzls8\" (UID: \"7cd49b06-890b-43ff-9d71-ae63ec00e193\") " pod="openshift-ingress/e2e-inf-a-1dc017-openshift-default-7489c84645-xzls8" Jul 19 15:34:47.947353 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:34:47.947319 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7cd49b06-890b-43ff-9d71-ae63ec00e193-istio-data\") pod \"e2e-inf-a-1dc017-openshift-default-7489c84645-xzls8\" (UID: \"7cd49b06-890b-43ff-9d71-ae63ec00e193\") " pod="openshift-ingress/e2e-inf-a-1dc017-openshift-default-7489c84645-xzls8" Jul 19 15:34:47.947353 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:34:47.947337 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7cd49b06-890b-43ff-9d71-ae63ec00e193-istio-podinfo\") pod \"e2e-inf-a-1dc017-openshift-default-7489c84645-xzls8\" (UID: \"7cd49b06-890b-43ff-9d71-ae63ec00e193\") " pod="openshift-ingress/e2e-inf-a-1dc017-openshift-default-7489c84645-xzls8" Jul 19 15:34:47.947426 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:34:47.947384 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7cd49b06-890b-43ff-9d71-ae63ec00e193-workload-certs\") pod \"e2e-inf-a-1dc017-openshift-default-7489c84645-xzls8\" (UID: \"7cd49b06-890b-43ff-9d71-ae63ec00e193\") " pod="openshift-ingress/e2e-inf-a-1dc017-openshift-default-7489c84645-xzls8" Jul 19 15:34:47.947426 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:34:47.947413 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7cd49b06-890b-43ff-9d71-ae63ec00e193-credential-socket\") pod \"e2e-inf-a-1dc017-openshift-default-7489c84645-xzls8\" (UID: \"7cd49b06-890b-43ff-9d71-ae63ec00e193\") " pod="openshift-ingress/e2e-inf-a-1dc017-openshift-default-7489c84645-xzls8" Jul 19 15:34:47.947503 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:34:47.947447 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7cd49b06-890b-43ff-9d71-ae63ec00e193-istio-envoy\") pod \"e2e-inf-a-1dc017-openshift-default-7489c84645-xzls8\" (UID: \"7cd49b06-890b-43ff-9d71-ae63ec00e193\") " pod="openshift-ingress/e2e-inf-a-1dc017-openshift-default-7489c84645-xzls8" Jul 19 15:34:47.947503 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:34:47.947470 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7cd49b06-890b-43ff-9d71-ae63ec00e193-istio-token\") pod \"e2e-inf-a-1dc017-openshift-default-7489c84645-xzls8\" (UID: \"7cd49b06-890b-43ff-9d71-ae63ec00e193\") " pod="openshift-ingress/e2e-inf-a-1dc017-openshift-default-7489c84645-xzls8" Jul 19 15:34:48.048540 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:34:48.048506 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7cd49b06-890b-43ff-9d71-ae63ec00e193-istiod-ca-cert\") pod \"e2e-inf-a-1dc017-openshift-default-7489c84645-xzls8\" (UID: \"7cd49b06-890b-43ff-9d71-ae63ec00e193\") " pod="openshift-ingress/e2e-inf-a-1dc017-openshift-default-7489c84645-xzls8" Jul 19 15:34:48.048724 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:34:48.048546 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7cd49b06-890b-43ff-9d71-ae63ec00e193-workload-socket\") pod \"e2e-inf-a-1dc017-openshift-default-7489c84645-xzls8\" (UID: \"7cd49b06-890b-43ff-9d71-ae63ec00e193\") " pod="openshift-ingress/e2e-inf-a-1dc017-openshift-default-7489c84645-xzls8" Jul 19 15:34:48.048724 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:34:48.048577 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hv4rd\" (UniqueName: \"kubernetes.io/projected/7cd49b06-890b-43ff-9d71-ae63ec00e193-kube-api-access-hv4rd\") pod \"e2e-inf-a-1dc017-openshift-default-7489c84645-xzls8\" (UID: \"7cd49b06-890b-43ff-9d71-ae63ec00e193\") " pod="openshift-ingress/e2e-inf-a-1dc017-openshift-default-7489c84645-xzls8" Jul 19 15:34:48.048724 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:34:48.048697 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7cd49b06-890b-43ff-9d71-ae63ec00e193-istio-data\") pod \"e2e-inf-a-1dc017-openshift-default-7489c84645-xzls8\" (UID: \"7cd49b06-890b-43ff-9d71-ae63ec00e193\") " pod="openshift-ingress/e2e-inf-a-1dc017-openshift-default-7489c84645-xzls8" Jul 19 15:34:48.048902 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:34:48.048741 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7cd49b06-890b-43ff-9d71-ae63ec00e193-istio-podinfo\") pod \"e2e-inf-a-1dc017-openshift-default-7489c84645-xzls8\" (UID: \"7cd49b06-890b-43ff-9d71-ae63ec00e193\") " pod="openshift-ingress/e2e-inf-a-1dc017-openshift-default-7489c84645-xzls8" Jul 19 15:34:48.048902 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:34:48.048811 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7cd49b06-890b-43ff-9d71-ae63ec00e193-workload-certs\") pod \"e2e-inf-a-1dc017-openshift-default-7489c84645-xzls8\" (UID: \"7cd49b06-890b-43ff-9d71-ae63ec00e193\") " pod="openshift-ingress/e2e-inf-a-1dc017-openshift-default-7489c84645-xzls8" Jul 19 15:34:48.048902 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:34:48.048840 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7cd49b06-890b-43ff-9d71-ae63ec00e193-credential-socket\") pod \"e2e-inf-a-1dc017-openshift-default-7489c84645-xzls8\" (UID: \"7cd49b06-890b-43ff-9d71-ae63ec00e193\") " pod="openshift-ingress/e2e-inf-a-1dc017-openshift-default-7489c84645-xzls8" Jul 19 15:34:48.048902 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:34:48.048890 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7cd49b06-890b-43ff-9d71-ae63ec00e193-istio-envoy\") pod \"e2e-inf-a-1dc017-openshift-default-7489c84645-xzls8\" (UID: \"7cd49b06-890b-43ff-9d71-ae63ec00e193\") " pod="openshift-ingress/e2e-inf-a-1dc017-openshift-default-7489c84645-xzls8" Jul 19 15:34:48.049142 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:34:48.048927 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7cd49b06-890b-43ff-9d71-ae63ec00e193-istio-token\") pod \"e2e-inf-a-1dc017-openshift-default-7489c84645-xzls8\" (UID: \"7cd49b06-890b-43ff-9d71-ae63ec00e193\") " pod="openshift-ingress/e2e-inf-a-1dc017-openshift-default-7489c84645-xzls8" Jul 19 15:34:48.049142 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:34:48.049012 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7cd49b06-890b-43ff-9d71-ae63ec00e193-workload-socket\") pod \"e2e-inf-a-1dc017-openshift-default-7489c84645-xzls8\" (UID: \"7cd49b06-890b-43ff-9d71-ae63ec00e193\") " pod="openshift-ingress/e2e-inf-a-1dc017-openshift-default-7489c84645-xzls8" Jul 19 15:34:48.049246 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:34:48.049180 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7cd49b06-890b-43ff-9d71-ae63ec00e193-workload-certs\") pod \"e2e-inf-a-1dc017-openshift-default-7489c84645-xzls8\" (UID: \"7cd49b06-890b-43ff-9d71-ae63ec00e193\") " pod="openshift-ingress/e2e-inf-a-1dc017-openshift-default-7489c84645-xzls8" Jul 19 15:34:48.049302 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:34:48.049291 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7cd49b06-890b-43ff-9d71-ae63ec00e193-istiod-ca-cert\") pod \"e2e-inf-a-1dc017-openshift-default-7489c84645-xzls8\" (UID: \"7cd49b06-890b-43ff-9d71-ae63ec00e193\") " pod="openshift-ingress/e2e-inf-a-1dc017-openshift-default-7489c84645-xzls8" Jul 19 15:34:48.049358 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:34:48.049333 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7cd49b06-890b-43ff-9d71-ae63ec00e193-istio-data\") pod \"e2e-inf-a-1dc017-openshift-default-7489c84645-xzls8\" (UID: \"7cd49b06-890b-43ff-9d71-ae63ec00e193\") " pod="openshift-ingress/e2e-inf-a-1dc017-openshift-default-7489c84645-xzls8" Jul 19 15:34:48.049623 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:34:48.049607 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7cd49b06-890b-43ff-9d71-ae63ec00e193-credential-socket\") pod \"e2e-inf-a-1dc017-openshift-default-7489c84645-xzls8\" (UID: \"7cd49b06-890b-43ff-9d71-ae63ec00e193\") " pod="openshift-ingress/e2e-inf-a-1dc017-openshift-default-7489c84645-xzls8" Jul 19 15:34:48.051295 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:34:48.051266 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7cd49b06-890b-43ff-9d71-ae63ec00e193-istio-envoy\") pod \"e2e-inf-a-1dc017-openshift-default-7489c84645-xzls8\" (UID: \"7cd49b06-890b-43ff-9d71-ae63ec00e193\") " pod="openshift-ingress/e2e-inf-a-1dc017-openshift-default-7489c84645-xzls8" Jul 19 15:34:48.051401 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:34:48.051312 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7cd49b06-890b-43ff-9d71-ae63ec00e193-istio-podinfo\") pod \"e2e-inf-a-1dc017-openshift-default-7489c84645-xzls8\" (UID: \"7cd49b06-890b-43ff-9d71-ae63ec00e193\") " pod="openshift-ingress/e2e-inf-a-1dc017-openshift-default-7489c84645-xzls8" Jul 19 15:34:48.055817 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:34:48.055791 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7cd49b06-890b-43ff-9d71-ae63ec00e193-istio-token\") pod \"e2e-inf-a-1dc017-openshift-default-7489c84645-xzls8\" (UID: \"7cd49b06-890b-43ff-9d71-ae63ec00e193\") " pod="openshift-ingress/e2e-inf-a-1dc017-openshift-default-7489c84645-xzls8" Jul 19 15:34:48.056058 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:34:48.056040 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hv4rd\" (UniqueName: \"kubernetes.io/projected/7cd49b06-890b-43ff-9d71-ae63ec00e193-kube-api-access-hv4rd\") pod \"e2e-inf-a-1dc017-openshift-default-7489c84645-xzls8\" (UID: \"7cd49b06-890b-43ff-9d71-ae63ec00e193\") " pod="openshift-ingress/e2e-inf-a-1dc017-openshift-default-7489c84645-xzls8" Jul 19 15:34:48.217171 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:34:48.217086 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-1dc017-openshift-default-7489c84645-xzls8" Jul 19 15:34:48.348473 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:34:48.348444 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-1dc017-openshift-default-7489c84645-xzls8"] Jul 19 15:34:48.350919 ip-10-0-138-220 kubenswrapper[2558]: W0719 15:34:48.350892 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7cd49b06_890b_43ff_9d71_ae63ec00e193.slice/crio-80051032d351e80e65eb273f32d5f9b4fb647a9ca80f22a902d3112d601e2166 WatchSource:0}: Error finding container 80051032d351e80e65eb273f32d5f9b4fb647a9ca80f22a902d3112d601e2166: Status 404 returned error can't find the container with id 80051032d351e80e65eb273f32d5f9b4fb647a9ca80f22a902d3112d601e2166 Jul 19 15:34:48.353157 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:34:48.353126 2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 19 15:34:48.353232 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:34:48.353198 2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 19 15:34:48.353232 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:34:48.353225 2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 19 15:34:48.533865 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:34:48.533768 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-1dc017-openshift-default-7489c84645-xzls8" event={"ID":"7cd49b06-890b-43ff-9d71-ae63ec00e193","Type":"ContainerStarted","Data":"bf83147199f49fd5d2fc114a2d201bb54a8cf926dd304982329b8ba95733c1bb"} Jul 19 15:34:48.533865 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:34:48.533823 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-1dc017-openshift-default-7489c84645-xzls8" event={"ID":"7cd49b06-890b-43ff-9d71-ae63ec00e193","Type":"ContainerStarted","Data":"80051032d351e80e65eb273f32d5f9b4fb647a9ca80f22a902d3112d601e2166"} Jul 19 15:34:48.553495 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:34:48.553449 2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-1dc017-openshift-default-7489c84645-xzls8" podStartSLOduration=1.553431008 podStartE2EDuration="1.553431008s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-19 15:34:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-19 15:34:48.551842115 +0000 UTC m=+2229.473544834" watchObservedRunningTime="2026-07-19 15:34:48.553431008 +0000 UTC m=+2229.475133723" Jul 19 15:34:49.218058 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:34:49.218007 2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-1dc017-openshift-default-7489c84645-xzls8" Jul 19 15:34:49.223095 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:34:49.223071 2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-1dc017-openshift-default-7489c84645-xzls8" Jul 19 15:34:49.537885 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:34:49.537806 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-1dc017-openshift-default-7489c84645-xzls8" Jul 19 15:34:49.538684 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:34:49.538660 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-1dc017-openshift-default-7489c84645-xzls8" Jul 19 15:34:50.534953 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:34:50.534921 2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-1dc017-856cb58975-hbjf2"] Jul 19 15:34:50.535434 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:34:50.535232 2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d1d23079-96b6-4021-9dbc-4f3795ed414e" containerName="istio-proxy" Jul 19 15:34:50.535434 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:34:50.535243 2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="d1d23079-96b6-4021-9dbc-4f3795ed414e" containerName="istio-proxy" Jul 19 15:34:50.538427 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:34:50.538403 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1dc017-856cb58975-hbjf2" Jul 19 15:34:50.540842 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:34:50.540821 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-1dc017-serving-cert\"" Jul 19 15:34:50.541168 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:34:50.541152 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Jul 19 15:34:50.541712 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:34:50.541690 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-ljt6w\"" Jul 19 15:34:50.541954 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:34:50.541937 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 19 15:34:50.550368 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:34:50.550230 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-1dc017-856cb58975-hbjf2"] Jul 19 15:34:50.567255 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:34:50.567220 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c794cda3-3fb4-4f1d-a291-808b26541154-openshift-service-ca\") pod \"maas-api-e2e-inf-a-1dc017-856cb58975-hbjf2\" (UID: \"c794cda3-3fb4-4f1d-a291-808b26541154\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1dc017-856cb58975-hbjf2" Jul 19 15:34:50.567413 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:34:50.567379 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c794cda3-3fb4-4f1d-a291-808b26541154-maas-api-tls\") pod \"maas-api-e2e-inf-a-1dc017-856cb58975-hbjf2\" (UID: \"c794cda3-3fb4-4f1d-a291-808b26541154\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1dc017-856cb58975-hbjf2" Jul 19 15:34:50.567519 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:34:50.567498 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jbhn8\" (UniqueName: \"kubernetes.io/projected/c794cda3-3fb4-4f1d-a291-808b26541154-kube-api-access-jbhn8\") pod \"maas-api-e2e-inf-a-1dc017-856cb58975-hbjf2\" (UID: \"c794cda3-3fb4-4f1d-a291-808b26541154\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1dc017-856cb58975-hbjf2" Jul 19 15:34:50.668908 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:34:50.668874 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c794cda3-3fb4-4f1d-a291-808b26541154-maas-api-tls\") pod \"maas-api-e2e-inf-a-1dc017-856cb58975-hbjf2\" (UID: \"c794cda3-3fb4-4f1d-a291-808b26541154\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1dc017-856cb58975-hbjf2" Jul 19 15:34:50.669120 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:34:50.668925 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jbhn8\" (UniqueName: \"kubernetes.io/projected/c794cda3-3fb4-4f1d-a291-808b26541154-kube-api-access-jbhn8\") pod \"maas-api-e2e-inf-a-1dc017-856cb58975-hbjf2\" (UID: \"c794cda3-3fb4-4f1d-a291-808b26541154\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1dc017-856cb58975-hbjf2" Jul 19 15:34:50.669120 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:34:50.668992 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c794cda3-3fb4-4f1d-a291-808b26541154-openshift-service-ca\") pod \"maas-api-e2e-inf-a-1dc017-856cb58975-hbjf2\" (UID: \"c794cda3-3fb4-4f1d-a291-808b26541154\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1dc017-856cb58975-hbjf2" Jul 19 15:34:50.669120 ip-10-0-138-220 kubenswrapper[2558]: E0719 15:34:50.669079 2558 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-a-1dc017-serving-cert: secret "maas-api-e2e-inf-a-1dc017-serving-cert" not found Jul 19 15:34:50.669304 ip-10-0-138-220 kubenswrapper[2558]: E0719 15:34:50.669158 2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c794cda3-3fb4-4f1d-a291-808b26541154-maas-api-tls podName:c794cda3-3fb4-4f1d-a291-808b26541154 nodeName:}" failed. No retries permitted until 2026-07-19 15:34:51.16913416 +0000 UTC m=+2232.090836853 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/c794cda3-3fb4-4f1d-a291-808b26541154-maas-api-tls") pod "maas-api-e2e-inf-a-1dc017-856cb58975-hbjf2" (UID: "c794cda3-3fb4-4f1d-a291-808b26541154") : secret "maas-api-e2e-inf-a-1dc017-serving-cert" not found Jul 19 15:34:50.669676 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:34:50.669656 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c794cda3-3fb4-4f1d-a291-808b26541154-openshift-service-ca\") pod \"maas-api-e2e-inf-a-1dc017-856cb58975-hbjf2\" (UID: \"c794cda3-3fb4-4f1d-a291-808b26541154\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1dc017-856cb58975-hbjf2" Jul 19 15:34:50.680062 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:34:50.680006 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jbhn8\" (UniqueName: \"kubernetes.io/projected/c794cda3-3fb4-4f1d-a291-808b26541154-kube-api-access-jbhn8\") pod \"maas-api-e2e-inf-a-1dc017-856cb58975-hbjf2\" (UID: \"c794cda3-3fb4-4f1d-a291-808b26541154\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1dc017-856cb58975-hbjf2" Jul 19 15:34:51.172724 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:34:51.172684 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c794cda3-3fb4-4f1d-a291-808b26541154-maas-api-tls\") pod \"maas-api-e2e-inf-a-1dc017-856cb58975-hbjf2\" (UID: \"c794cda3-3fb4-4f1d-a291-808b26541154\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1dc017-856cb58975-hbjf2" Jul 19 15:34:51.175334 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:34:51.175303 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c794cda3-3fb4-4f1d-a291-808b26541154-maas-api-tls\") pod \"maas-api-e2e-inf-a-1dc017-856cb58975-hbjf2\" (UID: \"c794cda3-3fb4-4f1d-a291-808b26541154\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1dc017-856cb58975-hbjf2" Jul 19 15:34:51.451308 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:34:51.451218 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1dc017-856cb58975-hbjf2" Jul 19 15:34:51.586431 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:34:51.586360 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-1dc017-856cb58975-hbjf2"] Jul 19 15:34:51.589481 ip-10-0-138-220 kubenswrapper[2558]: W0719 15:34:51.589456 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc794cda3_3fb4_4f1d_a291_808b26541154.slice/crio-ea57f3fc10ca7be3284891201853ae67e4fe0f646c363737e45fcdfced8a005e WatchSource:0}: Error finding container ea57f3fc10ca7be3284891201853ae67e4fe0f646c363737e45fcdfced8a005e: Status 404 returned error can't find the container with id ea57f3fc10ca7be3284891201853ae67e4fe0f646c363737e45fcdfced8a005e Jul 19 15:34:52.554874 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:34:52.554834 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1dc017-856cb58975-hbjf2" event={"ID":"c794cda3-3fb4-4f1d-a291-808b26541154","Type":"ContainerStarted","Data":"ebac201232116920ada8383d367c19d26d63d1babb53a6c2dac0dae9647c2748"} Jul 19 15:34:52.555071 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:34:52.554887 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1dc017-856cb58975-hbjf2" event={"ID":"c794cda3-3fb4-4f1d-a291-808b26541154","Type":"ContainerStarted","Data":"ea57f3fc10ca7be3284891201853ae67e4fe0f646c363737e45fcdfced8a005e"} Jul 19 15:34:52.555071 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:34:52.554953 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1dc017-856cb58975-hbjf2" Jul 19 15:34:52.572600 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:34:52.572557 2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1dc017-856cb58975-hbjf2" podStartSLOduration=2.180633663 podStartE2EDuration="2.572541297s" totalImagesPullingTime="391.907634ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-19 15:34:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-19 15:34:52.570901406 +0000 UTC m=+2233.492604120" watchObservedRunningTime="2026-07-19 15:34:52.572541297 +0000 UTC m=+2233.494244011" Jul 19 15:34:55.556766 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:34:55.556729 2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-5e6749-openshift-default-78f76d74cf-rg9w5"] Jul 19 15:34:55.560273 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:34:55.560243 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-5e6749-openshift-default-78f76d74cf-rg9w5" Jul 19 15:34:55.562781 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:34:55.562755 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-5e6749-openshift-default-dockercfg-pp8cf\"" Jul 19 15:34:55.575170 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:34:55.575146 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-5e6749-openshift-default-78f76d74cf-rg9w5"] Jul 19 15:34:55.605351 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:34:55.605327 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4e8305f3-2721-45bb-bb52-53991a5a49e1-istio-podinfo\") pod \"e2e-inf-b-5e6749-openshift-default-78f76d74cf-rg9w5\" (UID: \"4e8305f3-2721-45bb-bb52-53991a5a49e1\") " pod="openshift-ingress/e2e-inf-b-5e6749-openshift-default-78f76d74cf-rg9w5" Jul 19 15:34:55.605492 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:34:55.605360 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x487w\" (UniqueName: \"kubernetes.io/projected/4e8305f3-2721-45bb-bb52-53991a5a49e1-kube-api-access-x487w\") pod \"e2e-inf-b-5e6749-openshift-default-78f76d74cf-rg9w5\" (UID: \"4e8305f3-2721-45bb-bb52-53991a5a49e1\") " pod="openshift-ingress/e2e-inf-b-5e6749-openshift-default-78f76d74cf-rg9w5" Jul 19 15:34:55.605492 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:34:55.605378 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4e8305f3-2721-45bb-bb52-53991a5a49e1-istiod-ca-cert\") pod \"e2e-inf-b-5e6749-openshift-default-78f76d74cf-rg9w5\" (UID: \"4e8305f3-2721-45bb-bb52-53991a5a49e1\") " pod="openshift-ingress/e2e-inf-b-5e6749-openshift-default-78f76d74cf-rg9w5" Jul 19 15:34:55.605492 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:34:55.605396 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4e8305f3-2721-45bb-bb52-53991a5a49e1-workload-certs\") pod \"e2e-inf-b-5e6749-openshift-default-78f76d74cf-rg9w5\" (UID: \"4e8305f3-2721-45bb-bb52-53991a5a49e1\") " pod="openshift-ingress/e2e-inf-b-5e6749-openshift-default-78f76d74cf-rg9w5" Jul 19 15:34:55.605492 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:34:55.605437 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4e8305f3-2721-45bb-bb52-53991a5a49e1-istio-data\") pod \"e2e-inf-b-5e6749-openshift-default-78f76d74cf-rg9w5\" (UID: \"4e8305f3-2721-45bb-bb52-53991a5a49e1\") " pod="openshift-ingress/e2e-inf-b-5e6749-openshift-default-78f76d74cf-rg9w5" Jul 19 15:34:55.605721 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:34:55.605525 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e8305f3-2721-45bb-bb52-53991a5a49e1-workload-socket\") pod \"e2e-inf-b-5e6749-openshift-default-78f76d74cf-rg9w5\" (UID: \"4e8305f3-2721-45bb-bb52-53991a5a49e1\") " pod="openshift-ingress/e2e-inf-b-5e6749-openshift-default-78f76d74cf-rg9w5" Jul 19 15:34:55.605721 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:34:55.605573 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4e8305f3-2721-45bb-bb52-53991a5a49e1-istio-token\") pod \"e2e-inf-b-5e6749-openshift-default-78f76d74cf-rg9w5\" (UID: \"4e8305f3-2721-45bb-bb52-53991a5a49e1\") " pod="openshift-ingress/e2e-inf-b-5e6749-openshift-default-78f76d74cf-rg9w5" Jul 19 15:34:55.605721 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:34:55.605604 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e8305f3-2721-45bb-bb52-53991a5a49e1-credential-socket\") pod \"e2e-inf-b-5e6749-openshift-default-78f76d74cf-rg9w5\" (UID: \"4e8305f3-2721-45bb-bb52-53991a5a49e1\") " pod="openshift-ingress/e2e-inf-b-5e6749-openshift-default-78f76d74cf-rg9w5" Jul 19 15:34:55.605721 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:34:55.605654 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4e8305f3-2721-45bb-bb52-53991a5a49e1-istio-envoy\") pod \"e2e-inf-b-5e6749-openshift-default-78f76d74cf-rg9w5\" (UID: \"4e8305f3-2721-45bb-bb52-53991a5a49e1\") " pod="openshift-ingress/e2e-inf-b-5e6749-openshift-default-78f76d74cf-rg9w5" Jul 19 15:34:55.706887 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:34:55.706848 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e8305f3-2721-45bb-bb52-53991a5a49e1-workload-socket\") pod \"e2e-inf-b-5e6749-openshift-default-78f76d74cf-rg9w5\" (UID: \"4e8305f3-2721-45bb-bb52-53991a5a49e1\") " pod="openshift-ingress/e2e-inf-b-5e6749-openshift-default-78f76d74cf-rg9w5" Jul 19 15:34:55.707090 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:34:55.706903 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4e8305f3-2721-45bb-bb52-53991a5a49e1-istio-token\") pod \"e2e-inf-b-5e6749-openshift-default-78f76d74cf-rg9w5\" (UID: \"4e8305f3-2721-45bb-bb52-53991a5a49e1\") " pod="openshift-ingress/e2e-inf-b-5e6749-openshift-default-78f76d74cf-rg9w5" Jul 19 15:34:55.707090 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:34:55.706935 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e8305f3-2721-45bb-bb52-53991a5a49e1-credential-socket\") pod \"e2e-inf-b-5e6749-openshift-default-78f76d74cf-rg9w5\" (UID: \"4e8305f3-2721-45bb-bb52-53991a5a49e1\") " pod="openshift-ingress/e2e-inf-b-5e6749-openshift-default-78f76d74cf-rg9w5" Jul 19 15:34:55.707090 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:34:55.706968 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4e8305f3-2721-45bb-bb52-53991a5a49e1-istio-envoy\") pod \"e2e-inf-b-5e6749-openshift-default-78f76d74cf-rg9w5\" (UID: \"4e8305f3-2721-45bb-bb52-53991a5a49e1\") " pod="openshift-ingress/e2e-inf-b-5e6749-openshift-default-78f76d74cf-rg9w5" Jul 19 15:34:55.707272 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:34:55.707135 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4e8305f3-2721-45bb-bb52-53991a5a49e1-istio-podinfo\") pod \"e2e-inf-b-5e6749-openshift-default-78f76d74cf-rg9w5\" (UID: \"4e8305f3-2721-45bb-bb52-53991a5a49e1\") " pod="openshift-ingress/e2e-inf-b-5e6749-openshift-default-78f76d74cf-rg9w5" Jul 19 15:34:55.707272 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:34:55.707189 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x487w\" (UniqueName: \"kubernetes.io/projected/4e8305f3-2721-45bb-bb52-53991a5a49e1-kube-api-access-x487w\") pod \"e2e-inf-b-5e6749-openshift-default-78f76d74cf-rg9w5\" (UID: \"4e8305f3-2721-45bb-bb52-53991a5a49e1\") " pod="openshift-ingress/e2e-inf-b-5e6749-openshift-default-78f76d74cf-rg9w5" Jul 19 15:34:55.707272 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:34:55.707222 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4e8305f3-2721-45bb-bb52-53991a5a49e1-istiod-ca-cert\") pod \"e2e-inf-b-5e6749-openshift-default-78f76d74cf-rg9w5\" (UID: \"4e8305f3-2721-45bb-bb52-53991a5a49e1\") " pod="openshift-ingress/e2e-inf-b-5e6749-openshift-default-78f76d74cf-rg9w5" Jul 19 15:34:55.707272 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:34:55.707252 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4e8305f3-2721-45bb-bb52-53991a5a49e1-workload-certs\") pod \"e2e-inf-b-5e6749-openshift-default-78f76d74cf-rg9w5\" (UID: \"4e8305f3-2721-45bb-bb52-53991a5a49e1\") " pod="openshift-ingress/e2e-inf-b-5e6749-openshift-default-78f76d74cf-rg9w5" Jul 19 15:34:55.707452 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:34:55.707303 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e8305f3-2721-45bb-bb52-53991a5a49e1-workload-socket\") pod \"e2e-inf-b-5e6749-openshift-default-78f76d74cf-rg9w5\" (UID: \"4e8305f3-2721-45bb-bb52-53991a5a49e1\") " pod="openshift-ingress/e2e-inf-b-5e6749-openshift-default-78f76d74cf-rg9w5" Jul 19 15:34:55.708079 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:34:55.707979 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e8305f3-2721-45bb-bb52-53991a5a49e1-credential-socket\") pod \"e2e-inf-b-5e6749-openshift-default-78f76d74cf-rg9w5\" (UID: \"4e8305f3-2721-45bb-bb52-53991a5a49e1\") " pod="openshift-ingress/e2e-inf-b-5e6749-openshift-default-78f76d74cf-rg9w5" Jul 19 15:34:55.709589 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:34:55.708206 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4e8305f3-2721-45bb-bb52-53991a5a49e1-workload-certs\") pod \"e2e-inf-b-5e6749-openshift-default-78f76d74cf-rg9w5\" (UID: \"4e8305f3-2721-45bb-bb52-53991a5a49e1\") " pod="openshift-ingress/e2e-inf-b-5e6749-openshift-default-78f76d74cf-rg9w5" Jul 19 15:34:55.709589 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:34:55.708266 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4e8305f3-2721-45bb-bb52-53991a5a49e1-istio-data\") pod \"e2e-inf-b-5e6749-openshift-default-78f76d74cf-rg9w5\" (UID: \"4e8305f3-2721-45bb-bb52-53991a5a49e1\") " pod="openshift-ingress/e2e-inf-b-5e6749-openshift-default-78f76d74cf-rg9w5" Jul 19 15:34:55.709589 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:34:55.708508 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4e8305f3-2721-45bb-bb52-53991a5a49e1-istiod-ca-cert\") pod \"e2e-inf-b-5e6749-openshift-default-78f76d74cf-rg9w5\" (UID: \"4e8305f3-2721-45bb-bb52-53991a5a49e1\") " pod="openshift-ingress/e2e-inf-b-5e6749-openshift-default-78f76d74cf-rg9w5" Jul 19 15:34:55.709589 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:34:55.708785 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4e8305f3-2721-45bb-bb52-53991a5a49e1-istio-data\") pod \"e2e-inf-b-5e6749-openshift-default-78f76d74cf-rg9w5\" (UID: \"4e8305f3-2721-45bb-bb52-53991a5a49e1\") " pod="openshift-ingress/e2e-inf-b-5e6749-openshift-default-78f76d74cf-rg9w5" Jul 19 15:34:55.709867 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:34:55.709746 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4e8305f3-2721-45bb-bb52-53991a5a49e1-istio-envoy\") pod \"e2e-inf-b-5e6749-openshift-default-78f76d74cf-rg9w5\" (UID: \"4e8305f3-2721-45bb-bb52-53991a5a49e1\") " pod="openshift-ingress/e2e-inf-b-5e6749-openshift-default-78f76d74cf-rg9w5" Jul 19 15:34:55.713886 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:34:55.711334 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4e8305f3-2721-45bb-bb52-53991a5a49e1-istio-podinfo\") pod \"e2e-inf-b-5e6749-openshift-default-78f76d74cf-rg9w5\" (UID: \"4e8305f3-2721-45bb-bb52-53991a5a49e1\") " pod="openshift-ingress/e2e-inf-b-5e6749-openshift-default-78f76d74cf-rg9w5" Jul 19 15:34:55.715465 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:34:55.715436 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4e8305f3-2721-45bb-bb52-53991a5a49e1-istio-token\") pod \"e2e-inf-b-5e6749-openshift-default-78f76d74cf-rg9w5\" (UID: \"4e8305f3-2721-45bb-bb52-53991a5a49e1\") " pod="openshift-ingress/e2e-inf-b-5e6749-openshift-default-78f76d74cf-rg9w5" Jul 19 15:34:55.717265 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:34:55.717242 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x487w\" (UniqueName: \"kubernetes.io/projected/4e8305f3-2721-45bb-bb52-53991a5a49e1-kube-api-access-x487w\") pod \"e2e-inf-b-5e6749-openshift-default-78f76d74cf-rg9w5\" (UID: \"4e8305f3-2721-45bb-bb52-53991a5a49e1\") " pod="openshift-ingress/e2e-inf-b-5e6749-openshift-default-78f76d74cf-rg9w5" Jul 19 15:34:55.872059 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:34:55.871967 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-5e6749-openshift-default-78f76d74cf-rg9w5" Jul 19 15:34:56.004085 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:34:56.004052 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-5e6749-openshift-default-78f76d74cf-rg9w5"] Jul 19 15:34:56.007881 ip-10-0-138-220 kubenswrapper[2558]: W0719 15:34:56.007837 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4e8305f3_2721_45bb_bb52_53991a5a49e1.slice/crio-cb5fe98eccc9e7a511f0995bf27b8742ff9413a6dfe07cb8f731c75701bb348e WatchSource:0}: Error finding container cb5fe98eccc9e7a511f0995bf27b8742ff9413a6dfe07cb8f731c75701bb348e: Status 404 returned error can't find the container with id cb5fe98eccc9e7a511f0995bf27b8742ff9413a6dfe07cb8f731c75701bb348e Jul 19 15:34:56.010295 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:34:56.010260 2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 19 15:34:56.010407 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:34:56.010346 2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 19 15:34:56.010407 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:34:56.010395 2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 19 15:34:56.493001 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:34:56.492967 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-d4lll"] Jul 19 15:34:56.573343 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:34:56.573305 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-5e6749-openshift-default-78f76d74cf-rg9w5" event={"ID":"4e8305f3-2721-45bb-bb52-53991a5a49e1","Type":"ContainerStarted","Data":"c305bf835b3c53663d8bfb7b93fefc059a3f7b30186ddaaee947401089944dc6"} Jul 19 15:34:56.573693 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:34:56.573372 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-5e6749-openshift-default-78f76d74cf-rg9w5" event={"ID":"4e8305f3-2721-45bb-bb52-53991a5a49e1","Type":"ContainerStarted","Data":"cb5fe98eccc9e7a511f0995bf27b8742ff9413a6dfe07cb8f731c75701bb348e"} Jul 19 15:34:56.591851 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:34:56.591802 2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-5e6749-openshift-default-78f76d74cf-rg9w5" podStartSLOduration=1.591786728 podStartE2EDuration="1.591786728s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-19 15:34:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-19 15:34:56.590431706 +0000 UTC m=+2237.512134419" watchObservedRunningTime="2026-07-19 15:34:56.591786728 +0000 UTC m=+2237.513489433" Jul 19 15:34:56.873055 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:34:56.872944 2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-5e6749-openshift-default-78f76d74cf-rg9w5" Jul 19 15:34:57.877794 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:34:57.877763 2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-5e6749-openshift-default-78f76d74cf-rg9w5" Jul 19 15:34:58.569361 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:34:58.569321 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1dc017-856cb58975-hbjf2" Jul 19 15:34:58.583991 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:34:58.583963 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-5e6749-openshift-default-78f76d74cf-rg9w5" Jul 19 15:34:58.585087 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:34:58.585058 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-5e6749-openshift-default-78f76d74cf-rg9w5" Jul 19 15:34:58.816729 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:34:58.816695 2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-5e6749-849f76fcc6-hbj58"] Jul 19 15:34:58.820340 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:34:58.820284 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5e6749-849f76fcc6-hbj58" Jul 19 15:34:58.822507 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:34:58.822487 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-5e6749-serving-cert\"" Jul 19 15:34:58.832538 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:34:58.832518 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-5e6749-849f76fcc6-hbj58"] Jul 19 15:34:58.939585 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:34:58.939546 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/39c20738-ddf0-47f5-a7a1-25d37f145fb1-openshift-service-ca\") pod \"maas-api-e2e-inf-b-5e6749-849f76fcc6-hbj58\" (UID: \"39c20738-ddf0-47f5-a7a1-25d37f145fb1\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5e6749-849f76fcc6-hbj58" Jul 19 15:34:58.940070 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:34:58.939681 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/39c20738-ddf0-47f5-a7a1-25d37f145fb1-maas-api-tls\") pod \"maas-api-e2e-inf-b-5e6749-849f76fcc6-hbj58\" (UID: \"39c20738-ddf0-47f5-a7a1-25d37f145fb1\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5e6749-849f76fcc6-hbj58" Jul 19 15:34:58.940070 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:34:58.939736 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-km5bn\" (UniqueName: \"kubernetes.io/projected/39c20738-ddf0-47f5-a7a1-25d37f145fb1-kube-api-access-km5bn\") pod \"maas-api-e2e-inf-b-5e6749-849f76fcc6-hbj58\" (UID: \"39c20738-ddf0-47f5-a7a1-25d37f145fb1\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5e6749-849f76fcc6-hbj58" Jul 19 15:34:59.040984 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:34:59.040951 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/39c20738-ddf0-47f5-a7a1-25d37f145fb1-maas-api-tls\") pod \"maas-api-e2e-inf-b-5e6749-849f76fcc6-hbj58\" (UID: \"39c20738-ddf0-47f5-a7a1-25d37f145fb1\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5e6749-849f76fcc6-hbj58" Jul 19 15:34:59.041174 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:34:59.040994 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-km5bn\" (UniqueName: \"kubernetes.io/projected/39c20738-ddf0-47f5-a7a1-25d37f145fb1-kube-api-access-km5bn\") pod \"maas-api-e2e-inf-b-5e6749-849f76fcc6-hbj58\" (UID: \"39c20738-ddf0-47f5-a7a1-25d37f145fb1\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5e6749-849f76fcc6-hbj58" Jul 19 15:34:59.041174 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:34:59.041092 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/39c20738-ddf0-47f5-a7a1-25d37f145fb1-openshift-service-ca\") pod \"maas-api-e2e-inf-b-5e6749-849f76fcc6-hbj58\" (UID: \"39c20738-ddf0-47f5-a7a1-25d37f145fb1\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5e6749-849f76fcc6-hbj58" Jul 19 15:34:59.041747 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:34:59.041724 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/39c20738-ddf0-47f5-a7a1-25d37f145fb1-openshift-service-ca\") pod \"maas-api-e2e-inf-b-5e6749-849f76fcc6-hbj58\" (UID: \"39c20738-ddf0-47f5-a7a1-25d37f145fb1\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5e6749-849f76fcc6-hbj58" Jul 19 15:34:59.043569 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:34:59.043546 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/39c20738-ddf0-47f5-a7a1-25d37f145fb1-maas-api-tls\") pod \"maas-api-e2e-inf-b-5e6749-849f76fcc6-hbj58\" (UID: \"39c20738-ddf0-47f5-a7a1-25d37f145fb1\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5e6749-849f76fcc6-hbj58" Jul 19 15:34:59.054398 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:34:59.054346 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-km5bn\" (UniqueName: \"kubernetes.io/projected/39c20738-ddf0-47f5-a7a1-25d37f145fb1-kube-api-access-km5bn\") pod \"maas-api-e2e-inf-b-5e6749-849f76fcc6-hbj58\" (UID: \"39c20738-ddf0-47f5-a7a1-25d37f145fb1\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5e6749-849f76fcc6-hbj58" Jul 19 15:34:59.132206 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:34:59.132138 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5e6749-849f76fcc6-hbj58" Jul 19 15:34:59.308481 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:34:59.308449 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-5e6749-849f76fcc6-hbj58"] Jul 19 15:34:59.312016 ip-10-0-138-220 kubenswrapper[2558]: W0719 15:34:59.311990 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod39c20738_ddf0_47f5_a7a1_25d37f145fb1.slice/crio-e76da186944464fe6335d09ca0e3b873c7638d7d988e4f7f547b046354cc20f1 WatchSource:0}: Error finding container e76da186944464fe6335d09ca0e3b873c7638d7d988e4f7f547b046354cc20f1: Status 404 returned error can't find the container with id e76da186944464fe6335d09ca0e3b873c7638d7d988e4f7f547b046354cc20f1 Jul 19 15:34:59.589110 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:34:59.589071 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5e6749-849f76fcc6-hbj58" event={"ID":"39c20738-ddf0-47f5-a7a1-25d37f145fb1","Type":"ContainerStarted","Data":"e76da186944464fe6335d09ca0e3b873c7638d7d988e4f7f547b046354cc20f1"} Jul 19 15:35:00.594457 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:35:00.594417 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5e6749-849f76fcc6-hbj58" event={"ID":"39c20738-ddf0-47f5-a7a1-25d37f145fb1","Type":"ContainerStarted","Data":"3f68fded2ef4945a7d4a6b169b16288d2be5dae6b2b3a3f1a918fa3082339525"} Jul 19 15:35:00.594964 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:35:00.594750 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5e6749-849f76fcc6-hbj58" Jul 19 15:35:00.612309 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:35:00.612262 2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5e6749-849f76fcc6-hbj58" podStartSLOduration=2.303928428 podStartE2EDuration="2.612249339s" totalImagesPullingTime="308.320911ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-19 15:34:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-19 15:35:00.610057032 +0000 UTC m=+2241.531759743" watchObservedRunningTime="2026-07-19 15:35:00.612249339 +0000 UTC m=+2241.533952043" Jul 19 15:35:04.675799 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:35:04.675377 2558 patch_prober.go:28] interesting pod/e2e-inf-a-1dc017-openshift-default-7489c84645-xzls8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.38:15021/healthz/ready\": read tcp 10.133.0.2:52142->10.133.0.38:15021: read: connection reset by peer" start-of-body= Jul 19 15:35:04.675799 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:35:04.675443 2558 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-1dc017-openshift-default-7489c84645-xzls8" podUID="7cd49b06-890b-43ff-9d71-ae63ec00e193" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.38:15021/healthz/ready\": read tcp 10.133.0.2:52142->10.133.0.38:15021: read: connection reset by peer" Jul 19 15:35:05.618705 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:35:05.618668 2558 generic.go:356] "Generic (PLEG): container finished" podID="7cd49b06-890b-43ff-9d71-ae63ec00e193" containerID="bf83147199f49fd5d2fc114a2d201bb54a8cf926dd304982329b8ba95733c1bb" exitCode=0 Jul 19 15:35:05.618878 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:35:05.618740 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-1dc017-openshift-default-7489c84645-xzls8" event={"ID":"7cd49b06-890b-43ff-9d71-ae63ec00e193","Type":"ContainerDied","Data":"bf83147199f49fd5d2fc114a2d201bb54a8cf926dd304982329b8ba95733c1bb"} Jul 19 15:35:05.619119 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:35:05.619105 2558 scope.go:117] "RemoveContainer" containerID="bf83147199f49fd5d2fc114a2d201bb54a8cf926dd304982329b8ba95733c1bb" Jul 19 15:35:05.620006 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:35:05.619984 2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 19 15:35:05.620079 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:35:05.620063 2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 19 15:35:05.620113 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:35:05.620095 2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 19 15:35:06.625510 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:35:06.625477 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-1dc017-openshift-default-7489c84645-xzls8" event={"ID":"7cd49b06-890b-43ff-9d71-ae63ec00e193","Type":"ContainerStarted","Data":"b7a55700725edcd968e87775898934ef5e08725eabde77f8f7bfc65197271104"} Jul 19 15:35:07.217831 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:35:07.217803 2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-1dc017-openshift-default-7489c84645-xzls8" Jul 19 15:35:07.218031 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:35:07.217995 2558 patch_prober.go:28] interesting pod/e2e-inf-a-1dc017-openshift-default-7489c84645-xzls8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" start-of-body= Jul 19 15:35:07.218102 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:35:07.218076 2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-1dc017-openshift-default-7489c84645-xzls8" podUID="7cd49b06-890b-43ff-9d71-ae63ec00e193" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" Jul 19 15:35:07.612014 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:35:07.611942 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5e6749-849f76fcc6-hbj58" Jul 19 15:35:08.218436 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:35:08.218398 2558 patch_prober.go:28] interesting pod/e2e-inf-a-1dc017-openshift-default-7489c84645-xzls8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" start-of-body= Jul 19 15:35:08.218851 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:35:08.218465 2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-1dc017-openshift-default-7489c84645-xzls8" podUID="7cd49b06-890b-43ff-9d71-ae63ec00e193" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" Jul 19 15:35:09.218163 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:35:09.218124 2558 patch_prober.go:28] interesting pod/e2e-inf-a-1dc017-openshift-default-7489c84645-xzls8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" start-of-body= Jul 19 15:35:09.218337 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:35:09.218188 2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-1dc017-openshift-default-7489c84645-xzls8" podUID="7cd49b06-890b-43ff-9d71-ae63ec00e193" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" Jul 19 15:35:09.393533 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:35:09.393495 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-d4lll"] Jul 19 15:35:10.217953 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:35:10.217916 2558 patch_prober.go:28] interesting pod/e2e-inf-a-1dc017-openshift-default-7489c84645-xzls8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" start-of-body= Jul 19 15:35:10.218217 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:35:10.217974 2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-1dc017-openshift-default-7489c84645-xzls8" podUID="7cd49b06-890b-43ff-9d71-ae63ec00e193" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" Jul 19 15:35:10.645675 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:35:10.645645 2558 generic.go:356] "Generic (PLEG): container finished" podID="4e8305f3-2721-45bb-bb52-53991a5a49e1" containerID="c305bf835b3c53663d8bfb7b93fefc059a3f7b30186ddaaee947401089944dc6" exitCode=0 Jul 19 15:35:10.646127 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:35:10.645717 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-5e6749-openshift-default-78f76d74cf-rg9w5" event={"ID":"4e8305f3-2721-45bb-bb52-53991a5a49e1","Type":"ContainerDied","Data":"c305bf835b3c53663d8bfb7b93fefc059a3f7b30186ddaaee947401089944dc6"} Jul 19 15:35:10.646127 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:35:10.646122 2558 scope.go:117] "RemoveContainer" containerID="c305bf835b3c53663d8bfb7b93fefc059a3f7b30186ddaaee947401089944dc6" Jul 19 15:35:10.647166 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:35:10.647140 2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 19 15:35:10.647281 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:35:10.647219 2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 19 15:35:10.647281 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:35:10.647254 2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 19 15:35:11.220900 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:35:11.220867 2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-1dc017-openshift-default-7489c84645-xzls8" Jul 19 15:35:11.221320 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:35:11.221294 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-1dc017-openshift-default-7489c84645-xzls8" Jul 19 15:35:11.222000 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:35:11.221982 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-1dc017-openshift-default-7489c84645-xzls8" Jul 19 15:35:11.654482 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:35:11.654448 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-5e6749-openshift-default-78f76d74cf-rg9w5" event={"ID":"4e8305f3-2721-45bb-bb52-53991a5a49e1","Type":"ContainerStarted","Data":"d57f9ae7a0aeef056ec2da1d0fab1a500769258d5b05746627a7a50a86af731e"} Jul 19 15:35:11.872217 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:35:11.872177 2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-5e6749-openshift-default-78f76d74cf-rg9w5" Jul 19 15:35:11.873378 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:35:11.873352 2558 patch_prober.go:28] interesting pod/e2e-inf-b-5e6749-openshift-default-78f76d74cf-rg9w5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" start-of-body= Jul 19 15:35:11.873496 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:35:11.873404 2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-5e6749-openshift-default-78f76d74cf-rg9w5" podUID="4e8305f3-2721-45bb-bb52-53991a5a49e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" Jul 19 15:35:12.872498 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:35:12.872458 2558 patch_prober.go:28] interesting pod/e2e-inf-b-5e6749-openshift-default-78f76d74cf-rg9w5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" start-of-body= Jul 19 15:35:12.872940 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:35:12.872527 2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-5e6749-openshift-default-78f76d74cf-rg9w5" podUID="4e8305f3-2721-45bb-bb52-53991a5a49e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" Jul 19 15:35:13.584620 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:35:13.584585 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-5e6749-openshift-default-78f76d74cf-rg9w5" Jul 19 15:35:13.872724 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:35:13.872634 2558 patch_prober.go:28] interesting pod/e2e-inf-b-5e6749-openshift-default-78f76d74cf-rg9w5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" start-of-body= Jul 19 15:35:13.873168 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:35:13.872714 2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-5e6749-openshift-default-78f76d74cf-rg9w5" podUID="4e8305f3-2721-45bb-bb52-53991a5a49e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" Jul 19 15:35:14.872713 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:35:14.872679 2558 patch_prober.go:28] interesting pod/e2e-inf-b-5e6749-openshift-default-78f76d74cf-rg9w5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" start-of-body= Jul 19 15:35:14.873265 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:35:14.872753 2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-5e6749-openshift-default-78f76d74cf-rg9w5" podUID="4e8305f3-2721-45bb-bb52-53991a5a49e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" Jul 19 15:35:15.876848 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:35:15.876810 2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-5e6749-openshift-default-78f76d74cf-rg9w5" Jul 19 15:35:15.877953 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:35:15.877936 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-5e6749-openshift-default-78f76d74cf-rg9w5" Jul 19 15:35:18.894960 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:35:18.894924 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-d4lll"] Jul 19 15:35:39.771916 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:35:39.771886 2558 scope.go:117] "RemoveContainer" containerID="f0e462a2ee6bf3e00269022e661974728de4aa01565bb2bdd6529440bfb7d6d8" Jul 19 15:35:40.790656 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:35:40.790550 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-d4lll"] Jul 19 15:35:59.386520 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:35:59.386480 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-d4lll"] Jul 19 15:36:07.884143 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:36:07.884107 2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-1dc017-856cb58975-hbjf2"] Jul 19 15:36:07.884919 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:36:07.884861 2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1dc017-856cb58975-hbjf2" podUID="c794cda3-3fb4-4f1d-a291-808b26541154" containerName="maas-api" containerID="cri-o://ebac201232116920ada8383d367c19d26d63d1babb53a6c2dac0dae9647c2748" gracePeriod=30 Jul 19 15:36:08.130198 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:36:08.130176 2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1dc017-856cb58975-hbjf2" Jul 19 15:36:08.218081 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:36:08.218038 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c794cda3-3fb4-4f1d-a291-808b26541154-maas-api-tls\") pod \"c794cda3-3fb4-4f1d-a291-808b26541154\" (UID: \"c794cda3-3fb4-4f1d-a291-808b26541154\") " Jul 19 15:36:08.218266 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:36:08.218158 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c794cda3-3fb4-4f1d-a291-808b26541154-openshift-service-ca\") pod \"c794cda3-3fb4-4f1d-a291-808b26541154\" (UID: \"c794cda3-3fb4-4f1d-a291-808b26541154\") " Jul 19 15:36:08.218266 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:36:08.218192 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jbhn8\" (UniqueName: \"kubernetes.io/projected/c794cda3-3fb4-4f1d-a291-808b26541154-kube-api-access-jbhn8\") pod \"c794cda3-3fb4-4f1d-a291-808b26541154\" (UID: \"c794cda3-3fb4-4f1d-a291-808b26541154\") " Jul 19 15:36:08.218507 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:36:08.218485 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c794cda3-3fb4-4f1d-a291-808b26541154-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "c794cda3-3fb4-4f1d-a291-808b26541154" (UID: "c794cda3-3fb4-4f1d-a291-808b26541154"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 19 15:36:08.220288 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:36:08.220264 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c794cda3-3fb4-4f1d-a291-808b26541154-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "c794cda3-3fb4-4f1d-a291-808b26541154" (UID: "c794cda3-3fb4-4f1d-a291-808b26541154"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 19 15:36:08.220421 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:36:08.220397 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c794cda3-3fb4-4f1d-a291-808b26541154-kube-api-access-jbhn8" (OuterVolumeSpecName: "kube-api-access-jbhn8") pod "c794cda3-3fb4-4f1d-a291-808b26541154" (UID: "c794cda3-3fb4-4f1d-a291-808b26541154"). InnerVolumeSpecName "kube-api-access-jbhn8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 19 15:36:08.319702 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:36:08.319663 2558 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c794cda3-3fb4-4f1d-a291-808b26541154-openshift-service-ca\") on node \"ip-10-0-138-220.ec2.internal\" DevicePath \"\"" Jul 19 15:36:08.319702 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:36:08.319690 2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jbhn8\" (UniqueName: \"kubernetes.io/projected/c794cda3-3fb4-4f1d-a291-808b26541154-kube-api-access-jbhn8\") on node \"ip-10-0-138-220.ec2.internal\" DevicePath \"\"" Jul 19 15:36:08.319702 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:36:08.319701 2558 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c794cda3-3fb4-4f1d-a291-808b26541154-maas-api-tls\") on node \"ip-10-0-138-220.ec2.internal\" DevicePath \"\"" Jul 19 15:36:08.918931 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:36:08.918893 2558 generic.go:356] "Generic (PLEG): container finished" podID="c794cda3-3fb4-4f1d-a291-808b26541154" containerID="ebac201232116920ada8383d367c19d26d63d1babb53a6c2dac0dae9647c2748" exitCode=0 Jul 19 15:36:08.919440 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:36:08.918936 2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1dc017-856cb58975-hbjf2" Jul 19 15:36:08.919440 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:36:08.918983 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1dc017-856cb58975-hbjf2" event={"ID":"c794cda3-3fb4-4f1d-a291-808b26541154","Type":"ContainerDied","Data":"ebac201232116920ada8383d367c19d26d63d1babb53a6c2dac0dae9647c2748"} Jul 19 15:36:08.919440 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:36:08.919063 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-1dc017-856cb58975-hbjf2" event={"ID":"c794cda3-3fb4-4f1d-a291-808b26541154","Type":"ContainerDied","Data":"ea57f3fc10ca7be3284891201853ae67e4fe0f646c363737e45fcdfced8a005e"} Jul 19 15:36:08.919440 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:36:08.919100 2558 scope.go:117] "RemoveContainer" containerID="ebac201232116920ada8383d367c19d26d63d1babb53a6c2dac0dae9647c2748" Jul 19 15:36:08.928351 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:36:08.928333 2558 scope.go:117] "RemoveContainer" containerID="ebac201232116920ada8383d367c19d26d63d1babb53a6c2dac0dae9647c2748" Jul 19 15:36:08.928603 ip-10-0-138-220 kubenswrapper[2558]: E0719 15:36:08.928584 2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ebac201232116920ada8383d367c19d26d63d1babb53a6c2dac0dae9647c2748\": container with ID starting with ebac201232116920ada8383d367c19d26d63d1babb53a6c2dac0dae9647c2748 not found: ID does not exist" containerID="ebac201232116920ada8383d367c19d26d63d1babb53a6c2dac0dae9647c2748" Jul 19 15:36:08.928657 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:36:08.928614 2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ebac201232116920ada8383d367c19d26d63d1babb53a6c2dac0dae9647c2748"} err="failed to get container status \"ebac201232116920ada8383d367c19d26d63d1babb53a6c2dac0dae9647c2748\": rpc error: code = NotFound desc = could not find container \"ebac201232116920ada8383d367c19d26d63d1babb53a6c2dac0dae9647c2748\": container with ID starting with ebac201232116920ada8383d367c19d26d63d1babb53a6c2dac0dae9647c2748 not found: ID does not exist" Jul 19 15:36:08.938773 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:36:08.938733 2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-1dc017-856cb58975-hbjf2"] Jul 19 15:36:08.940917 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:36:08.940898 2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-1dc017-856cb58975-hbjf2"] Jul 19 15:36:09.565593 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:36:09.565559 2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c794cda3-3fb4-4f1d-a291-808b26541154" path="/var/lib/kubelet/pods/c794cda3-3fb4-4f1d-a291-808b26541154/volumes" Jul 19 15:36:13.793232 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:36:13.793197 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-d4lll"] Jul 19 15:36:18.984350 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:36:18.984315 2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-1dc017-openshift-default-7489c84645-xzls8"] Jul 19 15:36:18.984763 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:36:18.984532 2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-1dc017-openshift-default-7489c84645-xzls8" podUID="7cd49b06-890b-43ff-9d71-ae63ec00e193" containerName="istio-proxy" containerID="cri-o://b7a55700725edcd968e87775898934ef5e08725eabde77f8f7bfc65197271104" gracePeriod=30 Jul 19 15:36:24.235618 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:36:24.235592 2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-1dc017-openshift-default-7489c84645-xzls8" Jul 19 15:36:24.356663 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:36:24.356577 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7cd49b06-890b-43ff-9d71-ae63ec00e193-istio-data\") pod \"7cd49b06-890b-43ff-9d71-ae63ec00e193\" (UID: \"7cd49b06-890b-43ff-9d71-ae63ec00e193\") " Jul 19 15:36:24.356663 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:36:24.356611 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7cd49b06-890b-43ff-9d71-ae63ec00e193-istiod-ca-cert\") pod \"7cd49b06-890b-43ff-9d71-ae63ec00e193\" (UID: \"7cd49b06-890b-43ff-9d71-ae63ec00e193\") " Jul 19 15:36:24.356663 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:36:24.356630 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7cd49b06-890b-43ff-9d71-ae63ec00e193-workload-socket\") pod \"7cd49b06-890b-43ff-9d71-ae63ec00e193\" (UID: \"7cd49b06-890b-43ff-9d71-ae63ec00e193\") " Jul 19 15:36:24.356663 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:36:24.356647 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hv4rd\" (UniqueName: \"kubernetes.io/projected/7cd49b06-890b-43ff-9d71-ae63ec00e193-kube-api-access-hv4rd\") pod \"7cd49b06-890b-43ff-9d71-ae63ec00e193\" (UID: \"7cd49b06-890b-43ff-9d71-ae63ec00e193\") " Jul 19 15:36:24.356663 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:36:24.356665 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7cd49b06-890b-43ff-9d71-ae63ec00e193-credential-socket\") pod \"7cd49b06-890b-43ff-9d71-ae63ec00e193\" (UID: \"7cd49b06-890b-43ff-9d71-ae63ec00e193\") " Jul 19 15:36:24.357073 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:36:24.356715 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7cd49b06-890b-43ff-9d71-ae63ec00e193-istio-envoy\") pod \"7cd49b06-890b-43ff-9d71-ae63ec00e193\" (UID: \"7cd49b06-890b-43ff-9d71-ae63ec00e193\") " Jul 19 15:36:24.357073 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:36:24.356739 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7cd49b06-890b-43ff-9d71-ae63ec00e193-workload-certs\") pod \"7cd49b06-890b-43ff-9d71-ae63ec00e193\" (UID: \"7cd49b06-890b-43ff-9d71-ae63ec00e193\") " Jul 19 15:36:24.357073 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:36:24.356797 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7cd49b06-890b-43ff-9d71-ae63ec00e193-istio-podinfo\") pod \"7cd49b06-890b-43ff-9d71-ae63ec00e193\" (UID: \"7cd49b06-890b-43ff-9d71-ae63ec00e193\") " Jul 19 15:36:24.357073 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:36:24.356832 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7cd49b06-890b-43ff-9d71-ae63ec00e193-istio-token\") pod \"7cd49b06-890b-43ff-9d71-ae63ec00e193\" (UID: \"7cd49b06-890b-43ff-9d71-ae63ec00e193\") " Jul 19 15:36:24.357073 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:36:24.356940 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7cd49b06-890b-43ff-9d71-ae63ec00e193-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "7cd49b06-890b-43ff-9d71-ae63ec00e193" (UID: "7cd49b06-890b-43ff-9d71-ae63ec00e193"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 19 15:36:24.357314 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:36:24.357089 2558 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7cd49b06-890b-43ff-9d71-ae63ec00e193-workload-socket\") on node \"ip-10-0-138-220.ec2.internal\" DevicePath \"\"" Jul 19 15:36:24.357314 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:36:24.357089 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7cd49b06-890b-43ff-9d71-ae63ec00e193-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "7cd49b06-890b-43ff-9d71-ae63ec00e193" (UID: "7cd49b06-890b-43ff-9d71-ae63ec00e193"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 19 15:36:24.357314 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:36:24.357097 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7cd49b06-890b-43ff-9d71-ae63ec00e193-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "7cd49b06-890b-43ff-9d71-ae63ec00e193" (UID: "7cd49b06-890b-43ff-9d71-ae63ec00e193"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 19 15:36:24.357314 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:36:24.357187 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7cd49b06-890b-43ff-9d71-ae63ec00e193-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "7cd49b06-890b-43ff-9d71-ae63ec00e193" (UID: "7cd49b06-890b-43ff-9d71-ae63ec00e193"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 19 15:36:24.357575 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:36:24.357531 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7cd49b06-890b-43ff-9d71-ae63ec00e193-istio-data" (OuterVolumeSpecName: "istio-data") pod "7cd49b06-890b-43ff-9d71-ae63ec00e193" (UID: "7cd49b06-890b-43ff-9d71-ae63ec00e193"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 19 15:36:24.359224 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:36:24.359183 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7cd49b06-890b-43ff-9d71-ae63ec00e193-kube-api-access-hv4rd" (OuterVolumeSpecName: "kube-api-access-hv4rd") pod "7cd49b06-890b-43ff-9d71-ae63ec00e193" (UID: "7cd49b06-890b-43ff-9d71-ae63ec00e193"). InnerVolumeSpecName "kube-api-access-hv4rd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 19 15:36:24.359338 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:36:24.359269 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7cd49b06-890b-43ff-9d71-ae63ec00e193-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "7cd49b06-890b-43ff-9d71-ae63ec00e193" (UID: "7cd49b06-890b-43ff-9d71-ae63ec00e193"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 19 15:36:24.359338 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:36:24.359283 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7cd49b06-890b-43ff-9d71-ae63ec00e193-istio-token" (OuterVolumeSpecName: "istio-token") pod "7cd49b06-890b-43ff-9d71-ae63ec00e193" (UID: "7cd49b06-890b-43ff-9d71-ae63ec00e193"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 19 15:36:24.359338 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:36:24.359303 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7cd49b06-890b-43ff-9d71-ae63ec00e193-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "7cd49b06-890b-43ff-9d71-ae63ec00e193" (UID: "7cd49b06-890b-43ff-9d71-ae63ec00e193"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 19 15:36:24.458107 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:36:24.458073 2558 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7cd49b06-890b-43ff-9d71-ae63ec00e193-istiod-ca-cert\") on node \"ip-10-0-138-220.ec2.internal\" DevicePath \"\"" Jul 19 15:36:24.458107 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:36:24.458102 2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hv4rd\" (UniqueName: \"kubernetes.io/projected/7cd49b06-890b-43ff-9d71-ae63ec00e193-kube-api-access-hv4rd\") on node \"ip-10-0-138-220.ec2.internal\" DevicePath \"\"" Jul 19 15:36:24.458308 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:36:24.458117 2558 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7cd49b06-890b-43ff-9d71-ae63ec00e193-credential-socket\") on node \"ip-10-0-138-220.ec2.internal\" DevicePath \"\"" Jul 19 15:36:24.458308 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:36:24.458131 2558 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7cd49b06-890b-43ff-9d71-ae63ec00e193-istio-envoy\") on node \"ip-10-0-138-220.ec2.internal\" DevicePath \"\"" Jul 19 15:36:24.458308 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:36:24.458144 2558 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7cd49b06-890b-43ff-9d71-ae63ec00e193-workload-certs\") on node \"ip-10-0-138-220.ec2.internal\" DevicePath \"\"" Jul 19 15:36:24.458308 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:36:24.458157 2558 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7cd49b06-890b-43ff-9d71-ae63ec00e193-istio-podinfo\") on node \"ip-10-0-138-220.ec2.internal\" DevicePath \"\"" Jul 19 15:36:24.458308 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:36:24.458168 2558 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7cd49b06-890b-43ff-9d71-ae63ec00e193-istio-token\") on node \"ip-10-0-138-220.ec2.internal\" DevicePath \"\"" Jul 19 15:36:24.458308 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:36:24.458177 2558 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7cd49b06-890b-43ff-9d71-ae63ec00e193-istio-data\") on node \"ip-10-0-138-220.ec2.internal\" DevicePath \"\"" Jul 19 15:36:24.995937 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:36:24.995899 2558 generic.go:356] "Generic (PLEG): container finished" podID="7cd49b06-890b-43ff-9d71-ae63ec00e193" containerID="b7a55700725edcd968e87775898934ef5e08725eabde77f8f7bfc65197271104" exitCode=0 Jul 19 15:36:24.996124 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:36:24.995949 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-1dc017-openshift-default-7489c84645-xzls8" event={"ID":"7cd49b06-890b-43ff-9d71-ae63ec00e193","Type":"ContainerDied","Data":"b7a55700725edcd968e87775898934ef5e08725eabde77f8f7bfc65197271104"} Jul 19 15:36:24.996124 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:36:24.995957 2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-1dc017-openshift-default-7489c84645-xzls8" Jul 19 15:36:24.996124 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:36:24.996048 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-1dc017-openshift-default-7489c84645-xzls8" event={"ID":"7cd49b06-890b-43ff-9d71-ae63ec00e193","Type":"ContainerDied","Data":"80051032d351e80e65eb273f32d5f9b4fb647a9ca80f22a902d3112d601e2166"} Jul 19 15:36:24.996124 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:36:24.996082 2558 scope.go:117] "RemoveContainer" containerID="b7a55700725edcd968e87775898934ef5e08725eabde77f8f7bfc65197271104" Jul 19 15:36:25.005170 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:36:25.005152 2558 scope.go:117] "RemoveContainer" containerID="bf83147199f49fd5d2fc114a2d201bb54a8cf926dd304982329b8ba95733c1bb" Jul 19 15:36:25.012821 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:36:25.012803 2558 scope.go:117] "RemoveContainer" containerID="b7a55700725edcd968e87775898934ef5e08725eabde77f8f7bfc65197271104" Jul 19 15:36:25.013080 ip-10-0-138-220 kubenswrapper[2558]: E0719 15:36:25.013063 2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b7a55700725edcd968e87775898934ef5e08725eabde77f8f7bfc65197271104\": container with ID starting with b7a55700725edcd968e87775898934ef5e08725eabde77f8f7bfc65197271104 not found: ID does not exist" containerID="b7a55700725edcd968e87775898934ef5e08725eabde77f8f7bfc65197271104" Jul 19 15:36:25.013140 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:36:25.013086 2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b7a55700725edcd968e87775898934ef5e08725eabde77f8f7bfc65197271104"} err="failed to get container status \"b7a55700725edcd968e87775898934ef5e08725eabde77f8f7bfc65197271104\": rpc error: code = NotFound desc = could not find container \"b7a55700725edcd968e87775898934ef5e08725eabde77f8f7bfc65197271104\": container with ID starting with b7a55700725edcd968e87775898934ef5e08725eabde77f8f7bfc65197271104 not found: ID does not exist" Jul 19 15:36:25.013140 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:36:25.013100 2558 scope.go:117] "RemoveContainer" containerID="bf83147199f49fd5d2fc114a2d201bb54a8cf926dd304982329b8ba95733c1bb" Jul 19 15:36:25.013326 ip-10-0-138-220 kubenswrapper[2558]: E0719 15:36:25.013307 2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bf83147199f49fd5d2fc114a2d201bb54a8cf926dd304982329b8ba95733c1bb\": container with ID starting with bf83147199f49fd5d2fc114a2d201bb54a8cf926dd304982329b8ba95733c1bb not found: ID does not exist" containerID="bf83147199f49fd5d2fc114a2d201bb54a8cf926dd304982329b8ba95733c1bb" Jul 19 15:36:25.013365 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:36:25.013333 2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bf83147199f49fd5d2fc114a2d201bb54a8cf926dd304982329b8ba95733c1bb"} err="failed to get container status \"bf83147199f49fd5d2fc114a2d201bb54a8cf926dd304982329b8ba95733c1bb\": rpc error: code = NotFound desc = could not find container \"bf83147199f49fd5d2fc114a2d201bb54a8cf926dd304982329b8ba95733c1bb\": container with ID starting with bf83147199f49fd5d2fc114a2d201bb54a8cf926dd304982329b8ba95733c1bb not found: ID does not exist" Jul 19 15:36:25.018816 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:36:25.018793 2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-1dc017-openshift-default-7489c84645-xzls8"] Jul 19 15:36:25.022908 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:36:25.022889 2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-1dc017-openshift-default-7489c84645-xzls8"] Jul 19 15:36:25.564391 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:36:25.564357 2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7cd49b06-890b-43ff-9d71-ae63ec00e193" path="/var/lib/kubelet/pods/7cd49b06-890b-43ff-9d71-ae63ec00e193/volumes" Jul 19 15:36:29.743583 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:36:29.743549 2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-5e6749-849f76fcc6-hbj58"] Jul 19 15:36:30.016668 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:36:30.016577 2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5e6749-849f76fcc6-hbj58" podUID="39c20738-ddf0-47f5-a7a1-25d37f145fb1" containerName="maas-api" containerID="cri-o://3f68fded2ef4945a7d4a6b169b16288d2be5dae6b2b3a3f1a918fa3082339525" gracePeriod=30 Jul 19 15:36:30.259899 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:36:30.259877 2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5e6749-849f76fcc6-hbj58" Jul 19 15:36:30.300324 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:36:30.300266 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/39c20738-ddf0-47f5-a7a1-25d37f145fb1-maas-api-tls\") pod \"39c20738-ddf0-47f5-a7a1-25d37f145fb1\" (UID: \"39c20738-ddf0-47f5-a7a1-25d37f145fb1\") " Jul 19 15:36:30.300324 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:36:30.300313 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/39c20738-ddf0-47f5-a7a1-25d37f145fb1-openshift-service-ca\") pod \"39c20738-ddf0-47f5-a7a1-25d37f145fb1\" (UID: \"39c20738-ddf0-47f5-a7a1-25d37f145fb1\") " Jul 19 15:36:30.300512 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:36:30.300352 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-km5bn\" (UniqueName: \"kubernetes.io/projected/39c20738-ddf0-47f5-a7a1-25d37f145fb1-kube-api-access-km5bn\") pod \"39c20738-ddf0-47f5-a7a1-25d37f145fb1\" (UID: \"39c20738-ddf0-47f5-a7a1-25d37f145fb1\") " Jul 19 15:36:30.300697 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:36:30.300671 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/39c20738-ddf0-47f5-a7a1-25d37f145fb1-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "39c20738-ddf0-47f5-a7a1-25d37f145fb1" (UID: "39c20738-ddf0-47f5-a7a1-25d37f145fb1"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 19 15:36:30.302552 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:36:30.302531 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/39c20738-ddf0-47f5-a7a1-25d37f145fb1-kube-api-access-km5bn" (OuterVolumeSpecName: "kube-api-access-km5bn") pod "39c20738-ddf0-47f5-a7a1-25d37f145fb1" (UID: "39c20738-ddf0-47f5-a7a1-25d37f145fb1"). InnerVolumeSpecName "kube-api-access-km5bn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 19 15:36:30.302687 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:36:30.302664 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/39c20738-ddf0-47f5-a7a1-25d37f145fb1-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "39c20738-ddf0-47f5-a7a1-25d37f145fb1" (UID: "39c20738-ddf0-47f5-a7a1-25d37f145fb1"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 19 15:36:30.401116 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:36:30.401095 2558 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/39c20738-ddf0-47f5-a7a1-25d37f145fb1-openshift-service-ca\") on node \"ip-10-0-138-220.ec2.internal\" DevicePath \"\"" Jul 19 15:36:30.401116 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:36:30.401116 2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-km5bn\" (UniqueName: \"kubernetes.io/projected/39c20738-ddf0-47f5-a7a1-25d37f145fb1-kube-api-access-km5bn\") on node \"ip-10-0-138-220.ec2.internal\" DevicePath \"\"" Jul 19 15:36:30.401247 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:36:30.401126 2558 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/39c20738-ddf0-47f5-a7a1-25d37f145fb1-maas-api-tls\") on node \"ip-10-0-138-220.ec2.internal\" DevicePath \"\"" Jul 19 15:36:31.022016 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:36:31.021938 2558 generic.go:356] "Generic (PLEG): container finished" podID="39c20738-ddf0-47f5-a7a1-25d37f145fb1" containerID="3f68fded2ef4945a7d4a6b169b16288d2be5dae6b2b3a3f1a918fa3082339525" exitCode=0 Jul 19 15:36:31.022016 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:36:31.021984 2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5e6749-849f76fcc6-hbj58" Jul 19 15:36:31.022495 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:36:31.022052 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5e6749-849f76fcc6-hbj58" event={"ID":"39c20738-ddf0-47f5-a7a1-25d37f145fb1","Type":"ContainerDied","Data":"3f68fded2ef4945a7d4a6b169b16288d2be5dae6b2b3a3f1a918fa3082339525"} Jul 19 15:36:31.022495 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:36:31.022105 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-5e6749-849f76fcc6-hbj58" event={"ID":"39c20738-ddf0-47f5-a7a1-25d37f145fb1","Type":"ContainerDied","Data":"e76da186944464fe6335d09ca0e3b873c7638d7d988e4f7f547b046354cc20f1"} Jul 19 15:36:31.022495 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:36:31.022139 2558 scope.go:117] "RemoveContainer" containerID="3f68fded2ef4945a7d4a6b169b16288d2be5dae6b2b3a3f1a918fa3082339525" Jul 19 15:36:31.031279 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:36:31.031264 2558 scope.go:117] "RemoveContainer" containerID="3f68fded2ef4945a7d4a6b169b16288d2be5dae6b2b3a3f1a918fa3082339525" Jul 19 15:36:31.031529 ip-10-0-138-220 kubenswrapper[2558]: E0719 15:36:31.031508 2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3f68fded2ef4945a7d4a6b169b16288d2be5dae6b2b3a3f1a918fa3082339525\": container with ID starting with 3f68fded2ef4945a7d4a6b169b16288d2be5dae6b2b3a3f1a918fa3082339525 not found: ID does not exist" containerID="3f68fded2ef4945a7d4a6b169b16288d2be5dae6b2b3a3f1a918fa3082339525" Jul 19 15:36:31.031600 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:36:31.031540 2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3f68fded2ef4945a7d4a6b169b16288d2be5dae6b2b3a3f1a918fa3082339525"} err="failed to get container status \"3f68fded2ef4945a7d4a6b169b16288d2be5dae6b2b3a3f1a918fa3082339525\": rpc error: code = NotFound desc = could not find container \"3f68fded2ef4945a7d4a6b169b16288d2be5dae6b2b3a3f1a918fa3082339525\": container with ID starting with 3f68fded2ef4945a7d4a6b169b16288d2be5dae6b2b3a3f1a918fa3082339525 not found: ID does not exist" Jul 19 15:36:31.041893 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:36:31.041871 2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-5e6749-849f76fcc6-hbj58"] Jul 19 15:36:31.047318 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:36:31.047298 2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-5e6749-849f76fcc6-hbj58"] Jul 19 15:36:31.565093 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:36:31.565051 2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="39c20738-ddf0-47f5-a7a1-25d37f145fb1" path="/var/lib/kubelet/pods/39c20738-ddf0-47f5-a7a1-25d37f145fb1/volumes" Jul 19 15:36:31.592679 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:36:31.592649 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-d4lll"] Jul 19 15:36:40.819533 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:36:40.819439 2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-5e6749-openshift-default-78f76d74cf-rg9w5"] Jul 19 15:36:40.819992 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:36:40.819685 2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-5e6749-openshift-default-78f76d74cf-rg9w5" podUID="4e8305f3-2721-45bb-bb52-53991a5a49e1" containerName="istio-proxy" containerID="cri-o://d57f9ae7a0aeef056ec2da1d0fab1a500769258d5b05746627a7a50a86af731e" gracePeriod=30 Jul 19 15:36:41.756333 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:36:41.756295 2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-5c4f7c-openshift-default-69455b5bf5-p47m2"] Jul 19 15:36:41.756724 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:36:41.756703 2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c794cda3-3fb4-4f1d-a291-808b26541154" containerName="maas-api" Jul 19 15:36:41.756724 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:36:41.756723 2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="c794cda3-3fb4-4f1d-a291-808b26541154" containerName="maas-api" Jul 19 15:36:41.756926 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:36:41.756743 2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7cd49b06-890b-43ff-9d71-ae63ec00e193" containerName="istio-proxy" Jul 19 15:36:41.756926 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:36:41.756754 2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="7cd49b06-890b-43ff-9d71-ae63ec00e193" containerName="istio-proxy" Jul 19 15:36:41.756926 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:36:41.756775 2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="39c20738-ddf0-47f5-a7a1-25d37f145fb1" containerName="maas-api" Jul 19 15:36:41.756926 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:36:41.756783 2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="39c20738-ddf0-47f5-a7a1-25d37f145fb1" containerName="maas-api" Jul 19 15:36:41.756926 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:36:41.756863 2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="c794cda3-3fb4-4f1d-a291-808b26541154" containerName="maas-api" Jul 19 15:36:41.756926 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:36:41.756873 2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="7cd49b06-890b-43ff-9d71-ae63ec00e193" containerName="istio-proxy" Jul 19 15:36:41.756926 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:36:41.756882 2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="39c20738-ddf0-47f5-a7a1-25d37f145fb1" containerName="maas-api" Jul 19 15:36:41.761170 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:36:41.761150 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-5c4f7c-openshift-default-69455b5bf5-p47m2" Jul 19 15:36:41.763175 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:36:41.763150 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-5c4f7c-openshift-default-dockercfg-v6dj8\"" Jul 19 15:36:41.769430 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:36:41.768721 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-5c4f7c-openshift-default-69455b5bf5-p47m2"] Jul 19 15:36:41.784925 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:36:41.784905 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/81f9a2e7-69db-4eaa-9f5f-2909613cd9ee-istio-data\") pod \"e2e-api-a-5c4f7c-openshift-default-69455b5bf5-p47m2\" (UID: \"81f9a2e7-69db-4eaa-9f5f-2909613cd9ee\") " pod="openshift-ingress/e2e-api-a-5c4f7c-openshift-default-69455b5bf5-p47m2" Jul 19 15:36:41.785066 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:36:41.784936 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/81f9a2e7-69db-4eaa-9f5f-2909613cd9ee-istio-envoy\") pod \"e2e-api-a-5c4f7c-openshift-default-69455b5bf5-p47m2\" (UID: \"81f9a2e7-69db-4eaa-9f5f-2909613cd9ee\") " pod="openshift-ingress/e2e-api-a-5c4f7c-openshift-default-69455b5bf5-p47m2" Jul 19 15:36:41.785066 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:36:41.784962 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/81f9a2e7-69db-4eaa-9f5f-2909613cd9ee-istio-podinfo\") pod \"e2e-api-a-5c4f7c-openshift-default-69455b5bf5-p47m2\" (UID: \"81f9a2e7-69db-4eaa-9f5f-2909613cd9ee\") " pod="openshift-ingress/e2e-api-a-5c4f7c-openshift-default-69455b5bf5-p47m2" Jul 19 15:36:41.785066 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:36:41.784998 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/81f9a2e7-69db-4eaa-9f5f-2909613cd9ee-workload-certs\") pod \"e2e-api-a-5c4f7c-openshift-default-69455b5bf5-p47m2\" (UID: \"81f9a2e7-69db-4eaa-9f5f-2909613cd9ee\") " pod="openshift-ingress/e2e-api-a-5c4f7c-openshift-default-69455b5bf5-p47m2" Jul 19 15:36:41.785066 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:36:41.785016 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/81f9a2e7-69db-4eaa-9f5f-2909613cd9ee-workload-socket\") pod \"e2e-api-a-5c4f7c-openshift-default-69455b5bf5-p47m2\" (UID: \"81f9a2e7-69db-4eaa-9f5f-2909613cd9ee\") " pod="openshift-ingress/e2e-api-a-5c4f7c-openshift-default-69455b5bf5-p47m2" Jul 19 15:36:41.785207 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:36:41.785073 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/81f9a2e7-69db-4eaa-9f5f-2909613cd9ee-credential-socket\") pod \"e2e-api-a-5c4f7c-openshift-default-69455b5bf5-p47m2\" (UID: \"81f9a2e7-69db-4eaa-9f5f-2909613cd9ee\") " pod="openshift-ingress/e2e-api-a-5c4f7c-openshift-default-69455b5bf5-p47m2" Jul 19 15:36:41.785207 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:36:41.785093 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/81f9a2e7-69db-4eaa-9f5f-2909613cd9ee-istio-token\") pod \"e2e-api-a-5c4f7c-openshift-default-69455b5bf5-p47m2\" (UID: \"81f9a2e7-69db-4eaa-9f5f-2909613cd9ee\") " pod="openshift-ingress/e2e-api-a-5c4f7c-openshift-default-69455b5bf5-p47m2" Jul 19 15:36:41.785207 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:36:41.785114 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/81f9a2e7-69db-4eaa-9f5f-2909613cd9ee-istiod-ca-cert\") pod \"e2e-api-a-5c4f7c-openshift-default-69455b5bf5-p47m2\" (UID: \"81f9a2e7-69db-4eaa-9f5f-2909613cd9ee\") " pod="openshift-ingress/e2e-api-a-5c4f7c-openshift-default-69455b5bf5-p47m2" Jul 19 15:36:41.785207 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:36:41.785184 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h2s4v\" (UniqueName: \"kubernetes.io/projected/81f9a2e7-69db-4eaa-9f5f-2909613cd9ee-kube-api-access-h2s4v\") pod \"e2e-api-a-5c4f7c-openshift-default-69455b5bf5-p47m2\" (UID: \"81f9a2e7-69db-4eaa-9f5f-2909613cd9ee\") " pod="openshift-ingress/e2e-api-a-5c4f7c-openshift-default-69455b5bf5-p47m2" Jul 19 15:36:41.886297 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:36:41.886262 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/81f9a2e7-69db-4eaa-9f5f-2909613cd9ee-istio-data\") pod \"e2e-api-a-5c4f7c-openshift-default-69455b5bf5-p47m2\" (UID: \"81f9a2e7-69db-4eaa-9f5f-2909613cd9ee\") " pod="openshift-ingress/e2e-api-a-5c4f7c-openshift-default-69455b5bf5-p47m2" Jul 19 15:36:41.886297 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:36:41.886300 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/81f9a2e7-69db-4eaa-9f5f-2909613cd9ee-istio-envoy\") pod \"e2e-api-a-5c4f7c-openshift-default-69455b5bf5-p47m2\" (UID: \"81f9a2e7-69db-4eaa-9f5f-2909613cd9ee\") " pod="openshift-ingress/e2e-api-a-5c4f7c-openshift-default-69455b5bf5-p47m2" Jul 19 15:36:41.886839 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:36:41.886319 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/81f9a2e7-69db-4eaa-9f5f-2909613cd9ee-istio-podinfo\") pod \"e2e-api-a-5c4f7c-openshift-default-69455b5bf5-p47m2\" (UID: \"81f9a2e7-69db-4eaa-9f5f-2909613cd9ee\") " pod="openshift-ingress/e2e-api-a-5c4f7c-openshift-default-69455b5bf5-p47m2" Jul 19 15:36:41.886839 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:36:41.886346 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/81f9a2e7-69db-4eaa-9f5f-2909613cd9ee-workload-certs\") pod \"e2e-api-a-5c4f7c-openshift-default-69455b5bf5-p47m2\" (UID: \"81f9a2e7-69db-4eaa-9f5f-2909613cd9ee\") " pod="openshift-ingress/e2e-api-a-5c4f7c-openshift-default-69455b5bf5-p47m2" Jul 19 15:36:41.886839 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:36:41.886371 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/81f9a2e7-69db-4eaa-9f5f-2909613cd9ee-workload-socket\") pod \"e2e-api-a-5c4f7c-openshift-default-69455b5bf5-p47m2\" (UID: \"81f9a2e7-69db-4eaa-9f5f-2909613cd9ee\") " pod="openshift-ingress/e2e-api-a-5c4f7c-openshift-default-69455b5bf5-p47m2" Jul 19 15:36:41.886839 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:36:41.886412 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/81f9a2e7-69db-4eaa-9f5f-2909613cd9ee-credential-socket\") pod \"e2e-api-a-5c4f7c-openshift-default-69455b5bf5-p47m2\" (UID: \"81f9a2e7-69db-4eaa-9f5f-2909613cd9ee\") " pod="openshift-ingress/e2e-api-a-5c4f7c-openshift-default-69455b5bf5-p47m2" Jul 19 15:36:41.886839 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:36:41.886440 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/81f9a2e7-69db-4eaa-9f5f-2909613cd9ee-istio-token\") pod \"e2e-api-a-5c4f7c-openshift-default-69455b5bf5-p47m2\" (UID: \"81f9a2e7-69db-4eaa-9f5f-2909613cd9ee\") " pod="openshift-ingress/e2e-api-a-5c4f7c-openshift-default-69455b5bf5-p47m2" Jul 19 15:36:41.886839 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:36:41.886464 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/81f9a2e7-69db-4eaa-9f5f-2909613cd9ee-istiod-ca-cert\") pod \"e2e-api-a-5c4f7c-openshift-default-69455b5bf5-p47m2\" (UID: \"81f9a2e7-69db-4eaa-9f5f-2909613cd9ee\") " pod="openshift-ingress/e2e-api-a-5c4f7c-openshift-default-69455b5bf5-p47m2" Jul 19 15:36:41.886839 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:36:41.886510 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h2s4v\" (UniqueName: \"kubernetes.io/projected/81f9a2e7-69db-4eaa-9f5f-2909613cd9ee-kube-api-access-h2s4v\") pod \"e2e-api-a-5c4f7c-openshift-default-69455b5bf5-p47m2\" (UID: \"81f9a2e7-69db-4eaa-9f5f-2909613cd9ee\") " pod="openshift-ingress/e2e-api-a-5c4f7c-openshift-default-69455b5bf5-p47m2" Jul 19 15:36:41.886839 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:36:41.886711 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/81f9a2e7-69db-4eaa-9f5f-2909613cd9ee-istio-data\") pod \"e2e-api-a-5c4f7c-openshift-default-69455b5bf5-p47m2\" (UID: \"81f9a2e7-69db-4eaa-9f5f-2909613cd9ee\") " pod="openshift-ingress/e2e-api-a-5c4f7c-openshift-default-69455b5bf5-p47m2" Jul 19 15:36:41.887331 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:36:41.886931 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/81f9a2e7-69db-4eaa-9f5f-2909613cd9ee-credential-socket\") pod \"e2e-api-a-5c4f7c-openshift-default-69455b5bf5-p47m2\" (UID: \"81f9a2e7-69db-4eaa-9f5f-2909613cd9ee\") " pod="openshift-ingress/e2e-api-a-5c4f7c-openshift-default-69455b5bf5-p47m2" Jul 19 15:36:41.887331 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:36:41.886988 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/81f9a2e7-69db-4eaa-9f5f-2909613cd9ee-workload-socket\") pod \"e2e-api-a-5c4f7c-openshift-default-69455b5bf5-p47m2\" (UID: \"81f9a2e7-69db-4eaa-9f5f-2909613cd9ee\") " pod="openshift-ingress/e2e-api-a-5c4f7c-openshift-default-69455b5bf5-p47m2" Jul 19 15:36:41.887652 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:36:41.887624 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/81f9a2e7-69db-4eaa-9f5f-2909613cd9ee-istiod-ca-cert\") pod \"e2e-api-a-5c4f7c-openshift-default-69455b5bf5-p47m2\" (UID: \"81f9a2e7-69db-4eaa-9f5f-2909613cd9ee\") " pod="openshift-ingress/e2e-api-a-5c4f7c-openshift-default-69455b5bf5-p47m2" Jul 19 15:36:41.887765 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:36:41.887675 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/81f9a2e7-69db-4eaa-9f5f-2909613cd9ee-workload-certs\") pod \"e2e-api-a-5c4f7c-openshift-default-69455b5bf5-p47m2\" (UID: \"81f9a2e7-69db-4eaa-9f5f-2909613cd9ee\") " pod="openshift-ingress/e2e-api-a-5c4f7c-openshift-default-69455b5bf5-p47m2" Jul 19 15:36:41.888924 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:36:41.888900 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/81f9a2e7-69db-4eaa-9f5f-2909613cd9ee-istio-envoy\") pod \"e2e-api-a-5c4f7c-openshift-default-69455b5bf5-p47m2\" (UID: \"81f9a2e7-69db-4eaa-9f5f-2909613cd9ee\") " pod="openshift-ingress/e2e-api-a-5c4f7c-openshift-default-69455b5bf5-p47m2" Jul 19 15:36:41.889205 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:36:41.889187 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/81f9a2e7-69db-4eaa-9f5f-2909613cd9ee-istio-podinfo\") pod \"e2e-api-a-5c4f7c-openshift-default-69455b5bf5-p47m2\" (UID: \"81f9a2e7-69db-4eaa-9f5f-2909613cd9ee\") " pod="openshift-ingress/e2e-api-a-5c4f7c-openshift-default-69455b5bf5-p47m2" Jul 19 15:36:41.900058 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:36:41.899993 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/81f9a2e7-69db-4eaa-9f5f-2909613cd9ee-istio-token\") pod \"e2e-api-a-5c4f7c-openshift-default-69455b5bf5-p47m2\" (UID: \"81f9a2e7-69db-4eaa-9f5f-2909613cd9ee\") " pod="openshift-ingress/e2e-api-a-5c4f7c-openshift-default-69455b5bf5-p47m2" Jul 19 15:36:41.900321 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:36:41.900303 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h2s4v\" (UniqueName: \"kubernetes.io/projected/81f9a2e7-69db-4eaa-9f5f-2909613cd9ee-kube-api-access-h2s4v\") pod \"e2e-api-a-5c4f7c-openshift-default-69455b5bf5-p47m2\" (UID: \"81f9a2e7-69db-4eaa-9f5f-2909613cd9ee\") " pod="openshift-ingress/e2e-api-a-5c4f7c-openshift-default-69455b5bf5-p47m2" Jul 19 15:36:42.072911 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:36:42.072824 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-5c4f7c-openshift-default-69455b5bf5-p47m2" Jul 19 15:36:42.206451 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:36:42.206408 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-5c4f7c-openshift-default-69455b5bf5-p47m2"] Jul 19 15:36:42.209332 ip-10-0-138-220 kubenswrapper[2558]: W0719 15:36:42.209192 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod81f9a2e7_69db_4eaa_9f5f_2909613cd9ee.slice/crio-512fc0906ecb816d6bd516ca6dec014de71939e4701b61770ea679fcdc5ac5d2 WatchSource:0}: Error finding container 512fc0906ecb816d6bd516ca6dec014de71939e4701b61770ea679fcdc5ac5d2: Status 404 returned error can't find the container with id 512fc0906ecb816d6bd516ca6dec014de71939e4701b61770ea679fcdc5ac5d2 Jul 19 15:36:42.211318 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:36:42.211277 2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 19 15:36:42.211396 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:36:42.211372 2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 19 15:36:42.211439 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:36:42.211414 2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 19 15:36:43.078766 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:36:43.078731 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-5c4f7c-openshift-default-69455b5bf5-p47m2" event={"ID":"81f9a2e7-69db-4eaa-9f5f-2909613cd9ee","Type":"ContainerStarted","Data":"1c59cb8899a7b13b6181e4e207211cffab09154cd47600fde0b5fd8db150e76b"} Jul 19 15:36:43.079166 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:36:43.078781 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-5c4f7c-openshift-default-69455b5bf5-p47m2" event={"ID":"81f9a2e7-69db-4eaa-9f5f-2909613cd9ee","Type":"ContainerStarted","Data":"512fc0906ecb816d6bd516ca6dec014de71939e4701b61770ea679fcdc5ac5d2"} Jul 19 15:36:43.097754 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:36:43.097704 2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-5c4f7c-openshift-default-69455b5bf5-p47m2" podStartSLOduration=2.097659029 podStartE2EDuration="2.097659029s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-19 15:36:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-19 15:36:43.096373802 +0000 UTC m=+2344.018076517" watchObservedRunningTime="2026-07-19 15:36:43.097659029 +0000 UTC m=+2344.019361745" Jul 19 15:36:44.073774 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:36:44.073740 2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-5c4f7c-openshift-default-69455b5bf5-p47m2" Jul 19 15:36:44.078762 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:36:44.078737 2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-5c4f7c-openshift-default-69455b5bf5-p47m2" Jul 19 15:36:44.083338 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:36:44.083314 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-5c4f7c-openshift-default-69455b5bf5-p47m2" Jul 19 15:36:44.084153 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:36:44.084137 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-5c4f7c-openshift-default-69455b5bf5-p47m2" Jul 19 15:36:45.298849 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:36:45.298815 2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-5c4f7c-5478847ccb-8d95h"] Jul 19 15:36:45.299273 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:36:45.299168 2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7cd49b06-890b-43ff-9d71-ae63ec00e193" containerName="istio-proxy" Jul 19 15:36:45.299273 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:36:45.299181 2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="7cd49b06-890b-43ff-9d71-ae63ec00e193" containerName="istio-proxy" Jul 19 15:36:45.299273 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:36:45.299229 2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="7cd49b06-890b-43ff-9d71-ae63ec00e193" containerName="istio-proxy" Jul 19 15:36:45.302178 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:36:45.302156 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5c4f7c-5478847ccb-8d95h" Jul 19 15:36:45.304209 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:36:45.304185 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-5c4f7c-serving-cert\"" Jul 19 15:36:45.304436 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:36:45.304412 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 19 15:36:45.304971 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:36:45.304954 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Jul 19 15:36:45.305098 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:36:45.304984 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-ljt6w\"" Jul 19 15:36:45.313000 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:36:45.312972 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-5c4f7c-5478847ccb-8d95h"] Jul 19 15:36:45.417326 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:36:45.417289 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7c3a4e60-0549-425e-b089-579ded99578e-maas-api-tls\") pod \"maas-api-e2e-api-a-5c4f7c-5478847ccb-8d95h\" (UID: \"7c3a4e60-0549-425e-b089-579ded99578e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5c4f7c-5478847ccb-8d95h" Jul 19 15:36:45.417520 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:36:45.417365 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wvddq\" (UniqueName: \"kubernetes.io/projected/7c3a4e60-0549-425e-b089-579ded99578e-kube-api-access-wvddq\") pod \"maas-api-e2e-api-a-5c4f7c-5478847ccb-8d95h\" (UID: \"7c3a4e60-0549-425e-b089-579ded99578e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5c4f7c-5478847ccb-8d95h" Jul 19 15:36:45.417520 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:36:45.417399 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7c3a4e60-0549-425e-b089-579ded99578e-openshift-service-ca\") pod \"maas-api-e2e-api-a-5c4f7c-5478847ccb-8d95h\" (UID: \"7c3a4e60-0549-425e-b089-579ded99578e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5c4f7c-5478847ccb-8d95h" Jul 19 15:36:45.518097 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:36:45.518047 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wvddq\" (UniqueName: \"kubernetes.io/projected/7c3a4e60-0549-425e-b089-579ded99578e-kube-api-access-wvddq\") pod \"maas-api-e2e-api-a-5c4f7c-5478847ccb-8d95h\" (UID: \"7c3a4e60-0549-425e-b089-579ded99578e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5c4f7c-5478847ccb-8d95h" Jul 19 15:36:45.518273 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:36:45.518110 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7c3a4e60-0549-425e-b089-579ded99578e-openshift-service-ca\") pod \"maas-api-e2e-api-a-5c4f7c-5478847ccb-8d95h\" (UID: \"7c3a4e60-0549-425e-b089-579ded99578e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5c4f7c-5478847ccb-8d95h" Jul 19 15:36:45.518273 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:36:45.518137 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7c3a4e60-0549-425e-b089-579ded99578e-maas-api-tls\") pod \"maas-api-e2e-api-a-5c4f7c-5478847ccb-8d95h\" (UID: \"7c3a4e60-0549-425e-b089-579ded99578e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5c4f7c-5478847ccb-8d95h" Jul 19 15:36:45.518828 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:36:45.518799 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7c3a4e60-0549-425e-b089-579ded99578e-openshift-service-ca\") pod \"maas-api-e2e-api-a-5c4f7c-5478847ccb-8d95h\" (UID: \"7c3a4e60-0549-425e-b089-579ded99578e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5c4f7c-5478847ccb-8d95h" Jul 19 15:36:45.520715 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:36:45.520696 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7c3a4e60-0549-425e-b089-579ded99578e-maas-api-tls\") pod \"maas-api-e2e-api-a-5c4f7c-5478847ccb-8d95h\" (UID: \"7c3a4e60-0549-425e-b089-579ded99578e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5c4f7c-5478847ccb-8d95h" Jul 19 15:36:45.526286 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:36:45.526259 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wvddq\" (UniqueName: \"kubernetes.io/projected/7c3a4e60-0549-425e-b089-579ded99578e-kube-api-access-wvddq\") pod \"maas-api-e2e-api-a-5c4f7c-5478847ccb-8d95h\" (UID: \"7c3a4e60-0549-425e-b089-579ded99578e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5c4f7c-5478847ccb-8d95h" Jul 19 15:36:45.614413 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:36:45.614323 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5c4f7c-5478847ccb-8d95h" Jul 19 15:36:45.751575 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:36:45.751548 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-5c4f7c-5478847ccb-8d95h"] Jul 19 15:36:45.754329 ip-10-0-138-220 kubenswrapper[2558]: W0719 15:36:45.754298 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7c3a4e60_0549_425e_b089_579ded99578e.slice/crio-44203775ea938d3e9ce4f8567c9584ab3f0d0030f51cc58389ab32010146c000 WatchSource:0}: Error finding container 44203775ea938d3e9ce4f8567c9584ab3f0d0030f51cc58389ab32010146c000: Status 404 returned error can't find the container with id 44203775ea938d3e9ce4f8567c9584ab3f0d0030f51cc58389ab32010146c000 Jul 19 15:36:45.756272 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:36:45.756236 2558 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 19 15:36:45.878767 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:36:45.878666 2558 patch_prober.go:28] interesting pod/e2e-inf-b-5e6749-openshift-default-78f76d74cf-rg9w5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" start-of-body= Jul 19 15:36:45.878926 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:36:45.878745 2558 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-5e6749-openshift-default-78f76d74cf-rg9w5" podUID="4e8305f3-2721-45bb-bb52-53991a5a49e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" Jul 19 15:36:46.070092 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:36:46.070061 2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-5e6749-openshift-default-78f76d74cf-rg9w5" Jul 19 15:36:46.095365 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:36:46.095333 2558 generic.go:356] "Generic (PLEG): container finished" podID="4e8305f3-2721-45bb-bb52-53991a5a49e1" containerID="d57f9ae7a0aeef056ec2da1d0fab1a500769258d5b05746627a7a50a86af731e" exitCode=0 Jul 19 15:36:46.095530 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:36:46.095387 2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-5e6749-openshift-default-78f76d74cf-rg9w5" Jul 19 15:36:46.095530 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:36:46.095415 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-5e6749-openshift-default-78f76d74cf-rg9w5" event={"ID":"4e8305f3-2721-45bb-bb52-53991a5a49e1","Type":"ContainerDied","Data":"d57f9ae7a0aeef056ec2da1d0fab1a500769258d5b05746627a7a50a86af731e"} Jul 19 15:36:46.095530 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:36:46.095494 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-5e6749-openshift-default-78f76d74cf-rg9w5" event={"ID":"4e8305f3-2721-45bb-bb52-53991a5a49e1","Type":"ContainerDied","Data":"cb5fe98eccc9e7a511f0995bf27b8742ff9413a6dfe07cb8f731c75701bb348e"} Jul 19 15:36:46.095717 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:36:46.095532 2558 scope.go:117] "RemoveContainer" containerID="d57f9ae7a0aeef056ec2da1d0fab1a500769258d5b05746627a7a50a86af731e" Jul 19 15:36:46.096990 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:36:46.096962 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5c4f7c-5478847ccb-8d95h" event={"ID":"7c3a4e60-0549-425e-b089-579ded99578e","Type":"ContainerStarted","Data":"44203775ea938d3e9ce4f8567c9584ab3f0d0030f51cc58389ab32010146c000"} Jul 19 15:36:46.104590 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:36:46.104573 2558 scope.go:117] "RemoveContainer" containerID="c305bf835b3c53663d8bfb7b93fefc059a3f7b30186ddaaee947401089944dc6" Jul 19 15:36:46.112711 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:36:46.112693 2558 scope.go:117] "RemoveContainer" containerID="d57f9ae7a0aeef056ec2da1d0fab1a500769258d5b05746627a7a50a86af731e" Jul 19 15:36:46.113038 ip-10-0-138-220 kubenswrapper[2558]: E0719 15:36:46.112998 2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d57f9ae7a0aeef056ec2da1d0fab1a500769258d5b05746627a7a50a86af731e\": container with ID starting with d57f9ae7a0aeef056ec2da1d0fab1a500769258d5b05746627a7a50a86af731e not found: ID does not exist" containerID="d57f9ae7a0aeef056ec2da1d0fab1a500769258d5b05746627a7a50a86af731e" Jul 19 15:36:46.113137 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:36:46.113052 2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d57f9ae7a0aeef056ec2da1d0fab1a500769258d5b05746627a7a50a86af731e"} err="failed to get container status \"d57f9ae7a0aeef056ec2da1d0fab1a500769258d5b05746627a7a50a86af731e\": rpc error: code = NotFound desc = could not find container \"d57f9ae7a0aeef056ec2da1d0fab1a500769258d5b05746627a7a50a86af731e\": container with ID starting with d57f9ae7a0aeef056ec2da1d0fab1a500769258d5b05746627a7a50a86af731e not found: ID does not exist" Jul 19 15:36:46.113137 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:36:46.113073 2558 scope.go:117] "RemoveContainer" containerID="c305bf835b3c53663d8bfb7b93fefc059a3f7b30186ddaaee947401089944dc6" Jul 19 15:36:46.113353 ip-10-0-138-220 kubenswrapper[2558]: E0719 15:36:46.113333 2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c305bf835b3c53663d8bfb7b93fefc059a3f7b30186ddaaee947401089944dc6\": container with ID starting with c305bf835b3c53663d8bfb7b93fefc059a3f7b30186ddaaee947401089944dc6 not found: ID does not exist" containerID="c305bf835b3c53663d8bfb7b93fefc059a3f7b30186ddaaee947401089944dc6" Jul 19 15:36:46.113398 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:36:46.113360 2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c305bf835b3c53663d8bfb7b93fefc059a3f7b30186ddaaee947401089944dc6"} err="failed to get container status \"c305bf835b3c53663d8bfb7b93fefc059a3f7b30186ddaaee947401089944dc6\": rpc error: code = NotFound desc = could not find container \"c305bf835b3c53663d8bfb7b93fefc059a3f7b30186ddaaee947401089944dc6\": container with ID starting with c305bf835b3c53663d8bfb7b93fefc059a3f7b30186ddaaee947401089944dc6 not found: ID does not exist" Jul 19 15:36:46.123671 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:36:46.123643 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e8305f3-2721-45bb-bb52-53991a5a49e1-workload-socket\") pod \"4e8305f3-2721-45bb-bb52-53991a5a49e1\" (UID: \"4e8305f3-2721-45bb-bb52-53991a5a49e1\") " Jul 19 15:36:46.123784 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:36:46.123687 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4e8305f3-2721-45bb-bb52-53991a5a49e1-istio-token\") pod \"4e8305f3-2721-45bb-bb52-53991a5a49e1\" (UID: \"4e8305f3-2721-45bb-bb52-53991a5a49e1\") " Jul 19 15:36:46.123784 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:36:46.123723 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4e8305f3-2721-45bb-bb52-53991a5a49e1-istio-data\") pod \"4e8305f3-2721-45bb-bb52-53991a5a49e1\" (UID: \"4e8305f3-2721-45bb-bb52-53991a5a49e1\") " Jul 19 15:36:46.123784 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:36:46.123760 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-x487w\" (UniqueName: \"kubernetes.io/projected/4e8305f3-2721-45bb-bb52-53991a5a49e1-kube-api-access-x487w\") pod \"4e8305f3-2721-45bb-bb52-53991a5a49e1\" (UID: \"4e8305f3-2721-45bb-bb52-53991a5a49e1\") " Jul 19 15:36:46.123945 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:36:46.123793 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4e8305f3-2721-45bb-bb52-53991a5a49e1-istio-envoy\") pod \"4e8305f3-2721-45bb-bb52-53991a5a49e1\" (UID: \"4e8305f3-2721-45bb-bb52-53991a5a49e1\") " Jul 19 15:36:46.123945 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:36:46.123832 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4e8305f3-2721-45bb-bb52-53991a5a49e1-istio-podinfo\") pod \"4e8305f3-2721-45bb-bb52-53991a5a49e1\" (UID: \"4e8305f3-2721-45bb-bb52-53991a5a49e1\") " Jul 19 15:36:46.123945 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:36:46.123872 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e8305f3-2721-45bb-bb52-53991a5a49e1-credential-socket\") pod \"4e8305f3-2721-45bb-bb52-53991a5a49e1\" (UID: \"4e8305f3-2721-45bb-bb52-53991a5a49e1\") " Jul 19 15:36:46.123945 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:36:46.123921 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4e8305f3-2721-45bb-bb52-53991a5a49e1-workload-certs\") pod \"4e8305f3-2721-45bb-bb52-53991a5a49e1\" (UID: \"4e8305f3-2721-45bb-bb52-53991a5a49e1\") " Jul 19 15:36:46.123945 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:36:46.123936 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e8305f3-2721-45bb-bb52-53991a5a49e1-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "4e8305f3-2721-45bb-bb52-53991a5a49e1" (UID: "4e8305f3-2721-45bb-bb52-53991a5a49e1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 19 15:36:46.123945 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:36:46.123945 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4e8305f3-2721-45bb-bb52-53991a5a49e1-istiod-ca-cert\") pod \"4e8305f3-2721-45bb-bb52-53991a5a49e1\" (UID: \"4e8305f3-2721-45bb-bb52-53991a5a49e1\") " Jul 19 15:36:46.124277 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:36:46.124226 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e8305f3-2721-45bb-bb52-53991a5a49e1-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "4e8305f3-2721-45bb-bb52-53991a5a49e1" (UID: "4e8305f3-2721-45bb-bb52-53991a5a49e1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 19 15:36:46.124277 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:36:46.124251 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e8305f3-2721-45bb-bb52-53991a5a49e1-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "4e8305f3-2721-45bb-bb52-53991a5a49e1" (UID: "4e8305f3-2721-45bb-bb52-53991a5a49e1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 19 15:36:46.124585 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:36:46.124559 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e8305f3-2721-45bb-bb52-53991a5a49e1-istio-data" (OuterVolumeSpecName: "istio-data") pod "4e8305f3-2721-45bb-bb52-53991a5a49e1" (UID: "4e8305f3-2721-45bb-bb52-53991a5a49e1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 19 15:36:46.124585 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:36:46.124570 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4e8305f3-2721-45bb-bb52-53991a5a49e1-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "4e8305f3-2721-45bb-bb52-53991a5a49e1" (UID: "4e8305f3-2721-45bb-bb52-53991a5a49e1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 19 15:36:46.125077 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:36:46.125056 2558 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4e8305f3-2721-45bb-bb52-53991a5a49e1-workload-certs\") on node \"ip-10-0-138-220.ec2.internal\" DevicePath \"\"" Jul 19 15:36:46.125164 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:36:46.125081 2558 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4e8305f3-2721-45bb-bb52-53991a5a49e1-istiod-ca-cert\") on node \"ip-10-0-138-220.ec2.internal\" DevicePath \"\"" Jul 19 15:36:46.125164 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:36:46.125096 2558 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e8305f3-2721-45bb-bb52-53991a5a49e1-workload-socket\") on node \"ip-10-0-138-220.ec2.internal\" DevicePath \"\"" Jul 19 15:36:46.125164 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:36:46.125112 2558 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4e8305f3-2721-45bb-bb52-53991a5a49e1-istio-data\") on node \"ip-10-0-138-220.ec2.internal\" DevicePath \"\"" Jul 19 15:36:46.125164 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:36:46.125124 2558 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e8305f3-2721-45bb-bb52-53991a5a49e1-credential-socket\") on node \"ip-10-0-138-220.ec2.internal\" DevicePath \"\"" Jul 19 15:36:46.126414 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:36:46.126391 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e8305f3-2721-45bb-bb52-53991a5a49e1-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "4e8305f3-2721-45bb-bb52-53991a5a49e1" (UID: "4e8305f3-2721-45bb-bb52-53991a5a49e1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 19 15:36:46.126508 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:36:46.126464 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4e8305f3-2721-45bb-bb52-53991a5a49e1-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "4e8305f3-2721-45bb-bb52-53991a5a49e1" (UID: "4e8305f3-2721-45bb-bb52-53991a5a49e1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 19 15:36:46.126508 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:36:46.126488 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4e8305f3-2721-45bb-bb52-53991a5a49e1-istio-token" (OuterVolumeSpecName: "istio-token") pod "4e8305f3-2721-45bb-bb52-53991a5a49e1" (UID: "4e8305f3-2721-45bb-bb52-53991a5a49e1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 19 15:36:46.126664 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:36:46.126649 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4e8305f3-2721-45bb-bb52-53991a5a49e1-kube-api-access-x487w" (OuterVolumeSpecName: "kube-api-access-x487w") pod "4e8305f3-2721-45bb-bb52-53991a5a49e1" (UID: "4e8305f3-2721-45bb-bb52-53991a5a49e1"). InnerVolumeSpecName "kube-api-access-x487w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 19 15:36:46.226250 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:36:46.226218 2558 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4e8305f3-2721-45bb-bb52-53991a5a49e1-istio-token\") on node \"ip-10-0-138-220.ec2.internal\" DevicePath \"\"" Jul 19 15:36:46.226250 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:36:46.226251 2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x487w\" (UniqueName: \"kubernetes.io/projected/4e8305f3-2721-45bb-bb52-53991a5a49e1-kube-api-access-x487w\") on node \"ip-10-0-138-220.ec2.internal\" DevicePath \"\"" Jul 19 15:36:46.226514 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:36:46.226262 2558 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4e8305f3-2721-45bb-bb52-53991a5a49e1-istio-envoy\") on node \"ip-10-0-138-220.ec2.internal\" DevicePath \"\"" Jul 19 15:36:46.226514 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:36:46.226270 2558 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4e8305f3-2721-45bb-bb52-53991a5a49e1-istio-podinfo\") on node \"ip-10-0-138-220.ec2.internal\" DevicePath \"\"" Jul 19 15:36:46.415883 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:36:46.415848 2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-5e6749-openshift-default-78f76d74cf-rg9w5"] Jul 19 15:36:46.422363 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:36:46.422282 2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-5e6749-openshift-default-78f76d74cf-rg9w5"] Jul 19 15:36:47.103623 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:36:47.103583 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5c4f7c-5478847ccb-8d95h" event={"ID":"7c3a4e60-0549-425e-b089-579ded99578e","Type":"ContainerStarted","Data":"8ea8ce1d3866e1fc08fe768c3c4571b0c3bd04663a39618dcb7ca32579aff062"} Jul 19 15:36:47.103800 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:36:47.103660 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5c4f7c-5478847ccb-8d95h" Jul 19 15:36:47.121388 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:36:47.121341 2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5c4f7c-5478847ccb-8d95h" podStartSLOduration=1.706216064 podStartE2EDuration="2.121324787s" totalImagesPullingTime="415.108723ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-19 15:36:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-19 15:36:47.119145901 +0000 UTC m=+2348.040848635" watchObservedRunningTime="2026-07-19 15:36:47.121324787 +0000 UTC m=+2348.043027500" Jul 19 15:36:47.565088 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:36:47.565055 2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4e8305f3-2721-45bb-bb52-53991a5a49e1" path="/var/lib/kubelet/pods/4e8305f3-2721-45bb-bb52-53991a5a49e1/volumes" Jul 19 15:36:50.362423 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:36:50.362389 2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-b32976-openshift-default-658f97989f-g22cf"] Jul 19 15:36:50.362896 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:36:50.362713 2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4e8305f3-2721-45bb-bb52-53991a5a49e1" containerName="istio-proxy" Jul 19 15:36:50.362896 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:36:50.362725 2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="4e8305f3-2721-45bb-bb52-53991a5a49e1" containerName="istio-proxy" Jul 19 15:36:50.362896 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:36:50.362736 2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4e8305f3-2721-45bb-bb52-53991a5a49e1" containerName="istio-proxy" Jul 19 15:36:50.362896 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:36:50.362741 2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="4e8305f3-2721-45bb-bb52-53991a5a49e1" containerName="istio-proxy" Jul 19 15:36:50.362896 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:36:50.362787 2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="4e8305f3-2721-45bb-bb52-53991a5a49e1" containerName="istio-proxy" Jul 19 15:36:50.362896 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:36:50.362795 2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="4e8305f3-2721-45bb-bb52-53991a5a49e1" containerName="istio-proxy" Jul 19 15:36:50.365978 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:36:50.365953 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-b32976-openshift-default-658f97989f-g22cf" Jul 19 15:36:50.368066 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:36:50.368017 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-b32976-openshift-default-dockercfg-bdw5j\"" Jul 19 15:36:50.376286 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:36:50.376263 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-b32976-openshift-default-658f97989f-g22cf"] Jul 19 15:36:50.460573 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:36:50.460541 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/23584917-2aa2-4c99-a030-1e42c0808d7d-istio-token\") pod \"e2e-api-b-b32976-openshift-default-658f97989f-g22cf\" (UID: \"23584917-2aa2-4c99-a030-1e42c0808d7d\") " pod="openshift-ingress/e2e-api-b-b32976-openshift-default-658f97989f-g22cf" Jul 19 15:36:50.460573 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:36:50.460585 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/23584917-2aa2-4c99-a030-1e42c0808d7d-credential-socket\") pod \"e2e-api-b-b32976-openshift-default-658f97989f-g22cf\" (UID: \"23584917-2aa2-4c99-a030-1e42c0808d7d\") " pod="openshift-ingress/e2e-api-b-b32976-openshift-default-658f97989f-g22cf" Jul 19 15:36:50.460790 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:36:50.460608 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/23584917-2aa2-4c99-a030-1e42c0808d7d-istio-envoy\") pod \"e2e-api-b-b32976-openshift-default-658f97989f-g22cf\" (UID: \"23584917-2aa2-4c99-a030-1e42c0808d7d\") " pod="openshift-ingress/e2e-api-b-b32976-openshift-default-658f97989f-g22cf" Jul 19 15:36:50.460790 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:36:50.460630 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/23584917-2aa2-4c99-a030-1e42c0808d7d-istiod-ca-cert\") pod \"e2e-api-b-b32976-openshift-default-658f97989f-g22cf\" (UID: \"23584917-2aa2-4c99-a030-1e42c0808d7d\") " pod="openshift-ingress/e2e-api-b-b32976-openshift-default-658f97989f-g22cf" Jul 19 15:36:50.460790 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:36:50.460692 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/23584917-2aa2-4c99-a030-1e42c0808d7d-workload-certs\") pod \"e2e-api-b-b32976-openshift-default-658f97989f-g22cf\" (UID: \"23584917-2aa2-4c99-a030-1e42c0808d7d\") " pod="openshift-ingress/e2e-api-b-b32976-openshift-default-658f97989f-g22cf" Jul 19 15:36:50.460790 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:36:50.460732 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/23584917-2aa2-4c99-a030-1e42c0808d7d-workload-socket\") pod \"e2e-api-b-b32976-openshift-default-658f97989f-g22cf\" (UID: \"23584917-2aa2-4c99-a030-1e42c0808d7d\") " pod="openshift-ingress/e2e-api-b-b32976-openshift-default-658f97989f-g22cf" Jul 19 15:36:50.460790 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:36:50.460770 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/23584917-2aa2-4c99-a030-1e42c0808d7d-istio-podinfo\") pod \"e2e-api-b-b32976-openshift-default-658f97989f-g22cf\" (UID: \"23584917-2aa2-4c99-a030-1e42c0808d7d\") " pod="openshift-ingress/e2e-api-b-b32976-openshift-default-658f97989f-g22cf" Jul 19 15:36:50.460963 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:36:50.460808 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xrng9\" (UniqueName: \"kubernetes.io/projected/23584917-2aa2-4c99-a030-1e42c0808d7d-kube-api-access-xrng9\") pod \"e2e-api-b-b32976-openshift-default-658f97989f-g22cf\" (UID: \"23584917-2aa2-4c99-a030-1e42c0808d7d\") " pod="openshift-ingress/e2e-api-b-b32976-openshift-default-658f97989f-g22cf" Jul 19 15:36:50.460963 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:36:50.460838 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/23584917-2aa2-4c99-a030-1e42c0808d7d-istio-data\") pod \"e2e-api-b-b32976-openshift-default-658f97989f-g22cf\" (UID: \"23584917-2aa2-4c99-a030-1e42c0808d7d\") " pod="openshift-ingress/e2e-api-b-b32976-openshift-default-658f97989f-g22cf" Jul 19 15:36:50.562176 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:36:50.562150 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/23584917-2aa2-4c99-a030-1e42c0808d7d-istio-podinfo\") pod \"e2e-api-b-b32976-openshift-default-658f97989f-g22cf\" (UID: \"23584917-2aa2-4c99-a030-1e42c0808d7d\") " pod="openshift-ingress/e2e-api-b-b32976-openshift-default-658f97989f-g22cf" Jul 19 15:36:50.562341 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:36:50.562207 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xrng9\" (UniqueName: \"kubernetes.io/projected/23584917-2aa2-4c99-a030-1e42c0808d7d-kube-api-access-xrng9\") pod \"e2e-api-b-b32976-openshift-default-658f97989f-g22cf\" (UID: \"23584917-2aa2-4c99-a030-1e42c0808d7d\") " pod="openshift-ingress/e2e-api-b-b32976-openshift-default-658f97989f-g22cf" Jul 19 15:36:50.562341 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:36:50.562239 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/23584917-2aa2-4c99-a030-1e42c0808d7d-istio-data\") pod \"e2e-api-b-b32976-openshift-default-658f97989f-g22cf\" (UID: \"23584917-2aa2-4c99-a030-1e42c0808d7d\") " pod="openshift-ingress/e2e-api-b-b32976-openshift-default-658f97989f-g22cf" Jul 19 15:36:50.562341 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:36:50.562270 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/23584917-2aa2-4c99-a030-1e42c0808d7d-istio-token\") pod \"e2e-api-b-b32976-openshift-default-658f97989f-g22cf\" (UID: \"23584917-2aa2-4c99-a030-1e42c0808d7d\") " pod="openshift-ingress/e2e-api-b-b32976-openshift-default-658f97989f-g22cf" Jul 19 15:36:50.562341 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:36:50.562316 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/23584917-2aa2-4c99-a030-1e42c0808d7d-credential-socket\") pod \"e2e-api-b-b32976-openshift-default-658f97989f-g22cf\" (UID: \"23584917-2aa2-4c99-a030-1e42c0808d7d\") " pod="openshift-ingress/e2e-api-b-b32976-openshift-default-658f97989f-g22cf" Jul 19 15:36:50.562556 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:36:50.562337 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/23584917-2aa2-4c99-a030-1e42c0808d7d-istio-envoy\") pod \"e2e-api-b-b32976-openshift-default-658f97989f-g22cf\" (UID: \"23584917-2aa2-4c99-a030-1e42c0808d7d\") " pod="openshift-ingress/e2e-api-b-b32976-openshift-default-658f97989f-g22cf" Jul 19 15:36:50.562556 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:36:50.562379 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/23584917-2aa2-4c99-a030-1e42c0808d7d-istiod-ca-cert\") pod \"e2e-api-b-b32976-openshift-default-658f97989f-g22cf\" (UID: \"23584917-2aa2-4c99-a030-1e42c0808d7d\") " pod="openshift-ingress/e2e-api-b-b32976-openshift-default-658f97989f-g22cf" Jul 19 15:36:50.562742 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:36:50.562718 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/23584917-2aa2-4c99-a030-1e42c0808d7d-workload-certs\") pod \"e2e-api-b-b32976-openshift-default-658f97989f-g22cf\" (UID: \"23584917-2aa2-4c99-a030-1e42c0808d7d\") " pod="openshift-ingress/e2e-api-b-b32976-openshift-default-658f97989f-g22cf" Jul 19 15:36:50.562881 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:36:50.562866 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/23584917-2aa2-4c99-a030-1e42c0808d7d-workload-socket\") pod \"e2e-api-b-b32976-openshift-default-658f97989f-g22cf\" (UID: \"23584917-2aa2-4c99-a030-1e42c0808d7d\") " pod="openshift-ingress/e2e-api-b-b32976-openshift-default-658f97989f-g22cf" Jul 19 15:36:50.563243 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:36:50.563157 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/23584917-2aa2-4c99-a030-1e42c0808d7d-workload-certs\") pod \"e2e-api-b-b32976-openshift-default-658f97989f-g22cf\" (UID: \"23584917-2aa2-4c99-a030-1e42c0808d7d\") " pod="openshift-ingress/e2e-api-b-b32976-openshift-default-658f97989f-g22cf" Jul 19 15:36:50.563243 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:36:50.562774 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/23584917-2aa2-4c99-a030-1e42c0808d7d-credential-socket\") pod \"e2e-api-b-b32976-openshift-default-658f97989f-g22cf\" (UID: \"23584917-2aa2-4c99-a030-1e42c0808d7d\") " pod="openshift-ingress/e2e-api-b-b32976-openshift-default-658f97989f-g22cf" Jul 19 15:36:50.563243 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:36:50.563275 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/23584917-2aa2-4c99-a030-1e42c0808d7d-istiod-ca-cert\") pod \"e2e-api-b-b32976-openshift-default-658f97989f-g22cf\" (UID: \"23584917-2aa2-4c99-a030-1e42c0808d7d\") " pod="openshift-ingress/e2e-api-b-b32976-openshift-default-658f97989f-g22cf" Jul 19 15:36:50.563518 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:36:50.563410 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/23584917-2aa2-4c99-a030-1e42c0808d7d-istio-data\") pod \"e2e-api-b-b32976-openshift-default-658f97989f-g22cf\" (UID: \"23584917-2aa2-4c99-a030-1e42c0808d7d\") " pod="openshift-ingress/e2e-api-b-b32976-openshift-default-658f97989f-g22cf" Jul 19 15:36:50.563768 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:36:50.563742 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/23584917-2aa2-4c99-a030-1e42c0808d7d-workload-socket\") pod \"e2e-api-b-b32976-openshift-default-658f97989f-g22cf\" (UID: \"23584917-2aa2-4c99-a030-1e42c0808d7d\") " pod="openshift-ingress/e2e-api-b-b32976-openshift-default-658f97989f-g22cf" Jul 19 15:36:50.564805 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:36:50.564785 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/23584917-2aa2-4c99-a030-1e42c0808d7d-istio-podinfo\") pod \"e2e-api-b-b32976-openshift-default-658f97989f-g22cf\" (UID: \"23584917-2aa2-4c99-a030-1e42c0808d7d\") " pod="openshift-ingress/e2e-api-b-b32976-openshift-default-658f97989f-g22cf" Jul 19 15:36:50.564938 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:36:50.564913 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/23584917-2aa2-4c99-a030-1e42c0808d7d-istio-envoy\") pod \"e2e-api-b-b32976-openshift-default-658f97989f-g22cf\" (UID: \"23584917-2aa2-4c99-a030-1e42c0808d7d\") " pod="openshift-ingress/e2e-api-b-b32976-openshift-default-658f97989f-g22cf" Jul 19 15:36:50.569868 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:36:50.569836 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/23584917-2aa2-4c99-a030-1e42c0808d7d-istio-token\") pod \"e2e-api-b-b32976-openshift-default-658f97989f-g22cf\" (UID: \"23584917-2aa2-4c99-a030-1e42c0808d7d\") " pod="openshift-ingress/e2e-api-b-b32976-openshift-default-658f97989f-g22cf" Jul 19 15:36:50.569954 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:36:50.569904 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xrng9\" (UniqueName: \"kubernetes.io/projected/23584917-2aa2-4c99-a030-1e42c0808d7d-kube-api-access-xrng9\") pod \"e2e-api-b-b32976-openshift-default-658f97989f-g22cf\" (UID: \"23584917-2aa2-4c99-a030-1e42c0808d7d\") " pod="openshift-ingress/e2e-api-b-b32976-openshift-default-658f97989f-g22cf" Jul 19 15:36:50.676158 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:36:50.676138 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-b32976-openshift-default-658f97989f-g22cf" Jul 19 15:36:50.784915 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:36:50.784884 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-d4lll"] Jul 19 15:36:50.838195 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:36:50.838171 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-b32976-openshift-default-658f97989f-g22cf"] Jul 19 15:36:50.840411 ip-10-0-138-220 kubenswrapper[2558]: W0719 15:36:50.840388 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod23584917_2aa2_4c99_a030_1e42c0808d7d.slice/crio-e6624ed84304979b22e23120302773a5e3974eb33e782574db6e8b92e46c6162 WatchSource:0}: Error finding container e6624ed84304979b22e23120302773a5e3974eb33e782574db6e8b92e46c6162: Status 404 returned error can't find the container with id e6624ed84304979b22e23120302773a5e3974eb33e782574db6e8b92e46c6162 Jul 19 15:36:50.842667 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:36:50.842636 2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 19 15:36:50.842768 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:36:50.842715 2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 19 15:36:50.842768 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:36:50.842763 2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 19 15:36:51.121833 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:36:51.121743 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-b32976-openshift-default-658f97989f-g22cf" event={"ID":"23584917-2aa2-4c99-a030-1e42c0808d7d","Type":"ContainerStarted","Data":"e76279d2ba3187ebb6b3a88bb7381086ffcb7c7af104c724f3dbb07c4dfa77a7"} Jul 19 15:36:51.121833 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:36:51.121801 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-b32976-openshift-default-658f97989f-g22cf" event={"ID":"23584917-2aa2-4c99-a030-1e42c0808d7d","Type":"ContainerStarted","Data":"e6624ed84304979b22e23120302773a5e3974eb33e782574db6e8b92e46c6162"} Jul 19 15:36:51.143429 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:36:51.143385 2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-b32976-openshift-default-658f97989f-g22cf" podStartSLOduration=1.143369027 podStartE2EDuration="1.143369027s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-19 15:36:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-19 15:36:51.141920433 +0000 UTC m=+2352.063623150" watchObservedRunningTime="2026-07-19 15:36:51.143369027 +0000 UTC m=+2352.065071744" Jul 19 15:36:51.676768 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:36:51.676722 2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-b32976-openshift-default-658f97989f-g22cf" Jul 19 15:36:51.681844 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:36:51.681816 2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-b32976-openshift-default-658f97989f-g22cf" Jul 19 15:36:52.127134 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:36:52.127053 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-b32976-openshift-default-658f97989f-g22cf" Jul 19 15:36:52.128017 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:36:52.127997 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-b32976-openshift-default-658f97989f-g22cf" Jul 19 15:36:53.112865 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:36:53.112832 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5c4f7c-5478847ccb-8d95h" Jul 19 15:36:59.085474 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:36:59.085386 2558 patch_prober.go:28] interesting pod/e2e-api-a-5c4f7c-openshift-default-69455b5bf5-p47m2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" start-of-body= Jul 19 15:36:59.085474 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:36:59.085456 2558 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-5c4f7c-openshift-default-69455b5bf5-p47m2" podUID="81f9a2e7-69db-4eaa-9f5f-2909613cd9ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" Jul 19 15:36:59.163742 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:36:59.163707 2558 generic.go:356] "Generic (PLEG): container finished" podID="81f9a2e7-69db-4eaa-9f5f-2909613cd9ee" containerID="1c59cb8899a7b13b6181e4e207211cffab09154cd47600fde0b5fd8db150e76b" exitCode=0 Jul 19 15:36:59.163989 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:36:59.163782 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-5c4f7c-openshift-default-69455b5bf5-p47m2" event={"ID":"81f9a2e7-69db-4eaa-9f5f-2909613cd9ee","Type":"ContainerDied","Data":"1c59cb8899a7b13b6181e4e207211cffab09154cd47600fde0b5fd8db150e76b"} Jul 19 15:36:59.164623 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:36:59.164606 2558 scope.go:117] "RemoveContainer" containerID="1c59cb8899a7b13b6181e4e207211cffab09154cd47600fde0b5fd8db150e76b" Jul 19 15:36:59.166590 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:36:59.166549 2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 19 15:36:59.166706 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:36:59.166652 2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 19 15:36:59.166706 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:36:59.166699 2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 19 15:37:00.173011 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:37:00.172960 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-5c4f7c-openshift-default-69455b5bf5-p47m2" event={"ID":"81f9a2e7-69db-4eaa-9f5f-2909613cd9ee","Type":"ContainerStarted","Data":"c0d9b4718c4b1e485c923ca8eaf80a4733959ab28fab1b6aacbe2e6426f2c45a"} Jul 19 15:37:01.074280 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:37:01.074246 2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-5c4f7c-openshift-default-69455b5bf5-p47m2" Jul 19 15:37:01.074758 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:37:01.074734 2558 patch_prober.go:28] interesting pod/e2e-api-a-5c4f7c-openshift-default-69455b5bf5-p47m2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" start-of-body= Jul 19 15:37:01.074922 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:37:01.074898 2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-5c4f7c-openshift-default-69455b5bf5-p47m2" podUID="81f9a2e7-69db-4eaa-9f5f-2909613cd9ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" Jul 19 15:37:02.074296 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:37:02.074248 2558 patch_prober.go:28] interesting pod/e2e-api-a-5c4f7c-openshift-default-69455b5bf5-p47m2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" start-of-body= Jul 19 15:37:02.074825 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:37:02.074316 2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-5c4f7c-openshift-default-69455b5bf5-p47m2" podUID="81f9a2e7-69db-4eaa-9f5f-2909613cd9ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" Jul 19 15:37:03.073684 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:37:03.073647 2558 patch_prober.go:28] interesting pod/e2e-api-a-5c4f7c-openshift-default-69455b5bf5-p47m2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" start-of-body= Jul 19 15:37:03.073865 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:37:03.073714 2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-5c4f7c-openshift-default-69455b5bf5-p47m2" podUID="81f9a2e7-69db-4eaa-9f5f-2909613cd9ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" Jul 19 15:37:03.985006 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:37:03.984969 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-d4lll"] Jul 19 15:37:04.073633 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:37:04.073593 2558 patch_prober.go:28] interesting pod/e2e-api-a-5c4f7c-openshift-default-69455b5bf5-p47m2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" start-of-body= Jul 19 15:37:04.073812 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:37:04.073651 2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-5c4f7c-openshift-default-69455b5bf5-p47m2" podUID="81f9a2e7-69db-4eaa-9f5f-2909613cd9ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" Jul 19 15:37:05.076743 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:37:05.076654 2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-5c4f7c-openshift-default-69455b5bf5-p47m2" Jul 19 15:37:05.077243 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:37:05.077054 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-5c4f7c-openshift-default-69455b5bf5-p47m2" Jul 19 15:37:05.077768 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:37:05.077753 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-5c4f7c-openshift-default-69455b5bf5-p47m2" Jul 19 15:37:05.200069 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:37:05.200015 2558 generic.go:356] "Generic (PLEG): container finished" podID="23584917-2aa2-4c99-a030-1e42c0808d7d" containerID="e76279d2ba3187ebb6b3a88bb7381086ffcb7c7af104c724f3dbb07c4dfa77a7" exitCode=0 Jul 19 15:37:05.200069 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:37:05.200054 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-b32976-openshift-default-658f97989f-g22cf" event={"ID":"23584917-2aa2-4c99-a030-1e42c0808d7d","Type":"ContainerDied","Data":"e76279d2ba3187ebb6b3a88bb7381086ffcb7c7af104c724f3dbb07c4dfa77a7"} Jul 19 15:37:05.200498 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:37:05.200480 2558 scope.go:117] "RemoveContainer" containerID="e76279d2ba3187ebb6b3a88bb7381086ffcb7c7af104c724f3dbb07c4dfa77a7" Jul 19 15:37:05.201528 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:37:05.201502 2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 19 15:37:05.201604 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:37:05.201586 2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 19 15:37:05.201643 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:37:05.201632 2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 19 15:37:06.207929 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:37:06.207882 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-b32976-openshift-default-658f97989f-g22cf" event={"ID":"23584917-2aa2-4c99-a030-1e42c0808d7d","Type":"ContainerStarted","Data":"2edbab0d70dc45561d5d3e1c52ca14ac80e069bdf558112e689e2f68ca6ce207"} Jul 19 15:37:06.676594 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:37:06.676554 2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-b32976-openshift-default-658f97989f-g22cf" Jul 19 15:37:06.677731 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:37:06.677705 2558 patch_prober.go:28] interesting pod/e2e-api-b-b32976-openshift-default-658f97989f-g22cf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" start-of-body= Jul 19 15:37:06.677883 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:37:06.677774 2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-b32976-openshift-default-658f97989f-g22cf" podUID="23584917-2aa2-4c99-a030-1e42c0808d7d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" Jul 19 15:37:07.127930 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:37:07.127847 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-b32976-openshift-default-658f97989f-g22cf" Jul 19 15:37:07.677227 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:37:07.677188 2558 patch_prober.go:28] interesting pod/e2e-api-b-b32976-openshift-default-658f97989f-g22cf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" start-of-body= Jul 19 15:37:07.677625 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:37:07.677261 2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-b32976-openshift-default-658f97989f-g22cf" podUID="23584917-2aa2-4c99-a030-1e42c0808d7d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" Jul 19 15:37:08.676909 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:37:08.676864 2558 patch_prober.go:28] interesting pod/e2e-api-b-b32976-openshift-default-658f97989f-g22cf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" start-of-body= Jul 19 15:37:08.677119 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:37:08.676938 2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-b32976-openshift-default-658f97989f-g22cf" podUID="23584917-2aa2-4c99-a030-1e42c0808d7d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" Jul 19 15:37:09.676782 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:37:09.676747 2558 patch_prober.go:28] interesting pod/e2e-api-b-b32976-openshift-default-658f97989f-g22cf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" start-of-body= Jul 19 15:37:09.677180 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:37:09.676802 2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-b32976-openshift-default-658f97989f-g22cf" podUID="23584917-2aa2-4c99-a030-1e42c0808d7d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" Jul 19 15:37:10.680009 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:37:10.679979 2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-b32976-openshift-default-658f97989f-g22cf" Jul 19 15:37:10.680961 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:37:10.680943 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-b32976-openshift-default-658f97989f-g22cf" Jul 19 15:37:20.782475 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:37:20.782432 2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-5c4f7c-5478847ccb-8d95h"] Jul 19 15:37:20.782977 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:37:20.782716 2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5c4f7c-5478847ccb-8d95h" podUID="7c3a4e60-0549-425e-b089-579ded99578e" containerName="maas-api" containerID="cri-o://8ea8ce1d3866e1fc08fe768c3c4571b0c3bd04663a39618dcb7ca32579aff062" gracePeriod=30 Jul 19 15:37:21.030439 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:37:21.030408 2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5c4f7c-5478847ccb-8d95h" Jul 19 15:37:21.119384 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:37:21.119317 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7c3a4e60-0549-425e-b089-579ded99578e-maas-api-tls\") pod \"7c3a4e60-0549-425e-b089-579ded99578e\" (UID: \"7c3a4e60-0549-425e-b089-579ded99578e\") " Jul 19 15:37:21.119384 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:37:21.119355 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wvddq\" (UniqueName: \"kubernetes.io/projected/7c3a4e60-0549-425e-b089-579ded99578e-kube-api-access-wvddq\") pod \"7c3a4e60-0549-425e-b089-579ded99578e\" (UID: \"7c3a4e60-0549-425e-b089-579ded99578e\") " Jul 19 15:37:21.119553 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:37:21.119478 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7c3a4e60-0549-425e-b089-579ded99578e-openshift-service-ca\") pod \"7c3a4e60-0549-425e-b089-579ded99578e\" (UID: \"7c3a4e60-0549-425e-b089-579ded99578e\") " Jul 19 15:37:21.119781 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:37:21.119753 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7c3a4e60-0549-425e-b089-579ded99578e-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "7c3a4e60-0549-425e-b089-579ded99578e" (UID: "7c3a4e60-0549-425e-b089-579ded99578e"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 19 15:37:21.121530 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:37:21.121495 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7c3a4e60-0549-425e-b089-579ded99578e-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "7c3a4e60-0549-425e-b089-579ded99578e" (UID: "7c3a4e60-0549-425e-b089-579ded99578e"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 19 15:37:21.121600 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:37:21.121577 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7c3a4e60-0549-425e-b089-579ded99578e-kube-api-access-wvddq" (OuterVolumeSpecName: "kube-api-access-wvddq") pod "7c3a4e60-0549-425e-b089-579ded99578e" (UID: "7c3a4e60-0549-425e-b089-579ded99578e"). InnerVolumeSpecName "kube-api-access-wvddq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 19 15:37:21.220631 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:37:21.220604 2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wvddq\" (UniqueName: \"kubernetes.io/projected/7c3a4e60-0549-425e-b089-579ded99578e-kube-api-access-wvddq\") on node \"ip-10-0-138-220.ec2.internal\" DevicePath \"\"" Jul 19 15:37:21.220631 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:37:21.220629 2558 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7c3a4e60-0549-425e-b089-579ded99578e-openshift-service-ca\") on node \"ip-10-0-138-220.ec2.internal\" DevicePath \"\"" Jul 19 15:37:21.220771 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:37:21.220642 2558 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7c3a4e60-0549-425e-b089-579ded99578e-maas-api-tls\") on node \"ip-10-0-138-220.ec2.internal\" DevicePath \"\"" Jul 19 15:37:21.275772 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:37:21.275743 2558 generic.go:356] "Generic (PLEG): container finished" podID="7c3a4e60-0549-425e-b089-579ded99578e" containerID="8ea8ce1d3866e1fc08fe768c3c4571b0c3bd04663a39618dcb7ca32579aff062" exitCode=0 Jul 19 15:37:21.275890 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:37:21.275806 2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5c4f7c-5478847ccb-8d95h" Jul 19 15:37:21.275890 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:37:21.275814 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5c4f7c-5478847ccb-8d95h" event={"ID":"7c3a4e60-0549-425e-b089-579ded99578e","Type":"ContainerDied","Data":"8ea8ce1d3866e1fc08fe768c3c4571b0c3bd04663a39618dcb7ca32579aff062"} Jul 19 15:37:21.275890 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:37:21.275857 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5c4f7c-5478847ccb-8d95h" event={"ID":"7c3a4e60-0549-425e-b089-579ded99578e","Type":"ContainerDied","Data":"44203775ea938d3e9ce4f8567c9584ab3f0d0030f51cc58389ab32010146c000"} Jul 19 15:37:21.275890 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:37:21.275878 2558 scope.go:117] "RemoveContainer" containerID="8ea8ce1d3866e1fc08fe768c3c4571b0c3bd04663a39618dcb7ca32579aff062" Jul 19 15:37:21.283963 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:37:21.283946 2558 scope.go:117] "RemoveContainer" containerID="8ea8ce1d3866e1fc08fe768c3c4571b0c3bd04663a39618dcb7ca32579aff062" Jul 19 15:37:21.284229 ip-10-0-138-220 kubenswrapper[2558]: E0719 15:37:21.284212 2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8ea8ce1d3866e1fc08fe768c3c4571b0c3bd04663a39618dcb7ca32579aff062\": container with ID starting with 8ea8ce1d3866e1fc08fe768c3c4571b0c3bd04663a39618dcb7ca32579aff062 not found: ID does not exist" containerID="8ea8ce1d3866e1fc08fe768c3c4571b0c3bd04663a39618dcb7ca32579aff062" Jul 19 15:37:21.284305 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:37:21.284236 2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8ea8ce1d3866e1fc08fe768c3c4571b0c3bd04663a39618dcb7ca32579aff062"} err="failed to get container status \"8ea8ce1d3866e1fc08fe768c3c4571b0c3bd04663a39618dcb7ca32579aff062\": rpc error: code = NotFound desc = could not find container \"8ea8ce1d3866e1fc08fe768c3c4571b0c3bd04663a39618dcb7ca32579aff062\": container with ID starting with 8ea8ce1d3866e1fc08fe768c3c4571b0c3bd04663a39618dcb7ca32579aff062 not found: ID does not exist" Jul 19 15:37:21.296636 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:37:21.296605 2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-5c4f7c-5478847ccb-8d95h"] Jul 19 15:37:21.298748 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:37:21.298726 2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-5c4f7c-5478847ccb-8d95h"] Jul 19 15:37:21.564402 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:37:21.564374 2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7c3a4e60-0549-425e-b089-579ded99578e" path="/var/lib/kubelet/pods/7c3a4e60-0549-425e-b089-579ded99578e/volumes" Jul 19 15:37:26.990823 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:37:26.990777 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-d4lll"] Jul 19 15:37:31.903058 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:37:31.903004 2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-5c4f7c-openshift-default-69455b5bf5-p47m2"] Jul 19 15:37:31.903688 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:37:31.903267 2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-5c4f7c-openshift-default-69455b5bf5-p47m2" podUID="81f9a2e7-69db-4eaa-9f5f-2909613cd9ee" containerName="istio-proxy" containerID="cri-o://c0d9b4718c4b1e485c923ca8eaf80a4733959ab28fab1b6aacbe2e6426f2c45a" gracePeriod=30 Jul 19 15:37:35.079175 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:37:35.079140 2558 patch_prober.go:28] interesting pod/e2e-api-a-5c4f7c-openshift-default-69455b5bf5-p47m2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 19 15:37:35.079539 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:37:35.079203 2558 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-5c4f7c-openshift-default-69455b5bf5-p47m2" podUID="81f9a2e7-69db-4eaa-9f5f-2909613cd9ee" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 19 15:37:37.155342 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:37:37.155319 2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-5c4f7c-openshift-default-69455b5bf5-p47m2" Jul 19 15:37:37.238887 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:37:37.238813 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/81f9a2e7-69db-4eaa-9f5f-2909613cd9ee-istio-podinfo\") pod \"81f9a2e7-69db-4eaa-9f5f-2909613cd9ee\" (UID: \"81f9a2e7-69db-4eaa-9f5f-2909613cd9ee\") " Jul 19 15:37:37.238887 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:37:37.238847 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-h2s4v\" (UniqueName: \"kubernetes.io/projected/81f9a2e7-69db-4eaa-9f5f-2909613cd9ee-kube-api-access-h2s4v\") pod \"81f9a2e7-69db-4eaa-9f5f-2909613cd9ee\" (UID: \"81f9a2e7-69db-4eaa-9f5f-2909613cd9ee\") " Jul 19 15:37:37.238887 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:37:37.238867 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/81f9a2e7-69db-4eaa-9f5f-2909613cd9ee-credential-socket\") pod \"81f9a2e7-69db-4eaa-9f5f-2909613cd9ee\" (UID: \"81f9a2e7-69db-4eaa-9f5f-2909613cd9ee\") " Jul 19 15:37:37.239177 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:37:37.238897 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/81f9a2e7-69db-4eaa-9f5f-2909613cd9ee-istiod-ca-cert\") pod \"81f9a2e7-69db-4eaa-9f5f-2909613cd9ee\" (UID: \"81f9a2e7-69db-4eaa-9f5f-2909613cd9ee\") " Jul 19 15:37:37.239177 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:37:37.238951 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/81f9a2e7-69db-4eaa-9f5f-2909613cd9ee-workload-certs\") pod \"81f9a2e7-69db-4eaa-9f5f-2909613cd9ee\" (UID: \"81f9a2e7-69db-4eaa-9f5f-2909613cd9ee\") " Jul 19 15:37:37.239177 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:37:37.238976 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/81f9a2e7-69db-4eaa-9f5f-2909613cd9ee-istio-token\") pod \"81f9a2e7-69db-4eaa-9f5f-2909613cd9ee\" (UID: \"81f9a2e7-69db-4eaa-9f5f-2909613cd9ee\") " Jul 19 15:37:37.239177 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:37:37.239008 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/81f9a2e7-69db-4eaa-9f5f-2909613cd9ee-workload-socket\") pod \"81f9a2e7-69db-4eaa-9f5f-2909613cd9ee\" (UID: \"81f9a2e7-69db-4eaa-9f5f-2909613cd9ee\") " Jul 19 15:37:37.239177 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:37:37.239078 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/81f9a2e7-69db-4eaa-9f5f-2909613cd9ee-istio-data\") pod \"81f9a2e7-69db-4eaa-9f5f-2909613cd9ee\" (UID: \"81f9a2e7-69db-4eaa-9f5f-2909613cd9ee\") " Jul 19 15:37:37.239177 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:37:37.239121 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/81f9a2e7-69db-4eaa-9f5f-2909613cd9ee-istio-envoy\") pod \"81f9a2e7-69db-4eaa-9f5f-2909613cd9ee\" (UID: \"81f9a2e7-69db-4eaa-9f5f-2909613cd9ee\") " Jul 19 15:37:37.239491 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:37:37.239212 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/81f9a2e7-69db-4eaa-9f5f-2909613cd9ee-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "81f9a2e7-69db-4eaa-9f5f-2909613cd9ee" (UID: "81f9a2e7-69db-4eaa-9f5f-2909613cd9ee"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 19 15:37:37.239491 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:37:37.239352 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/81f9a2e7-69db-4eaa-9f5f-2909613cd9ee-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "81f9a2e7-69db-4eaa-9f5f-2909613cd9ee" (UID: "81f9a2e7-69db-4eaa-9f5f-2909613cd9ee"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 19 15:37:37.239491 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:37:37.239383 2558 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/81f9a2e7-69db-4eaa-9f5f-2909613cd9ee-credential-socket\") on node \"ip-10-0-138-220.ec2.internal\" DevicePath \"\"" Jul 19 15:37:37.239491 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:37:37.239439 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/81f9a2e7-69db-4eaa-9f5f-2909613cd9ee-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "81f9a2e7-69db-4eaa-9f5f-2909613cd9ee" (UID: "81f9a2e7-69db-4eaa-9f5f-2909613cd9ee"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 19 15:37:37.239876 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:37:37.239671 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/81f9a2e7-69db-4eaa-9f5f-2909613cd9ee-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "81f9a2e7-69db-4eaa-9f5f-2909613cd9ee" (UID: "81f9a2e7-69db-4eaa-9f5f-2909613cd9ee"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 19 15:37:37.240122 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:37:37.240102 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/81f9a2e7-69db-4eaa-9f5f-2909613cd9ee-istio-data" (OuterVolumeSpecName: "istio-data") pod "81f9a2e7-69db-4eaa-9f5f-2909613cd9ee" (UID: "81f9a2e7-69db-4eaa-9f5f-2909613cd9ee"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 19 15:37:37.241289 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:37:37.241266 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/81f9a2e7-69db-4eaa-9f5f-2909613cd9ee-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "81f9a2e7-69db-4eaa-9f5f-2909613cd9ee" (UID: "81f9a2e7-69db-4eaa-9f5f-2909613cd9ee"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 19 15:37:37.241388 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:37:37.241292 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/81f9a2e7-69db-4eaa-9f5f-2909613cd9ee-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "81f9a2e7-69db-4eaa-9f5f-2909613cd9ee" (UID: "81f9a2e7-69db-4eaa-9f5f-2909613cd9ee"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 19 15:37:37.241658 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:37:37.241639 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/81f9a2e7-69db-4eaa-9f5f-2909613cd9ee-kube-api-access-h2s4v" (OuterVolumeSpecName: "kube-api-access-h2s4v") pod "81f9a2e7-69db-4eaa-9f5f-2909613cd9ee" (UID: "81f9a2e7-69db-4eaa-9f5f-2909613cd9ee"). InnerVolumeSpecName "kube-api-access-h2s4v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 19 15:37:37.241700 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:37:37.241643 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/81f9a2e7-69db-4eaa-9f5f-2909613cd9ee-istio-token" (OuterVolumeSpecName: "istio-token") pod "81f9a2e7-69db-4eaa-9f5f-2909613cd9ee" (UID: "81f9a2e7-69db-4eaa-9f5f-2909613cd9ee"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 19 15:37:37.339785 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:37:37.339761 2558 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/81f9a2e7-69db-4eaa-9f5f-2909613cd9ee-istio-token\") on node \"ip-10-0-138-220.ec2.internal\" DevicePath \"\"" Jul 19 15:37:37.339785 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:37:37.339780 2558 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/81f9a2e7-69db-4eaa-9f5f-2909613cd9ee-workload-socket\") on node \"ip-10-0-138-220.ec2.internal\" DevicePath \"\"" Jul 19 15:37:37.339785 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:37:37.339789 2558 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/81f9a2e7-69db-4eaa-9f5f-2909613cd9ee-istio-data\") on node \"ip-10-0-138-220.ec2.internal\" DevicePath \"\"" Jul 19 15:37:37.339950 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:37:37.339798 2558 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/81f9a2e7-69db-4eaa-9f5f-2909613cd9ee-istio-envoy\") on node \"ip-10-0-138-220.ec2.internal\" DevicePath \"\"" Jul 19 15:37:37.339950 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:37:37.339805 2558 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/81f9a2e7-69db-4eaa-9f5f-2909613cd9ee-istio-podinfo\") on node \"ip-10-0-138-220.ec2.internal\" DevicePath \"\"" Jul 19 15:37:37.339950 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:37:37.339813 2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h2s4v\" (UniqueName: \"kubernetes.io/projected/81f9a2e7-69db-4eaa-9f5f-2909613cd9ee-kube-api-access-h2s4v\") on node \"ip-10-0-138-220.ec2.internal\" DevicePath \"\"" Jul 19 15:37:37.339950 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:37:37.339822 2558 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/81f9a2e7-69db-4eaa-9f5f-2909613cd9ee-istiod-ca-cert\") on node \"ip-10-0-138-220.ec2.internal\" DevicePath \"\"" Jul 19 15:37:37.339950 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:37:37.339831 2558 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/81f9a2e7-69db-4eaa-9f5f-2909613cd9ee-workload-certs\") on node \"ip-10-0-138-220.ec2.internal\" DevicePath \"\"" Jul 19 15:37:37.343626 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:37:37.343593 2558 generic.go:356] "Generic (PLEG): container finished" podID="81f9a2e7-69db-4eaa-9f5f-2909613cd9ee" containerID="c0d9b4718c4b1e485c923ca8eaf80a4733959ab28fab1b6aacbe2e6426f2c45a" exitCode=0 Jul 19 15:37:37.343718 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:37:37.343643 2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-5c4f7c-openshift-default-69455b5bf5-p47m2" Jul 19 15:37:37.343718 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:37:37.343647 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-5c4f7c-openshift-default-69455b5bf5-p47m2" event={"ID":"81f9a2e7-69db-4eaa-9f5f-2909613cd9ee","Type":"ContainerDied","Data":"c0d9b4718c4b1e485c923ca8eaf80a4733959ab28fab1b6aacbe2e6426f2c45a"} Jul 19 15:37:37.343718 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:37:37.343704 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-5c4f7c-openshift-default-69455b5bf5-p47m2" event={"ID":"81f9a2e7-69db-4eaa-9f5f-2909613cd9ee","Type":"ContainerDied","Data":"512fc0906ecb816d6bd516ca6dec014de71939e4701b61770ea679fcdc5ac5d2"} Jul 19 15:37:37.343877 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:37:37.343734 2558 scope.go:117] "RemoveContainer" containerID="c0d9b4718c4b1e485c923ca8eaf80a4733959ab28fab1b6aacbe2e6426f2c45a" Jul 19 15:37:37.352177 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:37:37.352162 2558 scope.go:117] "RemoveContainer" containerID="1c59cb8899a7b13b6181e4e207211cffab09154cd47600fde0b5fd8db150e76b" Jul 19 15:37:37.359981 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:37:37.359962 2558 scope.go:117] "RemoveContainer" containerID="c0d9b4718c4b1e485c923ca8eaf80a4733959ab28fab1b6aacbe2e6426f2c45a" Jul 19 15:37:37.360281 ip-10-0-138-220 kubenswrapper[2558]: E0719 15:37:37.360261 2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c0d9b4718c4b1e485c923ca8eaf80a4733959ab28fab1b6aacbe2e6426f2c45a\": container with ID starting with c0d9b4718c4b1e485c923ca8eaf80a4733959ab28fab1b6aacbe2e6426f2c45a not found: ID does not exist" containerID="c0d9b4718c4b1e485c923ca8eaf80a4733959ab28fab1b6aacbe2e6426f2c45a" Jul 19 15:37:37.360341 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:37:37.360290 2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c0d9b4718c4b1e485c923ca8eaf80a4733959ab28fab1b6aacbe2e6426f2c45a"} err="failed to get container status \"c0d9b4718c4b1e485c923ca8eaf80a4733959ab28fab1b6aacbe2e6426f2c45a\": rpc error: code = NotFound desc = could not find container \"c0d9b4718c4b1e485c923ca8eaf80a4733959ab28fab1b6aacbe2e6426f2c45a\": container with ID starting with c0d9b4718c4b1e485c923ca8eaf80a4733959ab28fab1b6aacbe2e6426f2c45a not found: ID does not exist" Jul 19 15:37:37.360341 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:37:37.360308 2558 scope.go:117] "RemoveContainer" containerID="1c59cb8899a7b13b6181e4e207211cffab09154cd47600fde0b5fd8db150e76b" Jul 19 15:37:37.360556 ip-10-0-138-220 kubenswrapper[2558]: E0719 15:37:37.360539 2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1c59cb8899a7b13b6181e4e207211cffab09154cd47600fde0b5fd8db150e76b\": container with ID starting with 1c59cb8899a7b13b6181e4e207211cffab09154cd47600fde0b5fd8db150e76b not found: ID does not exist" containerID="1c59cb8899a7b13b6181e4e207211cffab09154cd47600fde0b5fd8db150e76b" Jul 19 15:37:37.360604 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:37:37.360564 2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1c59cb8899a7b13b6181e4e207211cffab09154cd47600fde0b5fd8db150e76b"} err="failed to get container status \"1c59cb8899a7b13b6181e4e207211cffab09154cd47600fde0b5fd8db150e76b\": rpc error: code = NotFound desc = could not find container \"1c59cb8899a7b13b6181e4e207211cffab09154cd47600fde0b5fd8db150e76b\": container with ID starting with 1c59cb8899a7b13b6181e4e207211cffab09154cd47600fde0b5fd8db150e76b not found: ID does not exist" Jul 19 15:37:37.364357 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:37:37.364330 2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-5c4f7c-openshift-default-69455b5bf5-p47m2"] Jul 19 15:37:37.370626 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:37:37.370605 2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-5c4f7c-openshift-default-69455b5bf5-p47m2"] Jul 19 15:37:37.564828 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:37:37.564751 2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="81f9a2e7-69db-4eaa-9f5f-2909613cd9ee" path="/var/lib/kubelet/pods/81f9a2e7-69db-4eaa-9f5f-2909613cd9ee/volumes" Jul 19 15:37:39.668763 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:37:39.668738 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-5wfzs_5b2d3a07-86b2-4ba2-8a71-c2d89ada79c6/ovn-acl-logging/0.log" Jul 19 15:37:39.674577 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:37:39.674558 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-5wfzs_5b2d3a07-86b2-4ba2-8a71-c2d89ada79c6/ovn-acl-logging/0.log" Jul 19 15:37:39.806492 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:37:39.806465 2558 scope.go:117] "RemoveContainer" containerID="f96c3b20c174d13e6dddb72deb2474e73da78dd55e84f45880e57e4e1bb7ce33" Jul 19 15:37:52.486510 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:37:52.486475 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-d4lll"] Jul 19 15:37:58.945388 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:37:58.945352 2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-b32976-openshift-default-658f97989f-g22cf"] Jul 19 15:37:58.945816 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:37:58.945625 2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-b32976-openshift-default-658f97989f-g22cf" podUID="23584917-2aa2-4c99-a030-1e42c0808d7d" containerName="istio-proxy" containerID="cri-o://2edbab0d70dc45561d5d3e1c52ca14ac80e069bdf558112e689e2f68ca6ce207" gracePeriod=30 Jul 19 15:37:59.835579 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:37:59.835546 2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-51b4ec-openshift-default-65f5bd8fc5-7cz2s"] Jul 19 15:37:59.835847 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:37:59.835835 2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="81f9a2e7-69db-4eaa-9f5f-2909613cd9ee" containerName="istio-proxy" Jul 19 15:37:59.835892 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:37:59.835849 2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="81f9a2e7-69db-4eaa-9f5f-2909613cd9ee" containerName="istio-proxy" Jul 19 15:37:59.835892 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:37:59.835857 2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="81f9a2e7-69db-4eaa-9f5f-2909613cd9ee" containerName="istio-proxy" Jul 19 15:37:59.835892 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:37:59.835863 2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="81f9a2e7-69db-4eaa-9f5f-2909613cd9ee" containerName="istio-proxy" Jul 19 15:37:59.835892 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:37:59.835877 2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7c3a4e60-0549-425e-b089-579ded99578e" containerName="maas-api" Jul 19 15:37:59.835892 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:37:59.835882 2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="7c3a4e60-0549-425e-b089-579ded99578e" containerName="maas-api" Jul 19 15:37:59.836064 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:37:59.835932 2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="81f9a2e7-69db-4eaa-9f5f-2909613cd9ee" containerName="istio-proxy" Jul 19 15:37:59.836064 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:37:59.835939 2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="81f9a2e7-69db-4eaa-9f5f-2909613cd9ee" containerName="istio-proxy" Jul 19 15:37:59.836064 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:37:59.835944 2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="7c3a4e60-0549-425e-b089-579ded99578e" containerName="maas-api" Jul 19 15:37:59.838738 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:37:59.838721 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-51b4ec-openshift-default-65f5bd8fc5-7cz2s" Jul 19 15:37:59.840895 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:37:59.840867 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-51b4ec-openshift-default-dockercfg-2w9cm\"" Jul 19 15:37:59.849962 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:37:59.849925 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-51b4ec-openshift-default-65f5bd8fc5-7cz2s"] Jul 19 15:37:59.910803 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:37:59.910776 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/65ec992a-d1ff-4cfb-a12f-514137d3a4c1-istiod-ca-cert\") pod \"e2e-shared-a-51b4ec-openshift-default-65f5bd8fc5-7cz2s\" (UID: \"65ec992a-d1ff-4cfb-a12f-514137d3a4c1\") " pod="openshift-ingress/e2e-shared-a-51b4ec-openshift-default-65f5bd8fc5-7cz2s" Jul 19 15:37:59.910943 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:37:59.910812 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/65ec992a-d1ff-4cfb-a12f-514137d3a4c1-istio-data\") pod \"e2e-shared-a-51b4ec-openshift-default-65f5bd8fc5-7cz2s\" (UID: \"65ec992a-d1ff-4cfb-a12f-514137d3a4c1\") " pod="openshift-ingress/e2e-shared-a-51b4ec-openshift-default-65f5bd8fc5-7cz2s" Jul 19 15:37:59.910943 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:37:59.910843 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/65ec992a-d1ff-4cfb-a12f-514137d3a4c1-istio-podinfo\") pod \"e2e-shared-a-51b4ec-openshift-default-65f5bd8fc5-7cz2s\" (UID: \"65ec992a-d1ff-4cfb-a12f-514137d3a4c1\") " pod="openshift-ingress/e2e-shared-a-51b4ec-openshift-default-65f5bd8fc5-7cz2s" Jul 19 15:37:59.910943 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:37:59.910915 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/65ec992a-d1ff-4cfb-a12f-514137d3a4c1-istio-token\") pod \"e2e-shared-a-51b4ec-openshift-default-65f5bd8fc5-7cz2s\" (UID: \"65ec992a-d1ff-4cfb-a12f-514137d3a4c1\") " pod="openshift-ingress/e2e-shared-a-51b4ec-openshift-default-65f5bd8fc5-7cz2s" Jul 19 15:37:59.911131 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:37:59.910957 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/65ec992a-d1ff-4cfb-a12f-514137d3a4c1-credential-socket\") pod \"e2e-shared-a-51b4ec-openshift-default-65f5bd8fc5-7cz2s\" (UID: \"65ec992a-d1ff-4cfb-a12f-514137d3a4c1\") " pod="openshift-ingress/e2e-shared-a-51b4ec-openshift-default-65f5bd8fc5-7cz2s" Jul 19 15:37:59.911131 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:37:59.910985 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hkw9q\" (UniqueName: \"kubernetes.io/projected/65ec992a-d1ff-4cfb-a12f-514137d3a4c1-kube-api-access-hkw9q\") pod \"e2e-shared-a-51b4ec-openshift-default-65f5bd8fc5-7cz2s\" (UID: \"65ec992a-d1ff-4cfb-a12f-514137d3a4c1\") " pod="openshift-ingress/e2e-shared-a-51b4ec-openshift-default-65f5bd8fc5-7cz2s" Jul 19 15:37:59.911131 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:37:59.911042 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/65ec992a-d1ff-4cfb-a12f-514137d3a4c1-workload-certs\") pod \"e2e-shared-a-51b4ec-openshift-default-65f5bd8fc5-7cz2s\" (UID: \"65ec992a-d1ff-4cfb-a12f-514137d3a4c1\") " pod="openshift-ingress/e2e-shared-a-51b4ec-openshift-default-65f5bd8fc5-7cz2s" Jul 19 15:37:59.911131 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:37:59.911069 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/65ec992a-d1ff-4cfb-a12f-514137d3a4c1-istio-envoy\") pod \"e2e-shared-a-51b4ec-openshift-default-65f5bd8fc5-7cz2s\" (UID: \"65ec992a-d1ff-4cfb-a12f-514137d3a4c1\") " pod="openshift-ingress/e2e-shared-a-51b4ec-openshift-default-65f5bd8fc5-7cz2s" Jul 19 15:37:59.911333 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:37:59.911129 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/65ec992a-d1ff-4cfb-a12f-514137d3a4c1-workload-socket\") pod \"e2e-shared-a-51b4ec-openshift-default-65f5bd8fc5-7cz2s\" (UID: \"65ec992a-d1ff-4cfb-a12f-514137d3a4c1\") " pod="openshift-ingress/e2e-shared-a-51b4ec-openshift-default-65f5bd8fc5-7cz2s" Jul 19 15:38:00.012562 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:38:00.012530 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/65ec992a-d1ff-4cfb-a12f-514137d3a4c1-istiod-ca-cert\") pod \"e2e-shared-a-51b4ec-openshift-default-65f5bd8fc5-7cz2s\" (UID: \"65ec992a-d1ff-4cfb-a12f-514137d3a4c1\") " pod="openshift-ingress/e2e-shared-a-51b4ec-openshift-default-65f5bd8fc5-7cz2s" Jul 19 15:38:00.013010 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:38:00.012569 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/65ec992a-d1ff-4cfb-a12f-514137d3a4c1-istio-data\") pod \"e2e-shared-a-51b4ec-openshift-default-65f5bd8fc5-7cz2s\" (UID: \"65ec992a-d1ff-4cfb-a12f-514137d3a4c1\") " pod="openshift-ingress/e2e-shared-a-51b4ec-openshift-default-65f5bd8fc5-7cz2s" Jul 19 15:38:00.013010 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:38:00.012591 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/65ec992a-d1ff-4cfb-a12f-514137d3a4c1-istio-podinfo\") pod \"e2e-shared-a-51b4ec-openshift-default-65f5bd8fc5-7cz2s\" (UID: \"65ec992a-d1ff-4cfb-a12f-514137d3a4c1\") " pod="openshift-ingress/e2e-shared-a-51b4ec-openshift-default-65f5bd8fc5-7cz2s" Jul 19 15:38:00.013010 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:38:00.012619 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/65ec992a-d1ff-4cfb-a12f-514137d3a4c1-istio-token\") pod \"e2e-shared-a-51b4ec-openshift-default-65f5bd8fc5-7cz2s\" (UID: \"65ec992a-d1ff-4cfb-a12f-514137d3a4c1\") " pod="openshift-ingress/e2e-shared-a-51b4ec-openshift-default-65f5bd8fc5-7cz2s" Jul 19 15:38:00.013010 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:38:00.012637 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/65ec992a-d1ff-4cfb-a12f-514137d3a4c1-credential-socket\") pod \"e2e-shared-a-51b4ec-openshift-default-65f5bd8fc5-7cz2s\" (UID: \"65ec992a-d1ff-4cfb-a12f-514137d3a4c1\") " pod="openshift-ingress/e2e-shared-a-51b4ec-openshift-default-65f5bd8fc5-7cz2s" Jul 19 15:38:00.013010 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:38:00.012654 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hkw9q\" (UniqueName: \"kubernetes.io/projected/65ec992a-d1ff-4cfb-a12f-514137d3a4c1-kube-api-access-hkw9q\") pod \"e2e-shared-a-51b4ec-openshift-default-65f5bd8fc5-7cz2s\" (UID: \"65ec992a-d1ff-4cfb-a12f-514137d3a4c1\") " pod="openshift-ingress/e2e-shared-a-51b4ec-openshift-default-65f5bd8fc5-7cz2s" Jul 19 15:38:00.013010 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:38:00.012684 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/65ec992a-d1ff-4cfb-a12f-514137d3a4c1-workload-certs\") pod \"e2e-shared-a-51b4ec-openshift-default-65f5bd8fc5-7cz2s\" (UID: \"65ec992a-d1ff-4cfb-a12f-514137d3a4c1\") " pod="openshift-ingress/e2e-shared-a-51b4ec-openshift-default-65f5bd8fc5-7cz2s" Jul 19 15:38:00.013010 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:38:00.012712 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/65ec992a-d1ff-4cfb-a12f-514137d3a4c1-istio-envoy\") pod \"e2e-shared-a-51b4ec-openshift-default-65f5bd8fc5-7cz2s\" (UID: \"65ec992a-d1ff-4cfb-a12f-514137d3a4c1\") " pod="openshift-ingress/e2e-shared-a-51b4ec-openshift-default-65f5bd8fc5-7cz2s" Jul 19 15:38:00.013010 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:38:00.012737 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/65ec992a-d1ff-4cfb-a12f-514137d3a4c1-workload-socket\") pod \"e2e-shared-a-51b4ec-openshift-default-65f5bd8fc5-7cz2s\" (UID: \"65ec992a-d1ff-4cfb-a12f-514137d3a4c1\") " pod="openshift-ingress/e2e-shared-a-51b4ec-openshift-default-65f5bd8fc5-7cz2s" Jul 19 15:38:00.013413 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:38:00.013064 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/65ec992a-d1ff-4cfb-a12f-514137d3a4c1-istio-data\") pod \"e2e-shared-a-51b4ec-openshift-default-65f5bd8fc5-7cz2s\" (UID: \"65ec992a-d1ff-4cfb-a12f-514137d3a4c1\") " pod="openshift-ingress/e2e-shared-a-51b4ec-openshift-default-65f5bd8fc5-7cz2s" Jul 19 15:38:00.013413 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:38:00.013136 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/65ec992a-d1ff-4cfb-a12f-514137d3a4c1-workload-socket\") pod \"e2e-shared-a-51b4ec-openshift-default-65f5bd8fc5-7cz2s\" (UID: \"65ec992a-d1ff-4cfb-a12f-514137d3a4c1\") " pod="openshift-ingress/e2e-shared-a-51b4ec-openshift-default-65f5bd8fc5-7cz2s" Jul 19 15:38:00.013413 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:38:00.013181 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/65ec992a-d1ff-4cfb-a12f-514137d3a4c1-workload-certs\") pod \"e2e-shared-a-51b4ec-openshift-default-65f5bd8fc5-7cz2s\" (UID: \"65ec992a-d1ff-4cfb-a12f-514137d3a4c1\") " pod="openshift-ingress/e2e-shared-a-51b4ec-openshift-default-65f5bd8fc5-7cz2s" Jul 19 15:38:00.013413 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:38:00.013216 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/65ec992a-d1ff-4cfb-a12f-514137d3a4c1-credential-socket\") pod \"e2e-shared-a-51b4ec-openshift-default-65f5bd8fc5-7cz2s\" (UID: \"65ec992a-d1ff-4cfb-a12f-514137d3a4c1\") " pod="openshift-ingress/e2e-shared-a-51b4ec-openshift-default-65f5bd8fc5-7cz2s" Jul 19 15:38:00.013413 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:38:00.013331 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/65ec992a-d1ff-4cfb-a12f-514137d3a4c1-istiod-ca-cert\") pod \"e2e-shared-a-51b4ec-openshift-default-65f5bd8fc5-7cz2s\" (UID: \"65ec992a-d1ff-4cfb-a12f-514137d3a4c1\") " pod="openshift-ingress/e2e-shared-a-51b4ec-openshift-default-65f5bd8fc5-7cz2s" Jul 19 15:38:00.015290 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:38:00.015268 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/65ec992a-d1ff-4cfb-a12f-514137d3a4c1-istio-envoy\") pod \"e2e-shared-a-51b4ec-openshift-default-65f5bd8fc5-7cz2s\" (UID: \"65ec992a-d1ff-4cfb-a12f-514137d3a4c1\") " pod="openshift-ingress/e2e-shared-a-51b4ec-openshift-default-65f5bd8fc5-7cz2s" Jul 19 15:38:00.015471 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:38:00.015454 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/65ec992a-d1ff-4cfb-a12f-514137d3a4c1-istio-podinfo\") pod \"e2e-shared-a-51b4ec-openshift-default-65f5bd8fc5-7cz2s\" (UID: \"65ec992a-d1ff-4cfb-a12f-514137d3a4c1\") " pod="openshift-ingress/e2e-shared-a-51b4ec-openshift-default-65f5bd8fc5-7cz2s" Jul 19 15:38:00.020370 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:38:00.020351 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hkw9q\" (UniqueName: \"kubernetes.io/projected/65ec992a-d1ff-4cfb-a12f-514137d3a4c1-kube-api-access-hkw9q\") pod \"e2e-shared-a-51b4ec-openshift-default-65f5bd8fc5-7cz2s\" (UID: \"65ec992a-d1ff-4cfb-a12f-514137d3a4c1\") " pod="openshift-ingress/e2e-shared-a-51b4ec-openshift-default-65f5bd8fc5-7cz2s" Jul 19 15:38:00.020750 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:38:00.020734 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/65ec992a-d1ff-4cfb-a12f-514137d3a4c1-istio-token\") pod \"e2e-shared-a-51b4ec-openshift-default-65f5bd8fc5-7cz2s\" (UID: \"65ec992a-d1ff-4cfb-a12f-514137d3a4c1\") " pod="openshift-ingress/e2e-shared-a-51b4ec-openshift-default-65f5bd8fc5-7cz2s" Jul 19 15:38:00.150372 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:38:00.150352 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-51b4ec-openshift-default-65f5bd8fc5-7cz2s" Jul 19 15:38:00.284220 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:38:00.284188 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-51b4ec-openshift-default-65f5bd8fc5-7cz2s"] Jul 19 15:38:00.287146 ip-10-0-138-220 kubenswrapper[2558]: W0719 15:38:00.287120 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod65ec992a_d1ff_4cfb_a12f_514137d3a4c1.slice/crio-b3c779142b3dd1d3578c2c09fb39acbf1fee283b7c8319ce5343cecdbad25807 WatchSource:0}: Error finding container b3c779142b3dd1d3578c2c09fb39acbf1fee283b7c8319ce5343cecdbad25807: Status 404 returned error can't find the container with id b3c779142b3dd1d3578c2c09fb39acbf1fee283b7c8319ce5343cecdbad25807 Jul 19 15:38:00.289330 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:38:00.289293 2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 19 15:38:00.289393 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:38:00.289376 2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 19 15:38:00.289442 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:38:00.289425 2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 19 15:38:00.440781 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:38:00.440699 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-51b4ec-openshift-default-65f5bd8fc5-7cz2s" event={"ID":"65ec992a-d1ff-4cfb-a12f-514137d3a4c1","Type":"ContainerStarted","Data":"9f70ad46749bc3f82429d6abf68b10f88afcf79d4a677c332accfb7dc1f7251b"} Jul 19 15:38:00.440781 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:38:00.440761 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-51b4ec-openshift-default-65f5bd8fc5-7cz2s" event={"ID":"65ec992a-d1ff-4cfb-a12f-514137d3a4c1","Type":"ContainerStarted","Data":"b3c779142b3dd1d3578c2c09fb39acbf1fee283b7c8319ce5343cecdbad25807"} Jul 19 15:38:00.458391 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:38:00.458263 2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-51b4ec-openshift-default-65f5bd8fc5-7cz2s" podStartSLOduration=1.4582453659999999 podStartE2EDuration="1.458245366s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-19 15:37:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-19 15:38:00.457471549 +0000 UTC m=+2421.379174264" watchObservedRunningTime="2026-07-19 15:38:00.458245366 +0000 UTC m=+2421.379948083" Jul 19 15:38:01.150779 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:38:01.150741 2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-51b4ec-openshift-default-65f5bd8fc5-7cz2s" Jul 19 15:38:01.155738 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:38:01.155713 2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-51b4ec-openshift-default-65f5bd8fc5-7cz2s" Jul 19 15:38:01.446089 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:38:01.445986 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-51b4ec-openshift-default-65f5bd8fc5-7cz2s" Jul 19 15:38:01.446908 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:38:01.446887 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-51b4ec-openshift-default-65f5bd8fc5-7cz2s" Jul 19 15:38:02.445445 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:38:02.445414 2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-51b4ec-5c6cbb9457-x5ggx"] Jul 19 15:38:02.449628 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:38:02.449606 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-51b4ec-5c6cbb9457-x5ggx" Jul 19 15:38:02.451870 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:38:02.451850 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 19 15:38:02.452467 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:38:02.452443 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-51b4ec-serving-cert\"" Jul 19 15:38:02.452602 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:38:02.452446 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Jul 19 15:38:02.452602 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:38:02.452446 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-ljt6w\"" Jul 19 15:38:02.459993 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:38:02.459970 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-51b4ec-5c6cbb9457-x5ggx"] Jul 19 15:38:02.534005 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:38:02.533966 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2xmkj\" (UniqueName: \"kubernetes.io/projected/35de90fa-1778-452c-bc3c-46ef21c64fc0-kube-api-access-2xmkj\") pod \"maas-api-e2e-shared-a-51b4ec-5c6cbb9457-x5ggx\" (UID: \"35de90fa-1778-452c-bc3c-46ef21c64fc0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-51b4ec-5c6cbb9457-x5ggx" Jul 19 15:38:02.534225 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:38:02.534189 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/35de90fa-1778-452c-bc3c-46ef21c64fc0-maas-api-tls\") pod \"maas-api-e2e-shared-a-51b4ec-5c6cbb9457-x5ggx\" (UID: \"35de90fa-1778-452c-bc3c-46ef21c64fc0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-51b4ec-5c6cbb9457-x5ggx" Jul 19 15:38:02.534319 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:38:02.534295 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/35de90fa-1778-452c-bc3c-46ef21c64fc0-openshift-service-ca\") pod \"maas-api-e2e-shared-a-51b4ec-5c6cbb9457-x5ggx\" (UID: \"35de90fa-1778-452c-bc3c-46ef21c64fc0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-51b4ec-5c6cbb9457-x5ggx" Jul 19 15:38:02.635439 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:38:02.635401 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/35de90fa-1778-452c-bc3c-46ef21c64fc0-maas-api-tls\") pod \"maas-api-e2e-shared-a-51b4ec-5c6cbb9457-x5ggx\" (UID: \"35de90fa-1778-452c-bc3c-46ef21c64fc0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-51b4ec-5c6cbb9457-x5ggx" Jul 19 15:38:02.635627 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:38:02.635464 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/35de90fa-1778-452c-bc3c-46ef21c64fc0-openshift-service-ca\") pod \"maas-api-e2e-shared-a-51b4ec-5c6cbb9457-x5ggx\" (UID: \"35de90fa-1778-452c-bc3c-46ef21c64fc0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-51b4ec-5c6cbb9457-x5ggx" Jul 19 15:38:02.635627 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:38:02.635506 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2xmkj\" (UniqueName: \"kubernetes.io/projected/35de90fa-1778-452c-bc3c-46ef21c64fc0-kube-api-access-2xmkj\") pod \"maas-api-e2e-shared-a-51b4ec-5c6cbb9457-x5ggx\" (UID: \"35de90fa-1778-452c-bc3c-46ef21c64fc0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-51b4ec-5c6cbb9457-x5ggx" Jul 19 15:38:02.636106 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:38:02.636085 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/35de90fa-1778-452c-bc3c-46ef21c64fc0-openshift-service-ca\") pod \"maas-api-e2e-shared-a-51b4ec-5c6cbb9457-x5ggx\" (UID: \"35de90fa-1778-452c-bc3c-46ef21c64fc0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-51b4ec-5c6cbb9457-x5ggx" Jul 19 15:38:02.637972 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:38:02.637944 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/35de90fa-1778-452c-bc3c-46ef21c64fc0-maas-api-tls\") pod \"maas-api-e2e-shared-a-51b4ec-5c6cbb9457-x5ggx\" (UID: \"35de90fa-1778-452c-bc3c-46ef21c64fc0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-51b4ec-5c6cbb9457-x5ggx" Jul 19 15:38:02.643261 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:38:02.643227 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2xmkj\" (UniqueName: \"kubernetes.io/projected/35de90fa-1778-452c-bc3c-46ef21c64fc0-kube-api-access-2xmkj\") pod \"maas-api-e2e-shared-a-51b4ec-5c6cbb9457-x5ggx\" (UID: \"35de90fa-1778-452c-bc3c-46ef21c64fc0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-51b4ec-5c6cbb9457-x5ggx" Jul 19 15:38:02.762048 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:38:02.761964 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-51b4ec-5c6cbb9457-x5ggx" Jul 19 15:38:02.890168 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:38:02.890144 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-51b4ec-5c6cbb9457-x5ggx"] Jul 19 15:38:02.892899 ip-10-0-138-220 kubenswrapper[2558]: W0719 15:38:02.892863 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod35de90fa_1778_452c_bc3c_46ef21c64fc0.slice/crio-d07718ef7e5f40fbc1b58a6f6f4a9685d1a5568a734ced13616f3787660127a4 WatchSource:0}: Error finding container d07718ef7e5f40fbc1b58a6f6f4a9685d1a5568a734ced13616f3787660127a4: Status 404 returned error can't find the container with id d07718ef7e5f40fbc1b58a6f6f4a9685d1a5568a734ced13616f3787660127a4 Jul 19 15:38:03.456731 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:38:03.456684 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-51b4ec-5c6cbb9457-x5ggx" event={"ID":"35de90fa-1778-452c-bc3c-46ef21c64fc0","Type":"ContainerStarted","Data":"262daa86cd7e806296a8c43b05e4b7d1c07c988c3819ec0d0dcbab779005a10f"} Jul 19 15:38:03.457174 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:38:03.456758 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-51b4ec-5c6cbb9457-x5ggx" event={"ID":"35de90fa-1778-452c-bc3c-46ef21c64fc0","Type":"ContainerStarted","Data":"d07718ef7e5f40fbc1b58a6f6f4a9685d1a5568a734ced13616f3787660127a4"} Jul 19 15:38:03.474617 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:38:03.474552 2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-51b4ec-5c6cbb9457-x5ggx" podStartSLOduration=1.060033978 podStartE2EDuration="1.474535448s" totalImagesPullingTime="414.50147ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-19 15:38:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-19 15:38:03.472972161 +0000 UTC m=+2424.394674876" watchObservedRunningTime="2026-07-19 15:38:03.474535448 +0000 UTC m=+2424.396238163" Jul 19 15:38:04.189529 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:38:04.189505 2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-b32976-openshift-default-658f97989f-g22cf" Jul 19 15:38:04.250362 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:38:04.250268 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/23584917-2aa2-4c99-a030-1e42c0808d7d-credential-socket\") pod \"23584917-2aa2-4c99-a030-1e42c0808d7d\" (UID: \"23584917-2aa2-4c99-a030-1e42c0808d7d\") " Jul 19 15:38:04.250362 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:38:04.250312 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/23584917-2aa2-4c99-a030-1e42c0808d7d-workload-socket\") pod \"23584917-2aa2-4c99-a030-1e42c0808d7d\" (UID: \"23584917-2aa2-4c99-a030-1e42c0808d7d\") " Jul 19 15:38:04.250362 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:38:04.250343 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xrng9\" (UniqueName: \"kubernetes.io/projected/23584917-2aa2-4c99-a030-1e42c0808d7d-kube-api-access-xrng9\") pod \"23584917-2aa2-4c99-a030-1e42c0808d7d\" (UID: \"23584917-2aa2-4c99-a030-1e42c0808d7d\") " Jul 19 15:38:04.250634 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:38:04.250369 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/23584917-2aa2-4c99-a030-1e42c0808d7d-istio-envoy\") pod \"23584917-2aa2-4c99-a030-1e42c0808d7d\" (UID: \"23584917-2aa2-4c99-a030-1e42c0808d7d\") " Jul 19 15:38:04.250634 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:38:04.250397 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/23584917-2aa2-4c99-a030-1e42c0808d7d-istio-podinfo\") pod \"23584917-2aa2-4c99-a030-1e42c0808d7d\" (UID: \"23584917-2aa2-4c99-a030-1e42c0808d7d\") " Jul 19 15:38:04.250634 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:38:04.250440 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/23584917-2aa2-4c99-a030-1e42c0808d7d-istiod-ca-cert\") pod \"23584917-2aa2-4c99-a030-1e42c0808d7d\" (UID: \"23584917-2aa2-4c99-a030-1e42c0808d7d\") " Jul 19 15:38:04.250634 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:38:04.250471 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/23584917-2aa2-4c99-a030-1e42c0808d7d-istio-token\") pod \"23584917-2aa2-4c99-a030-1e42c0808d7d\" (UID: \"23584917-2aa2-4c99-a030-1e42c0808d7d\") " Jul 19 15:38:04.250634 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:38:04.250496 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/23584917-2aa2-4c99-a030-1e42c0808d7d-istio-data\") pod \"23584917-2aa2-4c99-a030-1e42c0808d7d\" (UID: \"23584917-2aa2-4c99-a030-1e42c0808d7d\") " Jul 19 15:38:04.250634 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:38:04.250556 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/23584917-2aa2-4c99-a030-1e42c0808d7d-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "23584917-2aa2-4c99-a030-1e42c0808d7d" (UID: "23584917-2aa2-4c99-a030-1e42c0808d7d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 19 15:38:04.250634 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:38:04.250564 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/23584917-2aa2-4c99-a030-1e42c0808d7d-workload-certs\") pod \"23584917-2aa2-4c99-a030-1e42c0808d7d\" (UID: \"23584917-2aa2-4c99-a030-1e42c0808d7d\") " Jul 19 15:38:04.250634 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:38:04.250607 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/23584917-2aa2-4c99-a030-1e42c0808d7d-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "23584917-2aa2-4c99-a030-1e42c0808d7d" (UID: "23584917-2aa2-4c99-a030-1e42c0808d7d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 19 15:38:04.251068 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:38:04.250804 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/23584917-2aa2-4c99-a030-1e42c0808d7d-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "23584917-2aa2-4c99-a030-1e42c0808d7d" (UID: "23584917-2aa2-4c99-a030-1e42c0808d7d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 19 15:38:04.251068 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:38:04.250897 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/23584917-2aa2-4c99-a030-1e42c0808d7d-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "23584917-2aa2-4c99-a030-1e42c0808d7d" (UID: "23584917-2aa2-4c99-a030-1e42c0808d7d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 19 15:38:04.251306 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:38:04.251217 2558 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/23584917-2aa2-4c99-a030-1e42c0808d7d-credential-socket\") on node \"ip-10-0-138-220.ec2.internal\" DevicePath \"\"" Jul 19 15:38:04.251306 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:38:04.251244 2558 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/23584917-2aa2-4c99-a030-1e42c0808d7d-workload-socket\") on node \"ip-10-0-138-220.ec2.internal\" DevicePath \"\"" Jul 19 15:38:04.251306 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:38:04.251257 2558 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/23584917-2aa2-4c99-a030-1e42c0808d7d-istiod-ca-cert\") on node \"ip-10-0-138-220.ec2.internal\" DevicePath \"\"" Jul 19 15:38:04.251306 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:38:04.251270 2558 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/23584917-2aa2-4c99-a030-1e42c0808d7d-workload-certs\") on node \"ip-10-0-138-220.ec2.internal\" DevicePath \"\"" Jul 19 15:38:04.251658 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:38:04.251406 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/23584917-2aa2-4c99-a030-1e42c0808d7d-istio-data" (OuterVolumeSpecName: "istio-data") pod "23584917-2aa2-4c99-a030-1e42c0808d7d" (UID: "23584917-2aa2-4c99-a030-1e42c0808d7d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 19 15:38:04.252957 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:38:04.252935 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/23584917-2aa2-4c99-a030-1e42c0808d7d-kube-api-access-xrng9" (OuterVolumeSpecName: "kube-api-access-xrng9") pod "23584917-2aa2-4c99-a030-1e42c0808d7d" (UID: "23584917-2aa2-4c99-a030-1e42c0808d7d"). InnerVolumeSpecName "kube-api-access-xrng9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 19 15:38:04.252957 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:38:04.252936 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/23584917-2aa2-4c99-a030-1e42c0808d7d-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "23584917-2aa2-4c99-a030-1e42c0808d7d" (UID: "23584917-2aa2-4c99-a030-1e42c0808d7d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 19 15:38:04.253257 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:38:04.253228 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/23584917-2aa2-4c99-a030-1e42c0808d7d-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "23584917-2aa2-4c99-a030-1e42c0808d7d" (UID: "23584917-2aa2-4c99-a030-1e42c0808d7d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 19 15:38:04.253257 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:38:04.253241 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/23584917-2aa2-4c99-a030-1e42c0808d7d-istio-token" (OuterVolumeSpecName: "istio-token") pod "23584917-2aa2-4c99-a030-1e42c0808d7d" (UID: "23584917-2aa2-4c99-a030-1e42c0808d7d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 19 15:38:04.352203 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:38:04.352169 2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xrng9\" (UniqueName: \"kubernetes.io/projected/23584917-2aa2-4c99-a030-1e42c0808d7d-kube-api-access-xrng9\") on node \"ip-10-0-138-220.ec2.internal\" DevicePath \"\"" Jul 19 15:38:04.352203 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:38:04.352195 2558 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/23584917-2aa2-4c99-a030-1e42c0808d7d-istio-envoy\") on node \"ip-10-0-138-220.ec2.internal\" DevicePath \"\"" Jul 19 15:38:04.352203 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:38:04.352206 2558 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/23584917-2aa2-4c99-a030-1e42c0808d7d-istio-podinfo\") on node \"ip-10-0-138-220.ec2.internal\" DevicePath \"\"" Jul 19 15:38:04.352203 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:38:04.352213 2558 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/23584917-2aa2-4c99-a030-1e42c0808d7d-istio-token\") on node \"ip-10-0-138-220.ec2.internal\" DevicePath \"\"" Jul 19 15:38:04.352455 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:38:04.352221 2558 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/23584917-2aa2-4c99-a030-1e42c0808d7d-istio-data\") on node \"ip-10-0-138-220.ec2.internal\" DevicePath \"\"" Jul 19 15:38:04.463285 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:38:04.463251 2558 generic.go:356] "Generic (PLEG): container finished" podID="23584917-2aa2-4c99-a030-1e42c0808d7d" containerID="2edbab0d70dc45561d5d3e1c52ca14ac80e069bdf558112e689e2f68ca6ce207" exitCode=0 Jul 19 15:38:04.463687 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:38:04.463300 2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-b32976-openshift-default-658f97989f-g22cf" Jul 19 15:38:04.463687 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:38:04.463340 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-b32976-openshift-default-658f97989f-g22cf" event={"ID":"23584917-2aa2-4c99-a030-1e42c0808d7d","Type":"ContainerDied","Data":"2edbab0d70dc45561d5d3e1c52ca14ac80e069bdf558112e689e2f68ca6ce207"} Jul 19 15:38:04.463687 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:38:04.463393 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-b32976-openshift-default-658f97989f-g22cf" event={"ID":"23584917-2aa2-4c99-a030-1e42c0808d7d","Type":"ContainerDied","Data":"e6624ed84304979b22e23120302773a5e3974eb33e782574db6e8b92e46c6162"} Jul 19 15:38:04.463687 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:38:04.463415 2558 scope.go:117] "RemoveContainer" containerID="2edbab0d70dc45561d5d3e1c52ca14ac80e069bdf558112e689e2f68ca6ce207" Jul 19 15:38:04.463687 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:38:04.463619 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-51b4ec-5c6cbb9457-x5ggx" Jul 19 15:38:04.472362 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:38:04.472343 2558 scope.go:117] "RemoveContainer" containerID="e76279d2ba3187ebb6b3a88bb7381086ffcb7c7af104c724f3dbb07c4dfa77a7" Jul 19 15:38:04.480002 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:38:04.479982 2558 scope.go:117] "RemoveContainer" containerID="2edbab0d70dc45561d5d3e1c52ca14ac80e069bdf558112e689e2f68ca6ce207" Jul 19 15:38:04.480279 ip-10-0-138-220 kubenswrapper[2558]: E0719 15:38:04.480257 2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2edbab0d70dc45561d5d3e1c52ca14ac80e069bdf558112e689e2f68ca6ce207\": container with ID starting with 2edbab0d70dc45561d5d3e1c52ca14ac80e069bdf558112e689e2f68ca6ce207 not found: ID does not exist" containerID="2edbab0d70dc45561d5d3e1c52ca14ac80e069bdf558112e689e2f68ca6ce207" Jul 19 15:38:04.480369 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:38:04.480292 2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2edbab0d70dc45561d5d3e1c52ca14ac80e069bdf558112e689e2f68ca6ce207"} err="failed to get container status \"2edbab0d70dc45561d5d3e1c52ca14ac80e069bdf558112e689e2f68ca6ce207\": rpc error: code = NotFound desc = could not find container \"2edbab0d70dc45561d5d3e1c52ca14ac80e069bdf558112e689e2f68ca6ce207\": container with ID starting with 2edbab0d70dc45561d5d3e1c52ca14ac80e069bdf558112e689e2f68ca6ce207 not found: ID does not exist" Jul 19 15:38:04.480369 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:38:04.480315 2558 scope.go:117] "RemoveContainer" containerID="e76279d2ba3187ebb6b3a88bb7381086ffcb7c7af104c724f3dbb07c4dfa77a7" Jul 19 15:38:04.480583 ip-10-0-138-220 kubenswrapper[2558]: E0719 15:38:04.480563 2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e76279d2ba3187ebb6b3a88bb7381086ffcb7c7af104c724f3dbb07c4dfa77a7\": container with ID starting with e76279d2ba3187ebb6b3a88bb7381086ffcb7c7af104c724f3dbb07c4dfa77a7 not found: ID does not exist" containerID="e76279d2ba3187ebb6b3a88bb7381086ffcb7c7af104c724f3dbb07c4dfa77a7" Jul 19 15:38:04.480620 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:38:04.480589 2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e76279d2ba3187ebb6b3a88bb7381086ffcb7c7af104c724f3dbb07c4dfa77a7"} err="failed to get container status \"e76279d2ba3187ebb6b3a88bb7381086ffcb7c7af104c724f3dbb07c4dfa77a7\": rpc error: code = NotFound desc = could not find container \"e76279d2ba3187ebb6b3a88bb7381086ffcb7c7af104c724f3dbb07c4dfa77a7\": container with ID starting with e76279d2ba3187ebb6b3a88bb7381086ffcb7c7af104c724f3dbb07c4dfa77a7 not found: ID does not exist" Jul 19 15:38:04.490706 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:38:04.490675 2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-b32976-openshift-default-658f97989f-g22cf"] Jul 19 15:38:04.504842 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:38:04.501888 2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-b32976-openshift-default-658f97989f-g22cf"] Jul 19 15:38:05.565011 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:38:05.564976 2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="23584917-2aa2-4c99-a030-1e42c0808d7d" path="/var/lib/kubelet/pods/23584917-2aa2-4c99-a030-1e42c0808d7d/volumes" Jul 19 15:38:07.193527 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:38:07.193491 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-d4lll"] Jul 19 15:38:07.485966 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:38:07.485884 2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-c17947-openshift-default-5748bf899-hxgq8"] Jul 19 15:38:07.486318 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:38:07.486301 2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="23584917-2aa2-4c99-a030-1e42c0808d7d" containerName="istio-proxy" Jul 19 15:38:07.486459 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:38:07.486320 2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="23584917-2aa2-4c99-a030-1e42c0808d7d" containerName="istio-proxy" Jul 19 15:38:07.486459 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:38:07.486346 2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="23584917-2aa2-4c99-a030-1e42c0808d7d" containerName="istio-proxy" Jul 19 15:38:07.486459 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:38:07.486355 2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="23584917-2aa2-4c99-a030-1e42c0808d7d" containerName="istio-proxy" Jul 19 15:38:07.486459 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:38:07.486446 2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="23584917-2aa2-4c99-a030-1e42c0808d7d" containerName="istio-proxy" Jul 19 15:38:07.490976 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:38:07.490952 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-c17947-openshift-default-5748bf899-hxgq8" Jul 19 15:38:07.493258 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:38:07.493235 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-c17947-openshift-default-dockercfg-qmwsj\"" Jul 19 15:38:07.501281 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:38:07.501248 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-c17947-openshift-default-5748bf899-hxgq8"] Jul 19 15:38:07.578325 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:38:07.578288 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/51bae637-42b6-4b76-89d7-5406a7b62fcd-credential-socket\") pod \"e2e-shared-b-c17947-openshift-default-5748bf899-hxgq8\" (UID: \"51bae637-42b6-4b76-89d7-5406a7b62fcd\") " pod="openshift-ingress/e2e-shared-b-c17947-openshift-default-5748bf899-hxgq8" Jul 19 15:38:07.578535 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:38:07.578339 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/51bae637-42b6-4b76-89d7-5406a7b62fcd-istio-envoy\") pod \"e2e-shared-b-c17947-openshift-default-5748bf899-hxgq8\" (UID: \"51bae637-42b6-4b76-89d7-5406a7b62fcd\") " pod="openshift-ingress/e2e-shared-b-c17947-openshift-default-5748bf899-hxgq8" Jul 19 15:38:07.578535 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:38:07.578445 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/51bae637-42b6-4b76-89d7-5406a7b62fcd-istio-data\") pod \"e2e-shared-b-c17947-openshift-default-5748bf899-hxgq8\" (UID: \"51bae637-42b6-4b76-89d7-5406a7b62fcd\") " pod="openshift-ingress/e2e-shared-b-c17947-openshift-default-5748bf899-hxgq8" Jul 19 15:38:07.578535 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:38:07.578488 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/51bae637-42b6-4b76-89d7-5406a7b62fcd-workload-certs\") pod \"e2e-shared-b-c17947-openshift-default-5748bf899-hxgq8\" (UID: \"51bae637-42b6-4b76-89d7-5406a7b62fcd\") " pod="openshift-ingress/e2e-shared-b-c17947-openshift-default-5748bf899-hxgq8" Jul 19 15:38:07.578535 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:38:07.578524 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-68qw6\" (UniqueName: \"kubernetes.io/projected/51bae637-42b6-4b76-89d7-5406a7b62fcd-kube-api-access-68qw6\") pod \"e2e-shared-b-c17947-openshift-default-5748bf899-hxgq8\" (UID: \"51bae637-42b6-4b76-89d7-5406a7b62fcd\") " pod="openshift-ingress/e2e-shared-b-c17947-openshift-default-5748bf899-hxgq8" Jul 19 15:38:07.578761 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:38:07.578558 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/51bae637-42b6-4b76-89d7-5406a7b62fcd-istio-podinfo\") pod \"e2e-shared-b-c17947-openshift-default-5748bf899-hxgq8\" (UID: \"51bae637-42b6-4b76-89d7-5406a7b62fcd\") " pod="openshift-ingress/e2e-shared-b-c17947-openshift-default-5748bf899-hxgq8" Jul 19 15:38:07.578761 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:38:07.578596 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/51bae637-42b6-4b76-89d7-5406a7b62fcd-istiod-ca-cert\") pod \"e2e-shared-b-c17947-openshift-default-5748bf899-hxgq8\" (UID: \"51bae637-42b6-4b76-89d7-5406a7b62fcd\") " pod="openshift-ingress/e2e-shared-b-c17947-openshift-default-5748bf899-hxgq8" Jul 19 15:38:07.578761 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:38:07.578633 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/51bae637-42b6-4b76-89d7-5406a7b62fcd-workload-socket\") pod \"e2e-shared-b-c17947-openshift-default-5748bf899-hxgq8\" (UID: \"51bae637-42b6-4b76-89d7-5406a7b62fcd\") " pod="openshift-ingress/e2e-shared-b-c17947-openshift-default-5748bf899-hxgq8" Jul 19 15:38:07.578761 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:38:07.578715 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/51bae637-42b6-4b76-89d7-5406a7b62fcd-istio-token\") pod \"e2e-shared-b-c17947-openshift-default-5748bf899-hxgq8\" (UID: \"51bae637-42b6-4b76-89d7-5406a7b62fcd\") " pod="openshift-ingress/e2e-shared-b-c17947-openshift-default-5748bf899-hxgq8" Jul 19 15:38:07.679609 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:38:07.679563 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/51bae637-42b6-4b76-89d7-5406a7b62fcd-istio-token\") pod \"e2e-shared-b-c17947-openshift-default-5748bf899-hxgq8\" (UID: \"51bae637-42b6-4b76-89d7-5406a7b62fcd\") " pod="openshift-ingress/e2e-shared-b-c17947-openshift-default-5748bf899-hxgq8" Jul 19 15:38:07.679818 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:38:07.679643 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/51bae637-42b6-4b76-89d7-5406a7b62fcd-credential-socket\") pod \"e2e-shared-b-c17947-openshift-default-5748bf899-hxgq8\" (UID: \"51bae637-42b6-4b76-89d7-5406a7b62fcd\") " pod="openshift-ingress/e2e-shared-b-c17947-openshift-default-5748bf899-hxgq8" Jul 19 15:38:07.679818 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:38:07.679672 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/51bae637-42b6-4b76-89d7-5406a7b62fcd-istio-envoy\") pod \"e2e-shared-b-c17947-openshift-default-5748bf899-hxgq8\" (UID: \"51bae637-42b6-4b76-89d7-5406a7b62fcd\") " pod="openshift-ingress/e2e-shared-b-c17947-openshift-default-5748bf899-hxgq8" Jul 19 15:38:07.679818 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:38:07.679715 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/51bae637-42b6-4b76-89d7-5406a7b62fcd-istio-data\") pod \"e2e-shared-b-c17947-openshift-default-5748bf899-hxgq8\" (UID: \"51bae637-42b6-4b76-89d7-5406a7b62fcd\") " pod="openshift-ingress/e2e-shared-b-c17947-openshift-default-5748bf899-hxgq8" Jul 19 15:38:07.679818 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:38:07.679747 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/51bae637-42b6-4b76-89d7-5406a7b62fcd-workload-certs\") pod \"e2e-shared-b-c17947-openshift-default-5748bf899-hxgq8\" (UID: \"51bae637-42b6-4b76-89d7-5406a7b62fcd\") " pod="openshift-ingress/e2e-shared-b-c17947-openshift-default-5748bf899-hxgq8" Jul 19 15:38:07.679818 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:38:07.679773 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-68qw6\" (UniqueName: \"kubernetes.io/projected/51bae637-42b6-4b76-89d7-5406a7b62fcd-kube-api-access-68qw6\") pod \"e2e-shared-b-c17947-openshift-default-5748bf899-hxgq8\" (UID: \"51bae637-42b6-4b76-89d7-5406a7b62fcd\") " pod="openshift-ingress/e2e-shared-b-c17947-openshift-default-5748bf899-hxgq8" Jul 19 15:38:07.679818 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:38:07.679808 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/51bae637-42b6-4b76-89d7-5406a7b62fcd-istio-podinfo\") pod \"e2e-shared-b-c17947-openshift-default-5748bf899-hxgq8\" (UID: \"51bae637-42b6-4b76-89d7-5406a7b62fcd\") " pod="openshift-ingress/e2e-shared-b-c17947-openshift-default-5748bf899-hxgq8" Jul 19 15:38:07.680257 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:38:07.680157 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/51bae637-42b6-4b76-89d7-5406a7b62fcd-credential-socket\") pod \"e2e-shared-b-c17947-openshift-default-5748bf899-hxgq8\" (UID: \"51bae637-42b6-4b76-89d7-5406a7b62fcd\") " pod="openshift-ingress/e2e-shared-b-c17947-openshift-default-5748bf899-hxgq8" Jul 19 15:38:07.680257 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:38:07.680210 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/51bae637-42b6-4b76-89d7-5406a7b62fcd-istio-data\") pod \"e2e-shared-b-c17947-openshift-default-5748bf899-hxgq8\" (UID: \"51bae637-42b6-4b76-89d7-5406a7b62fcd\") " pod="openshift-ingress/e2e-shared-b-c17947-openshift-default-5748bf899-hxgq8" Jul 19 15:38:07.680370 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:38:07.680302 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/51bae637-42b6-4b76-89d7-5406a7b62fcd-istiod-ca-cert\") pod \"e2e-shared-b-c17947-openshift-default-5748bf899-hxgq8\" (UID: \"51bae637-42b6-4b76-89d7-5406a7b62fcd\") " pod="openshift-ingress/e2e-shared-b-c17947-openshift-default-5748bf899-hxgq8" Jul 19 15:38:07.680370 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:38:07.680356 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/51bae637-42b6-4b76-89d7-5406a7b62fcd-workload-socket\") pod \"e2e-shared-b-c17947-openshift-default-5748bf899-hxgq8\" (UID: \"51bae637-42b6-4b76-89d7-5406a7b62fcd\") " pod="openshift-ingress/e2e-shared-b-c17947-openshift-default-5748bf899-hxgq8" Jul 19 15:38:07.680477 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:38:07.680354 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/51bae637-42b6-4b76-89d7-5406a7b62fcd-workload-certs\") pod \"e2e-shared-b-c17947-openshift-default-5748bf899-hxgq8\" (UID: \"51bae637-42b6-4b76-89d7-5406a7b62fcd\") " pod="openshift-ingress/e2e-shared-b-c17947-openshift-default-5748bf899-hxgq8" Jul 19 15:38:07.680605 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:38:07.680580 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/51bae637-42b6-4b76-89d7-5406a7b62fcd-workload-socket\") pod \"e2e-shared-b-c17947-openshift-default-5748bf899-hxgq8\" (UID: \"51bae637-42b6-4b76-89d7-5406a7b62fcd\") " pod="openshift-ingress/e2e-shared-b-c17947-openshift-default-5748bf899-hxgq8" Jul 19 15:38:07.680723 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:38:07.680708 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/51bae637-42b6-4b76-89d7-5406a7b62fcd-istiod-ca-cert\") pod \"e2e-shared-b-c17947-openshift-default-5748bf899-hxgq8\" (UID: \"51bae637-42b6-4b76-89d7-5406a7b62fcd\") " pod="openshift-ingress/e2e-shared-b-c17947-openshift-default-5748bf899-hxgq8" Jul 19 15:38:07.682761 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:38:07.682736 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/51bae637-42b6-4b76-89d7-5406a7b62fcd-istio-envoy\") pod \"e2e-shared-b-c17947-openshift-default-5748bf899-hxgq8\" (UID: \"51bae637-42b6-4b76-89d7-5406a7b62fcd\") " pod="openshift-ingress/e2e-shared-b-c17947-openshift-default-5748bf899-hxgq8" Jul 19 15:38:07.682994 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:38:07.682972 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/51bae637-42b6-4b76-89d7-5406a7b62fcd-istio-podinfo\") pod \"e2e-shared-b-c17947-openshift-default-5748bf899-hxgq8\" (UID: \"51bae637-42b6-4b76-89d7-5406a7b62fcd\") " pod="openshift-ingress/e2e-shared-b-c17947-openshift-default-5748bf899-hxgq8" Jul 19 15:38:07.691276 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:38:07.691248 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/51bae637-42b6-4b76-89d7-5406a7b62fcd-istio-token\") pod \"e2e-shared-b-c17947-openshift-default-5748bf899-hxgq8\" (UID: \"51bae637-42b6-4b76-89d7-5406a7b62fcd\") " pod="openshift-ingress/e2e-shared-b-c17947-openshift-default-5748bf899-hxgq8" Jul 19 15:38:07.691828 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:38:07.691808 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-68qw6\" (UniqueName: \"kubernetes.io/projected/51bae637-42b6-4b76-89d7-5406a7b62fcd-kube-api-access-68qw6\") pod \"e2e-shared-b-c17947-openshift-default-5748bf899-hxgq8\" (UID: \"51bae637-42b6-4b76-89d7-5406a7b62fcd\") " pod="openshift-ingress/e2e-shared-b-c17947-openshift-default-5748bf899-hxgq8" Jul 19 15:38:07.803958 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:38:07.803862 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-c17947-openshift-default-5748bf899-hxgq8" Jul 19 15:38:07.945813 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:38:07.945787 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-c17947-openshift-default-5748bf899-hxgq8"] Jul 19 15:38:07.947675 ip-10-0-138-220 kubenswrapper[2558]: W0719 15:38:07.947640 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod51bae637_42b6_4b76_89d7_5406a7b62fcd.slice/crio-63f812d5048061d020be557a3694699b7becd522423b0291d3e629f9991c726d WatchSource:0}: Error finding container 63f812d5048061d020be557a3694699b7becd522423b0291d3e629f9991c726d: Status 404 returned error can't find the container with id 63f812d5048061d020be557a3694699b7becd522423b0291d3e629f9991c726d Jul 19 15:38:07.949886 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:38:07.949852 2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 19 15:38:07.949950 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:38:07.949930 2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 19 15:38:07.949986 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:38:07.949970 2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 19 15:38:08.482861 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:38:08.482823 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-c17947-openshift-default-5748bf899-hxgq8" event={"ID":"51bae637-42b6-4b76-89d7-5406a7b62fcd","Type":"ContainerStarted","Data":"ae6937ca774a5b56758194155835eb038c1a9bec6e9dc05a84d2ec2918ae8aa2"} Jul 19 15:38:08.483296 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:38:08.482878 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-c17947-openshift-default-5748bf899-hxgq8" event={"ID":"51bae637-42b6-4b76-89d7-5406a7b62fcd","Type":"ContainerStarted","Data":"63f812d5048061d020be557a3694699b7becd522423b0291d3e629f9991c726d"} Jul 19 15:38:08.501833 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:38:08.501780 2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-c17947-openshift-default-5748bf899-hxgq8" podStartSLOduration=1.5017642279999999 podStartE2EDuration="1.501764228s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-19 15:38:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-19 15:38:08.50005324 +0000 UTC m=+2429.421755945" watchObservedRunningTime="2026-07-19 15:38:08.501764228 +0000 UTC m=+2429.423466942" Jul 19 15:38:08.804442 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:38:08.804360 2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-c17947-openshift-default-5748bf899-hxgq8" Jul 19 15:38:08.809299 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:38:08.809277 2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-c17947-openshift-default-5748bf899-hxgq8" Jul 19 15:38:09.487806 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:38:09.487773 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-c17947-openshift-default-5748bf899-hxgq8" Jul 19 15:38:09.488686 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:38:09.488660 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-c17947-openshift-default-5748bf899-hxgq8" Jul 19 15:38:10.469625 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:38:10.469584 2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-c17947-8cc7947f9-zkdqm"] Jul 19 15:38:10.470200 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:38:10.470171 2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="23584917-2aa2-4c99-a030-1e42c0808d7d" containerName="istio-proxy" Jul 19 15:38:10.482050 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:38:10.479664 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-c17947-8cc7947f9-zkdqm" Jul 19 15:38:10.483855 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:38:10.483832 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-c17947-8cc7947f9-zkdqm"] Jul 19 15:38:10.484063 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:38:10.484016 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-c17947-serving-cert\"" Jul 19 15:38:10.489094 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:38:10.488735 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-51b4ec-5c6cbb9457-x5ggx" Jul 19 15:38:10.603646 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:38:10.603611 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9644f172-e0ae-493b-b013-0a387feea206-openshift-service-ca\") pod \"maas-api-e2e-shared-b-c17947-8cc7947f9-zkdqm\" (UID: \"9644f172-e0ae-493b-b013-0a387feea206\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-c17947-8cc7947f9-zkdqm" Jul 19 15:38:10.603850 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:38:10.603660 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4fjjf\" (UniqueName: \"kubernetes.io/projected/9644f172-e0ae-493b-b013-0a387feea206-kube-api-access-4fjjf\") pod \"maas-api-e2e-shared-b-c17947-8cc7947f9-zkdqm\" (UID: \"9644f172-e0ae-493b-b013-0a387feea206\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-c17947-8cc7947f9-zkdqm" Jul 19 15:38:10.603850 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:38:10.603762 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9644f172-e0ae-493b-b013-0a387feea206-maas-api-tls\") pod \"maas-api-e2e-shared-b-c17947-8cc7947f9-zkdqm\" (UID: \"9644f172-e0ae-493b-b013-0a387feea206\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-c17947-8cc7947f9-zkdqm" Jul 19 15:38:10.704831 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:38:10.704790 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9644f172-e0ae-493b-b013-0a387feea206-openshift-service-ca\") pod \"maas-api-e2e-shared-b-c17947-8cc7947f9-zkdqm\" (UID: \"9644f172-e0ae-493b-b013-0a387feea206\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-c17947-8cc7947f9-zkdqm" Jul 19 15:38:10.705063 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:38:10.704836 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4fjjf\" (UniqueName: \"kubernetes.io/projected/9644f172-e0ae-493b-b013-0a387feea206-kube-api-access-4fjjf\") pod \"maas-api-e2e-shared-b-c17947-8cc7947f9-zkdqm\" (UID: \"9644f172-e0ae-493b-b013-0a387feea206\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-c17947-8cc7947f9-zkdqm" Jul 19 15:38:10.705063 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:38:10.704880 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9644f172-e0ae-493b-b013-0a387feea206-maas-api-tls\") pod \"maas-api-e2e-shared-b-c17947-8cc7947f9-zkdqm\" (UID: \"9644f172-e0ae-493b-b013-0a387feea206\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-c17947-8cc7947f9-zkdqm" Jul 19 15:38:10.705214 ip-10-0-138-220 kubenswrapper[2558]: E0719 15:38:10.705064 2558 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-c17947-serving-cert: secret "maas-api-e2e-shared-b-c17947-serving-cert" not found Jul 19 15:38:10.705214 ip-10-0-138-220 kubenswrapper[2558]: E0719 15:38:10.705144 2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9644f172-e0ae-493b-b013-0a387feea206-maas-api-tls podName:9644f172-e0ae-493b-b013-0a387feea206 nodeName:}" failed. No retries permitted until 2026-07-19 15:38:11.205123392 +0000 UTC m=+2432.126826100 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/9644f172-e0ae-493b-b013-0a387feea206-maas-api-tls") pod "maas-api-e2e-shared-b-c17947-8cc7947f9-zkdqm" (UID: "9644f172-e0ae-493b-b013-0a387feea206") : secret "maas-api-e2e-shared-b-c17947-serving-cert" not found Jul 19 15:38:10.705553 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:38:10.705531 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9644f172-e0ae-493b-b013-0a387feea206-openshift-service-ca\") pod \"maas-api-e2e-shared-b-c17947-8cc7947f9-zkdqm\" (UID: \"9644f172-e0ae-493b-b013-0a387feea206\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-c17947-8cc7947f9-zkdqm" Jul 19 15:38:10.712971 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:38:10.712944 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4fjjf\" (UniqueName: \"kubernetes.io/projected/9644f172-e0ae-493b-b013-0a387feea206-kube-api-access-4fjjf\") pod \"maas-api-e2e-shared-b-c17947-8cc7947f9-zkdqm\" (UID: \"9644f172-e0ae-493b-b013-0a387feea206\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-c17947-8cc7947f9-zkdqm" Jul 19 15:38:11.209162 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:38:11.209123 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9644f172-e0ae-493b-b013-0a387feea206-maas-api-tls\") pod \"maas-api-e2e-shared-b-c17947-8cc7947f9-zkdqm\" (UID: \"9644f172-e0ae-493b-b013-0a387feea206\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-c17947-8cc7947f9-zkdqm" Jul 19 15:38:11.211632 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:38:11.211610 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9644f172-e0ae-493b-b013-0a387feea206-maas-api-tls\") pod \"maas-api-e2e-shared-b-c17947-8cc7947f9-zkdqm\" (UID: \"9644f172-e0ae-493b-b013-0a387feea206\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-c17947-8cc7947f9-zkdqm" Jul 19 15:38:11.396828 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:38:11.396787 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-c17947-8cc7947f9-zkdqm" Jul 19 15:38:11.579181 ip-10-0-138-220 kubenswrapper[2558]: W0719 15:38:11.579141 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9644f172_e0ae_493b_b013_0a387feea206.slice/crio-e8d3e67b7e405dbc687a5fca571b52cca111077e1d3f4069c16ac4fe7ee07ae9 WatchSource:0}: Error finding container e8d3e67b7e405dbc687a5fca571b52cca111077e1d3f4069c16ac4fe7ee07ae9: Status 404 returned error can't find the container with id e8d3e67b7e405dbc687a5fca571b52cca111077e1d3f4069c16ac4fe7ee07ae9 Jul 19 15:38:11.582143 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:38:11.582120 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-c17947-8cc7947f9-zkdqm"] Jul 19 15:38:12.507780 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:38:12.507748 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-c17947-8cc7947f9-zkdqm" event={"ID":"9644f172-e0ae-493b-b013-0a387feea206","Type":"ContainerStarted","Data":"6f319a685f828e4749d665d5506cf9e5e281d799b969b3d7e93692cf68f76b94"} Jul 19 15:38:12.507926 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:38:12.507801 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-c17947-8cc7947f9-zkdqm" event={"ID":"9644f172-e0ae-493b-b013-0a387feea206","Type":"ContainerStarted","Data":"e8d3e67b7e405dbc687a5fca571b52cca111077e1d3f4069c16ac4fe7ee07ae9"} Jul 19 15:38:12.507926 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:38:12.507858 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-c17947-8cc7947f9-zkdqm" Jul 19 15:38:12.527221 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:38:12.527165 2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-c17947-8cc7947f9-zkdqm" podStartSLOduration=2.0395577559999998 podStartE2EDuration="2.527146285s" totalImagesPullingTime="487.588529ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-19 15:38:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-19 15:38:12.524970043 +0000 UTC m=+2433.446672772" watchObservedRunningTime="2026-07-19 15:38:12.527146285 +0000 UTC m=+2433.448849000" Jul 19 15:38:15.528143 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:38:15.528110 2558 generic.go:356] "Generic (PLEG): container finished" podID="65ec992a-d1ff-4cfb-a12f-514137d3a4c1" containerID="9f70ad46749bc3f82429d6abf68b10f88afcf79d4a677c332accfb7dc1f7251b" exitCode=0 Jul 19 15:38:15.528546 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:38:15.528184 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-51b4ec-openshift-default-65f5bd8fc5-7cz2s" event={"ID":"65ec992a-d1ff-4cfb-a12f-514137d3a4c1","Type":"ContainerDied","Data":"9f70ad46749bc3f82429d6abf68b10f88afcf79d4a677c332accfb7dc1f7251b"} Jul 19 15:38:15.528594 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:38:15.528545 2558 scope.go:117] "RemoveContainer" containerID="9f70ad46749bc3f82429d6abf68b10f88afcf79d4a677c332accfb7dc1f7251b" Jul 19 15:38:15.529538 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:38:15.529510 2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 19 15:38:15.529635 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:38:15.529616 2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 19 15:38:15.529677 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:38:15.529668 2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 19 15:38:16.091011 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:38:16.090968 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-d4lll"] Jul 19 15:38:16.446844 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:38:16.446813 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-51b4ec-openshift-default-65f5bd8fc5-7cz2s" Jul 19 15:38:16.534279 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:38:16.534241 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-51b4ec-openshift-default-65f5bd8fc5-7cz2s" event={"ID":"65ec992a-d1ff-4cfb-a12f-514137d3a4c1","Type":"ContainerStarted","Data":"b4cdac0ff10e2b2cc995ef2a194a60f87b409d3724f0f56f57331432193ecab9"} Jul 19 15:38:17.152235 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:38:17.152200 2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-51b4ec-openshift-default-65f5bd8fc5-7cz2s" Jul 19 15:38:17.152514 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:38:17.152254 2558 patch_prober.go:28] interesting pod/e2e-shared-a-51b4ec-openshift-default-65f5bd8fc5-7cz2s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" start-of-body= Jul 19 15:38:17.152689 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:38:17.152665 2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-51b4ec-openshift-default-65f5bd8fc5-7cz2s" podUID="65ec992a-d1ff-4cfb-a12f-514137d3a4c1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" Jul 19 15:38:18.151384 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:38:18.151344 2558 patch_prober.go:28] interesting pod/e2e-shared-a-51b4ec-openshift-default-65f5bd8fc5-7cz2s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" start-of-body= Jul 19 15:38:18.152601 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:38:18.152168 2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-51b4ec-openshift-default-65f5bd8fc5-7cz2s" podUID="65ec992a-d1ff-4cfb-a12f-514137d3a4c1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" Jul 19 15:38:18.523455 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:38:18.523245 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-c17947-8cc7947f9-zkdqm" Jul 19 15:38:19.151246 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:38:19.151207 2558 patch_prober.go:28] interesting pod/e2e-shared-a-51b4ec-openshift-default-65f5bd8fc5-7cz2s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" start-of-body= Jul 19 15:38:19.151535 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:38:19.151507 2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-51b4ec-openshift-default-65f5bd8fc5-7cz2s" podUID="65ec992a-d1ff-4cfb-a12f-514137d3a4c1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" Jul 19 15:38:20.151584 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:38:20.151532 2558 patch_prober.go:28] interesting pod/e2e-shared-a-51b4ec-openshift-default-65f5bd8fc5-7cz2s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" start-of-body= Jul 19 15:38:20.152127 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:38:20.151619 2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-51b4ec-openshift-default-65f5bd8fc5-7cz2s" podUID="65ec992a-d1ff-4cfb-a12f-514137d3a4c1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" Jul 19 15:38:21.157339 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:38:21.157295 2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-51b4ec-openshift-default-65f5bd8fc5-7cz2s" Jul 19 15:38:21.157836 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:38:21.157760 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-51b4ec-openshift-default-65f5bd8fc5-7cz2s" Jul 19 15:38:21.163655 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:38:21.163616 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-51b4ec-openshift-default-65f5bd8fc5-7cz2s" Jul 19 15:38:24.489741 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:38:24.489653 2558 patch_prober.go:28] interesting pod/e2e-shared-b-c17947-openshift-default-5748bf899-hxgq8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" start-of-body= Jul 19 15:38:24.489741 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:38:24.489718 2558 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-c17947-openshift-default-5748bf899-hxgq8" podUID="51bae637-42b6-4b76-89d7-5406a7b62fcd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" Jul 19 15:38:24.577221 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:38:24.577135 2558 generic.go:356] "Generic (PLEG): container finished" podID="51bae637-42b6-4b76-89d7-5406a7b62fcd" containerID="ae6937ca774a5b56758194155835eb038c1a9bec6e9dc05a84d2ec2918ae8aa2" exitCode=0 Jul 19 15:38:24.577372 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:38:24.577212 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-c17947-openshift-default-5748bf899-hxgq8" event={"ID":"51bae637-42b6-4b76-89d7-5406a7b62fcd","Type":"ContainerDied","Data":"ae6937ca774a5b56758194155835eb038c1a9bec6e9dc05a84d2ec2918ae8aa2"} Jul 19 15:38:24.577615 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:38:24.577601 2558 scope.go:117] "RemoveContainer" containerID="ae6937ca774a5b56758194155835eb038c1a9bec6e9dc05a84d2ec2918ae8aa2" Jul 19 15:38:24.578692 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:38:24.578666 2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 19 15:38:24.578761 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:38:24.578730 2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 19 15:38:24.578800 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:38:24.578761 2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 19 15:38:25.589214 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:38:25.589182 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-c17947-openshift-default-5748bf899-hxgq8" event={"ID":"51bae637-42b6-4b76-89d7-5406a7b62fcd","Type":"ContainerStarted","Data":"c89370003d3ad9557990f64b47d042e8e664ba2ecfa3da93e5deaac195d30ed6"} Jul 19 15:38:25.804122 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:38:25.804081 2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-c17947-openshift-default-5748bf899-hxgq8" Jul 19 15:38:25.804379 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:38:25.804352 2558 patch_prober.go:28] interesting pod/e2e-shared-b-c17947-openshift-default-5748bf899-hxgq8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" start-of-body= Jul 19 15:38:25.804440 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:38:25.804419 2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-c17947-openshift-default-5748bf899-hxgq8" podUID="51bae637-42b6-4b76-89d7-5406a7b62fcd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" Jul 19 15:38:26.804771 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:38:26.804729 2558 patch_prober.go:28] interesting pod/e2e-shared-b-c17947-openshift-default-5748bf899-hxgq8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" start-of-body= Jul 19 15:38:26.805175 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:38:26.804789 2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-c17947-openshift-default-5748bf899-hxgq8" podUID="51bae637-42b6-4b76-89d7-5406a7b62fcd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" Jul 19 15:38:27.805159 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:38:27.805117 2558 patch_prober.go:28] interesting pod/e2e-shared-b-c17947-openshift-default-5748bf899-hxgq8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" start-of-body= Jul 19 15:38:27.805557 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:38:27.805182 2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-c17947-openshift-default-5748bf899-hxgq8" podUID="51bae637-42b6-4b76-89d7-5406a7b62fcd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" Jul 19 15:38:28.805235 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:38:28.805196 2558 patch_prober.go:28] interesting pod/e2e-shared-b-c17947-openshift-default-5748bf899-hxgq8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" start-of-body= Jul 19 15:38:28.805655 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:38:28.805255 2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-c17947-openshift-default-5748bf899-hxgq8" podUID="51bae637-42b6-4b76-89d7-5406a7b62fcd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" Jul 19 15:38:29.808707 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:38:29.808676 2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-c17947-openshift-default-5748bf899-hxgq8" Jul 19 15:38:29.809154 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:38:29.809056 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-c17947-openshift-default-5748bf899-hxgq8" Jul 19 15:38:29.809913 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:38:29.809890 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-c17947-openshift-default-5748bf899-hxgq8" Jul 19 15:38:30.890396 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:38:30.890358 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-d4lll"] Jul 19 15:39:16.083978 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:39:16.083935 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-d4lll"] Jul 19 15:39:29.696862 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:39:29.696823 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-d4lll"] Jul 19 15:39:45.991368 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:39:45.991330 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-d4lll"] Jul 19 15:40:05.387468 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:40:05.387423 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-d4lll"] Jul 19 15:40:20.888104 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:40:20.888065 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-d4lll"] Jul 19 15:40:37.491147 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:40:37.491109 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-d4lll"] Jul 19 15:40:54.091394 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:40:54.091353 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-d4lll"] Jul 19 15:41:12.293952 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:41:12.293912 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-d4lll"] Jul 19 15:41:30.191597 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:41:30.191562 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-d4lll"] Jul 19 15:41:39.683012 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:41:39.682981 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-d4lll"] Jul 19 15:41:54.185715 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:41:54.185678 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-d4lll"] Jul 19 15:42:13.589725 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:42:13.589693 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-d4lll"] Jul 19 15:42:27.696253 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:42:27.696216 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-d4lll"] Jul 19 15:42:39.691939 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:42:39.691867 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-5wfzs_5b2d3a07-86b2-4ba2-8a71-c2d89ada79c6/ovn-acl-logging/0.log" Jul 19 15:42:39.698293 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:42:39.698271 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-5wfzs_5b2d3a07-86b2-4ba2-8a71-c2d89ada79c6/ovn-acl-logging/0.log" Jul 19 15:42:46.893644 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:42:46.893611 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-d4lll"] Jul 19 15:43:22.184600 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:43:22.184561 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-d4lll"] Jul 19 15:43:39.193160 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:43:39.193128 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-d4lll"] Jul 19 15:43:53.595526 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:43:53.595471 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-d4lll"] Jul 19 15:44:12.593057 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:44:12.593005 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-d4lll"] Jul 19 15:44:31.746240 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:44:31.746196 2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-51b4ec-5c6cbb9457-x5ggx"] Jul 19 15:44:31.746810 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:44:31.746496 2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-51b4ec-5c6cbb9457-x5ggx" podUID="35de90fa-1778-452c-bc3c-46ef21c64fc0" containerName="maas-api" containerID="cri-o://262daa86cd7e806296a8c43b05e4b7d1c07c988c3819ec0d0dcbab779005a10f" gracePeriod=30 Jul 19 15:44:32.001471 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:44:32.001409 2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-51b4ec-5c6cbb9457-x5ggx" Jul 19 15:44:32.123858 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:44:32.123803 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/35de90fa-1778-452c-bc3c-46ef21c64fc0-maas-api-tls\") pod \"35de90fa-1778-452c-bc3c-46ef21c64fc0\" (UID: \"35de90fa-1778-452c-bc3c-46ef21c64fc0\") " Jul 19 15:44:32.124079 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:44:32.123867 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/35de90fa-1778-452c-bc3c-46ef21c64fc0-openshift-service-ca\") pod \"35de90fa-1778-452c-bc3c-46ef21c64fc0\" (UID: \"35de90fa-1778-452c-bc3c-46ef21c64fc0\") " Jul 19 15:44:32.124079 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:44:32.123996 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2xmkj\" (UniqueName: \"kubernetes.io/projected/35de90fa-1778-452c-bc3c-46ef21c64fc0-kube-api-access-2xmkj\") pod \"35de90fa-1778-452c-bc3c-46ef21c64fc0\" (UID: \"35de90fa-1778-452c-bc3c-46ef21c64fc0\") " Jul 19 15:44:32.124297 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:44:32.124273 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/35de90fa-1778-452c-bc3c-46ef21c64fc0-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "35de90fa-1778-452c-bc3c-46ef21c64fc0" (UID: "35de90fa-1778-452c-bc3c-46ef21c64fc0"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 19 15:44:32.126221 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:44:32.126194 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/35de90fa-1778-452c-bc3c-46ef21c64fc0-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "35de90fa-1778-452c-bc3c-46ef21c64fc0" (UID: "35de90fa-1778-452c-bc3c-46ef21c64fc0"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 19 15:44:32.126334 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:44:32.126312 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/35de90fa-1778-452c-bc3c-46ef21c64fc0-kube-api-access-2xmkj" (OuterVolumeSpecName: "kube-api-access-2xmkj") pod "35de90fa-1778-452c-bc3c-46ef21c64fc0" (UID: "35de90fa-1778-452c-bc3c-46ef21c64fc0"). InnerVolumeSpecName "kube-api-access-2xmkj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 19 15:44:32.144468 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:44:32.144434 2558 generic.go:356] "Generic (PLEG): container finished" podID="35de90fa-1778-452c-bc3c-46ef21c64fc0" containerID="262daa86cd7e806296a8c43b05e4b7d1c07c988c3819ec0d0dcbab779005a10f" exitCode=0 Jul 19 15:44:32.144609 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:44:32.144484 2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-51b4ec-5c6cbb9457-x5ggx" Jul 19 15:44:32.144609 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:44:32.144530 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-51b4ec-5c6cbb9457-x5ggx" event={"ID":"35de90fa-1778-452c-bc3c-46ef21c64fc0","Type":"ContainerDied","Data":"262daa86cd7e806296a8c43b05e4b7d1c07c988c3819ec0d0dcbab779005a10f"} Jul 19 15:44:32.144609 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:44:32.144589 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-51b4ec-5c6cbb9457-x5ggx" event={"ID":"35de90fa-1778-452c-bc3c-46ef21c64fc0","Type":"ContainerDied","Data":"d07718ef7e5f40fbc1b58a6f6f4a9685d1a5568a734ced13616f3787660127a4"} Jul 19 15:44:32.144806 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:44:32.144619 2558 scope.go:117] "RemoveContainer" containerID="262daa86cd7e806296a8c43b05e4b7d1c07c988c3819ec0d0dcbab779005a10f" Jul 19 15:44:32.153578 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:44:32.153558 2558 scope.go:117] "RemoveContainer" containerID="262daa86cd7e806296a8c43b05e4b7d1c07c988c3819ec0d0dcbab779005a10f" Jul 19 15:44:32.153831 ip-10-0-138-220 kubenswrapper[2558]: E0719 15:44:32.153811 2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"262daa86cd7e806296a8c43b05e4b7d1c07c988c3819ec0d0dcbab779005a10f\": container with ID starting with 262daa86cd7e806296a8c43b05e4b7d1c07c988c3819ec0d0dcbab779005a10f not found: ID does not exist" containerID="262daa86cd7e806296a8c43b05e4b7d1c07c988c3819ec0d0dcbab779005a10f" Jul 19 15:44:32.153901 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:44:32.153840 2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"262daa86cd7e806296a8c43b05e4b7d1c07c988c3819ec0d0dcbab779005a10f"} err="failed to get container status \"262daa86cd7e806296a8c43b05e4b7d1c07c988c3819ec0d0dcbab779005a10f\": rpc error: code = NotFound desc = could not find container \"262daa86cd7e806296a8c43b05e4b7d1c07c988c3819ec0d0dcbab779005a10f\": container with ID starting with 262daa86cd7e806296a8c43b05e4b7d1c07c988c3819ec0d0dcbab779005a10f not found: ID does not exist" Jul 19 15:44:32.165516 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:44:32.165491 2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-51b4ec-5c6cbb9457-x5ggx"] Jul 19 15:44:32.169004 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:44:32.168976 2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-51b4ec-5c6cbb9457-x5ggx"] Jul 19 15:44:32.225410 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:44:32.225378 2558 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/35de90fa-1778-452c-bc3c-46ef21c64fc0-maas-api-tls\") on node \"ip-10-0-138-220.ec2.internal\" DevicePath \"\"" Jul 19 15:44:32.225410 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:44:32.225406 2558 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/35de90fa-1778-452c-bc3c-46ef21c64fc0-openshift-service-ca\") on node \"ip-10-0-138-220.ec2.internal\" DevicePath \"\"" Jul 19 15:44:32.225410 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:44:32.225416 2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2xmkj\" (UniqueName: \"kubernetes.io/projected/35de90fa-1778-452c-bc3c-46ef21c64fc0-kube-api-access-2xmkj\") on node \"ip-10-0-138-220.ec2.internal\" DevicePath \"\"" Jul 19 15:44:33.564743 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:44:33.564710 2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="35de90fa-1778-452c-bc3c-46ef21c64fc0" path="/var/lib/kubelet/pods/35de90fa-1778-452c-bc3c-46ef21c64fc0/volumes" Jul 19 15:44:34.587587 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:44:34.587555 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-d4lll"] Jul 19 15:44:42.660723 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:44:42.660689 2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-51b4ec-openshift-default-65f5bd8fc5-7cz2s"] Jul 19 15:44:42.661149 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:44:42.660916 2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-51b4ec-openshift-default-65f5bd8fc5-7cz2s" podUID="65ec992a-d1ff-4cfb-a12f-514137d3a4c1" containerName="istio-proxy" containerID="cri-o://b4cdac0ff10e2b2cc995ef2a194a60f87b409d3724f0f56f57331432193ecab9" gracePeriod=30 Jul 19 15:44:47.914770 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:44:47.914746 2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-51b4ec-openshift-default-65f5bd8fc5-7cz2s" Jul 19 15:44:48.059403 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:44:48.059314 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/65ec992a-d1ff-4cfb-a12f-514137d3a4c1-istio-token\") pod \"65ec992a-d1ff-4cfb-a12f-514137d3a4c1\" (UID: \"65ec992a-d1ff-4cfb-a12f-514137d3a4c1\") " Jul 19 15:44:48.059403 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:44:48.059354 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/65ec992a-d1ff-4cfb-a12f-514137d3a4c1-istio-podinfo\") pod \"65ec992a-d1ff-4cfb-a12f-514137d3a4c1\" (UID: \"65ec992a-d1ff-4cfb-a12f-514137d3a4c1\") " Jul 19 15:44:48.059403 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:44:48.059385 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/65ec992a-d1ff-4cfb-a12f-514137d3a4c1-istio-envoy\") pod \"65ec992a-d1ff-4cfb-a12f-514137d3a4c1\" (UID: \"65ec992a-d1ff-4cfb-a12f-514137d3a4c1\") " Jul 19 15:44:48.059661 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:44:48.059413 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/65ec992a-d1ff-4cfb-a12f-514137d3a4c1-istiod-ca-cert\") pod \"65ec992a-d1ff-4cfb-a12f-514137d3a4c1\" (UID: \"65ec992a-d1ff-4cfb-a12f-514137d3a4c1\") " Jul 19 15:44:48.059661 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:44:48.059431 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/65ec992a-d1ff-4cfb-a12f-514137d3a4c1-workload-certs\") pod \"65ec992a-d1ff-4cfb-a12f-514137d3a4c1\" (UID: \"65ec992a-d1ff-4cfb-a12f-514137d3a4c1\") " Jul 19 15:44:48.059661 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:44:48.059469 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hkw9q\" (UniqueName: \"kubernetes.io/projected/65ec992a-d1ff-4cfb-a12f-514137d3a4c1-kube-api-access-hkw9q\") pod \"65ec992a-d1ff-4cfb-a12f-514137d3a4c1\" (UID: \"65ec992a-d1ff-4cfb-a12f-514137d3a4c1\") " Jul 19 15:44:48.059661 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:44:48.059497 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/65ec992a-d1ff-4cfb-a12f-514137d3a4c1-istio-data\") pod \"65ec992a-d1ff-4cfb-a12f-514137d3a4c1\" (UID: \"65ec992a-d1ff-4cfb-a12f-514137d3a4c1\") " Jul 19 15:44:48.059661 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:44:48.059529 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/65ec992a-d1ff-4cfb-a12f-514137d3a4c1-workload-socket\") pod \"65ec992a-d1ff-4cfb-a12f-514137d3a4c1\" (UID: \"65ec992a-d1ff-4cfb-a12f-514137d3a4c1\") " Jul 19 15:44:48.059661 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:44:48.059567 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/65ec992a-d1ff-4cfb-a12f-514137d3a4c1-credential-socket\") pod \"65ec992a-d1ff-4cfb-a12f-514137d3a4c1\" (UID: \"65ec992a-d1ff-4cfb-a12f-514137d3a4c1\") " Jul 19 15:44:48.059972 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:44:48.059790 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/65ec992a-d1ff-4cfb-a12f-514137d3a4c1-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "65ec992a-d1ff-4cfb-a12f-514137d3a4c1" (UID: "65ec992a-d1ff-4cfb-a12f-514137d3a4c1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 19 15:44:48.059972 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:44:48.059816 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/65ec992a-d1ff-4cfb-a12f-514137d3a4c1-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "65ec992a-d1ff-4cfb-a12f-514137d3a4c1" (UID: "65ec992a-d1ff-4cfb-a12f-514137d3a4c1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 19 15:44:48.059972 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:44:48.059924 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/65ec992a-d1ff-4cfb-a12f-514137d3a4c1-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "65ec992a-d1ff-4cfb-a12f-514137d3a4c1" (UID: "65ec992a-d1ff-4cfb-a12f-514137d3a4c1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 19 15:44:48.060166 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:44:48.060055 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/65ec992a-d1ff-4cfb-a12f-514137d3a4c1-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "65ec992a-d1ff-4cfb-a12f-514137d3a4c1" (UID: "65ec992a-d1ff-4cfb-a12f-514137d3a4c1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 19 15:44:48.060551 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:44:48.060525 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/65ec992a-d1ff-4cfb-a12f-514137d3a4c1-istio-data" (OuterVolumeSpecName: "istio-data") pod "65ec992a-d1ff-4cfb-a12f-514137d3a4c1" (UID: "65ec992a-d1ff-4cfb-a12f-514137d3a4c1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 19 15:44:48.061842 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:44:48.061819 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/65ec992a-d1ff-4cfb-a12f-514137d3a4c1-istio-token" (OuterVolumeSpecName: "istio-token") pod "65ec992a-d1ff-4cfb-a12f-514137d3a4c1" (UID: "65ec992a-d1ff-4cfb-a12f-514137d3a4c1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 19 15:44:48.062101 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:44:48.062075 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/65ec992a-d1ff-4cfb-a12f-514137d3a4c1-kube-api-access-hkw9q" (OuterVolumeSpecName: "kube-api-access-hkw9q") pod "65ec992a-d1ff-4cfb-a12f-514137d3a4c1" (UID: "65ec992a-d1ff-4cfb-a12f-514137d3a4c1"). InnerVolumeSpecName "kube-api-access-hkw9q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 19 15:44:48.062218 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:44:48.062203 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/65ec992a-d1ff-4cfb-a12f-514137d3a4c1-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "65ec992a-d1ff-4cfb-a12f-514137d3a4c1" (UID: "65ec992a-d1ff-4cfb-a12f-514137d3a4c1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 19 15:44:48.062263 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:44:48.062210 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/65ec992a-d1ff-4cfb-a12f-514137d3a4c1-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "65ec992a-d1ff-4cfb-a12f-514137d3a4c1" (UID: "65ec992a-d1ff-4cfb-a12f-514137d3a4c1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 19 15:44:48.160501 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:44:48.160470 2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hkw9q\" (UniqueName: \"kubernetes.io/projected/65ec992a-d1ff-4cfb-a12f-514137d3a4c1-kube-api-access-hkw9q\") on node \"ip-10-0-138-220.ec2.internal\" DevicePath \"\"" Jul 19 15:44:48.160501 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:44:48.160497 2558 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/65ec992a-d1ff-4cfb-a12f-514137d3a4c1-istio-data\") on node \"ip-10-0-138-220.ec2.internal\" DevicePath \"\"" Jul 19 15:44:48.160685 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:44:48.160506 2558 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/65ec992a-d1ff-4cfb-a12f-514137d3a4c1-workload-socket\") on node \"ip-10-0-138-220.ec2.internal\" DevicePath \"\"" Jul 19 15:44:48.160685 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:44:48.160528 2558 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/65ec992a-d1ff-4cfb-a12f-514137d3a4c1-credential-socket\") on node \"ip-10-0-138-220.ec2.internal\" DevicePath \"\"" Jul 19 15:44:48.160685 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:44:48.160539 2558 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/65ec992a-d1ff-4cfb-a12f-514137d3a4c1-istio-token\") on node \"ip-10-0-138-220.ec2.internal\" DevicePath \"\"" Jul 19 15:44:48.160685 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:44:48.160548 2558 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/65ec992a-d1ff-4cfb-a12f-514137d3a4c1-istio-podinfo\") on node \"ip-10-0-138-220.ec2.internal\" DevicePath \"\"" Jul 19 15:44:48.160685 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:44:48.160556 2558 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/65ec992a-d1ff-4cfb-a12f-514137d3a4c1-istio-envoy\") on node \"ip-10-0-138-220.ec2.internal\" DevicePath \"\"" Jul 19 15:44:48.160685 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:44:48.160565 2558 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/65ec992a-d1ff-4cfb-a12f-514137d3a4c1-istiod-ca-cert\") on node \"ip-10-0-138-220.ec2.internal\" DevicePath \"\"" Jul 19 15:44:48.160685 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:44:48.160572 2558 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/65ec992a-d1ff-4cfb-a12f-514137d3a4c1-workload-certs\") on node \"ip-10-0-138-220.ec2.internal\" DevicePath \"\"" Jul 19 15:44:48.221387 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:44:48.221352 2558 generic.go:356] "Generic (PLEG): container finished" podID="65ec992a-d1ff-4cfb-a12f-514137d3a4c1" containerID="b4cdac0ff10e2b2cc995ef2a194a60f87b409d3724f0f56f57331432193ecab9" exitCode=0 Jul 19 15:44:48.221559 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:44:48.221410 2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-51b4ec-openshift-default-65f5bd8fc5-7cz2s" Jul 19 15:44:48.221559 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:44:48.221437 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-51b4ec-openshift-default-65f5bd8fc5-7cz2s" event={"ID":"65ec992a-d1ff-4cfb-a12f-514137d3a4c1","Type":"ContainerDied","Data":"b4cdac0ff10e2b2cc995ef2a194a60f87b409d3724f0f56f57331432193ecab9"} Jul 19 15:44:48.221559 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:44:48.221520 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-51b4ec-openshift-default-65f5bd8fc5-7cz2s" event={"ID":"65ec992a-d1ff-4cfb-a12f-514137d3a4c1","Type":"ContainerDied","Data":"b3c779142b3dd1d3578c2c09fb39acbf1fee283b7c8319ce5343cecdbad25807"} Jul 19 15:44:48.221559 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:44:48.221550 2558 scope.go:117] "RemoveContainer" containerID="b4cdac0ff10e2b2cc995ef2a194a60f87b409d3724f0f56f57331432193ecab9" Jul 19 15:44:48.230584 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:44:48.230560 2558 scope.go:117] "RemoveContainer" containerID="9f70ad46749bc3f82429d6abf68b10f88afcf79d4a677c332accfb7dc1f7251b" Jul 19 15:44:48.238441 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:44:48.238424 2558 scope.go:117] "RemoveContainer" containerID="b4cdac0ff10e2b2cc995ef2a194a60f87b409d3724f0f56f57331432193ecab9" Jul 19 15:44:48.238696 ip-10-0-138-220 kubenswrapper[2558]: E0719 15:44:48.238669 2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b4cdac0ff10e2b2cc995ef2a194a60f87b409d3724f0f56f57331432193ecab9\": container with ID starting with b4cdac0ff10e2b2cc995ef2a194a60f87b409d3724f0f56f57331432193ecab9 not found: ID does not exist" containerID="b4cdac0ff10e2b2cc995ef2a194a60f87b409d3724f0f56f57331432193ecab9" Jul 19 15:44:48.238775 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:44:48.238696 2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b4cdac0ff10e2b2cc995ef2a194a60f87b409d3724f0f56f57331432193ecab9"} err="failed to get container status \"b4cdac0ff10e2b2cc995ef2a194a60f87b409d3724f0f56f57331432193ecab9\": rpc error: code = NotFound desc = could not find container \"b4cdac0ff10e2b2cc995ef2a194a60f87b409d3724f0f56f57331432193ecab9\": container with ID starting with b4cdac0ff10e2b2cc995ef2a194a60f87b409d3724f0f56f57331432193ecab9 not found: ID does not exist" Jul 19 15:44:48.238775 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:44:48.238714 2558 scope.go:117] "RemoveContainer" containerID="9f70ad46749bc3f82429d6abf68b10f88afcf79d4a677c332accfb7dc1f7251b" Jul 19 15:44:48.238929 ip-10-0-138-220 kubenswrapper[2558]: E0719 15:44:48.238914 2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9f70ad46749bc3f82429d6abf68b10f88afcf79d4a677c332accfb7dc1f7251b\": container with ID starting with 9f70ad46749bc3f82429d6abf68b10f88afcf79d4a677c332accfb7dc1f7251b not found: ID does not exist" containerID="9f70ad46749bc3f82429d6abf68b10f88afcf79d4a677c332accfb7dc1f7251b" Jul 19 15:44:48.238968 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:44:48.238932 2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9f70ad46749bc3f82429d6abf68b10f88afcf79d4a677c332accfb7dc1f7251b"} err="failed to get container status \"9f70ad46749bc3f82429d6abf68b10f88afcf79d4a677c332accfb7dc1f7251b\": rpc error: code = NotFound desc = could not find container \"9f70ad46749bc3f82429d6abf68b10f88afcf79d4a677c332accfb7dc1f7251b\": container with ID starting with 9f70ad46749bc3f82429d6abf68b10f88afcf79d4a677c332accfb7dc1f7251b not found: ID does not exist" Jul 19 15:44:48.242518 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:44:48.242497 2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-51b4ec-openshift-default-65f5bd8fc5-7cz2s"] Jul 19 15:44:48.246103 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:44:48.246083 2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-51b4ec-openshift-default-65f5bd8fc5-7cz2s"] Jul 19 15:44:49.564928 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:44:49.564894 2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="65ec992a-d1ff-4cfb-a12f-514137d3a4c1" path="/var/lib/kubelet/pods/65ec992a-d1ff-4cfb-a12f-514137d3a4c1/volumes" Jul 19 15:44:53.389819 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:44:53.389786 2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-c17947-8cc7947f9-zkdqm"] Jul 19 15:44:53.390330 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:44:53.389995 2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-c17947-8cc7947f9-zkdqm" podUID="9644f172-e0ae-493b-b013-0a387feea206" containerName="maas-api" containerID="cri-o://6f319a685f828e4749d665d5506cf9e5e281d799b969b3d7e93692cf68f76b94" gracePeriod=30 Jul 19 15:44:53.519245 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:44:53.518864 2558 prober.go:120] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-c17947-8cc7947f9-zkdqm" podUID="9644f172-e0ae-493b-b013-0a387feea206" containerName="maas-api" probeResult="failure" output="Get \"https://10.133.0.48:8443/health\": dial tcp 10.133.0.48:8443: connect: connection refused" Jul 19 15:44:53.842306 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:44:53.842277 2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-c17947-8cc7947f9-zkdqm" Jul 19 15:44:53.908874 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:44:53.908840 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4fjjf\" (UniqueName: \"kubernetes.io/projected/9644f172-e0ae-493b-b013-0a387feea206-kube-api-access-4fjjf\") pod \"9644f172-e0ae-493b-b013-0a387feea206\" (UID: \"9644f172-e0ae-493b-b013-0a387feea206\") " Jul 19 15:44:53.909050 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:44:53.908926 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9644f172-e0ae-493b-b013-0a387feea206-maas-api-tls\") pod \"9644f172-e0ae-493b-b013-0a387feea206\" (UID: \"9644f172-e0ae-493b-b013-0a387feea206\") " Jul 19 15:44:53.909050 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:44:53.908958 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9644f172-e0ae-493b-b013-0a387feea206-openshift-service-ca\") pod \"9644f172-e0ae-493b-b013-0a387feea206\" (UID: \"9644f172-e0ae-493b-b013-0a387feea206\") " Jul 19 15:44:53.909359 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:44:53.909334 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9644f172-e0ae-493b-b013-0a387feea206-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "9644f172-e0ae-493b-b013-0a387feea206" (UID: "9644f172-e0ae-493b-b013-0a387feea206"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 19 15:44:53.911088 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:44:53.911062 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9644f172-e0ae-493b-b013-0a387feea206-kube-api-access-4fjjf" (OuterVolumeSpecName: "kube-api-access-4fjjf") pod "9644f172-e0ae-493b-b013-0a387feea206" (UID: "9644f172-e0ae-493b-b013-0a387feea206"). InnerVolumeSpecName "kube-api-access-4fjjf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 19 15:44:53.911194 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:44:53.911175 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9644f172-e0ae-493b-b013-0a387feea206-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "9644f172-e0ae-493b-b013-0a387feea206" (UID: "9644f172-e0ae-493b-b013-0a387feea206"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 19 15:44:54.010042 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:44:54.009989 2558 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9644f172-e0ae-493b-b013-0a387feea206-maas-api-tls\") on node \"ip-10-0-138-220.ec2.internal\" DevicePath \"\"" Jul 19 15:44:54.010042 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:44:54.010043 2558 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9644f172-e0ae-493b-b013-0a387feea206-openshift-service-ca\") on node \"ip-10-0-138-220.ec2.internal\" DevicePath \"\"" Jul 19 15:44:54.010222 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:44:54.010062 2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4fjjf\" (UniqueName: \"kubernetes.io/projected/9644f172-e0ae-493b-b013-0a387feea206-kube-api-access-4fjjf\") on node \"ip-10-0-138-220.ec2.internal\" DevicePath \"\"" Jul 19 15:44:54.247822 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:44:54.247739 2558 generic.go:356] "Generic (PLEG): container finished" podID="9644f172-e0ae-493b-b013-0a387feea206" containerID="6f319a685f828e4749d665d5506cf9e5e281d799b969b3d7e93692cf68f76b94" exitCode=0 Jul 19 15:44:54.247822 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:44:54.247803 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-c17947-8cc7947f9-zkdqm" event={"ID":"9644f172-e0ae-493b-b013-0a387feea206","Type":"ContainerDied","Data":"6f319a685f828e4749d665d5506cf9e5e281d799b969b3d7e93692cf68f76b94"} Jul 19 15:44:54.247822 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:44:54.247804 2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-c17947-8cc7947f9-zkdqm" Jul 19 15:44:54.248070 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:44:54.247868 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-c17947-8cc7947f9-zkdqm" event={"ID":"9644f172-e0ae-493b-b013-0a387feea206","Type":"ContainerDied","Data":"e8d3e67b7e405dbc687a5fca571b52cca111077e1d3f4069c16ac4fe7ee07ae9"} Jul 19 15:44:54.248070 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:44:54.247889 2558 scope.go:117] "RemoveContainer" containerID="6f319a685f828e4749d665d5506cf9e5e281d799b969b3d7e93692cf68f76b94" Jul 19 15:44:54.256729 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:44:54.256707 2558 scope.go:117] "RemoveContainer" containerID="6f319a685f828e4749d665d5506cf9e5e281d799b969b3d7e93692cf68f76b94" Jul 19 15:44:54.256976 ip-10-0-138-220 kubenswrapper[2558]: E0719 15:44:54.256957 2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6f319a685f828e4749d665d5506cf9e5e281d799b969b3d7e93692cf68f76b94\": container with ID starting with 6f319a685f828e4749d665d5506cf9e5e281d799b969b3d7e93692cf68f76b94 not found: ID does not exist" containerID="6f319a685f828e4749d665d5506cf9e5e281d799b969b3d7e93692cf68f76b94" Jul 19 15:44:54.257056 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:44:54.256985 2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6f319a685f828e4749d665d5506cf9e5e281d799b969b3d7e93692cf68f76b94"} err="failed to get container status \"6f319a685f828e4749d665d5506cf9e5e281d799b969b3d7e93692cf68f76b94\": rpc error: code = NotFound desc = could not find container \"6f319a685f828e4749d665d5506cf9e5e281d799b969b3d7e93692cf68f76b94\": container with ID starting with 6f319a685f828e4749d665d5506cf9e5e281d799b969b3d7e93692cf68f76b94 not found: ID does not exist" Jul 19 15:44:54.267314 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:44:54.267292 2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-c17947-8cc7947f9-zkdqm"] Jul 19 15:44:54.271182 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:44:54.271160 2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-c17947-8cc7947f9-zkdqm"] Jul 19 15:44:55.564590 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:44:55.564556 2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9644f172-e0ae-493b-b013-0a387feea206" path="/var/lib/kubelet/pods/9644f172-e0ae-493b-b013-0a387feea206/volumes" Jul 19 15:44:56.184431 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:44:56.184394 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-d4lll"] Jul 19 15:45:00.134527 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:45:00.134497 2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29741265-xltj4"] Jul 19 15:45:00.135182 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:45:00.134936 2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="65ec992a-d1ff-4cfb-a12f-514137d3a4c1" containerName="istio-proxy" Jul 19 15:45:00.135182 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:45:00.134956 2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="65ec992a-d1ff-4cfb-a12f-514137d3a4c1" containerName="istio-proxy" Jul 19 15:45:00.135182 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:45:00.134967 2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9644f172-e0ae-493b-b013-0a387feea206" containerName="maas-api" Jul 19 15:45:00.135182 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:45:00.134977 2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="9644f172-e0ae-493b-b013-0a387feea206" containerName="maas-api" Jul 19 15:45:00.135182 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:45:00.135003 2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="35de90fa-1778-452c-bc3c-46ef21c64fc0" containerName="maas-api" Jul 19 15:45:00.135182 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:45:00.135012 2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="35de90fa-1778-452c-bc3c-46ef21c64fc0" containerName="maas-api" Jul 19 15:45:00.135182 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:45:00.135040 2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="65ec992a-d1ff-4cfb-a12f-514137d3a4c1" containerName="istio-proxy" Jul 19 15:45:00.135182 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:45:00.135049 2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="65ec992a-d1ff-4cfb-a12f-514137d3a4c1" containerName="istio-proxy" Jul 19 15:45:00.135182 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:45:00.135124 2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="35de90fa-1778-452c-bc3c-46ef21c64fc0" containerName="maas-api" Jul 19 15:45:00.135182 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:45:00.135138 2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="9644f172-e0ae-493b-b013-0a387feea206" containerName="maas-api" Jul 19 15:45:00.135182 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:45:00.135149 2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="65ec992a-d1ff-4cfb-a12f-514137d3a4c1" containerName="istio-proxy" Jul 19 15:45:00.139336 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:45:00.139314 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29741265-xltj4" Jul 19 15:45:00.141405 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:45:00.141380 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qddbb\"" Jul 19 15:45:00.142123 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:45:00.142099 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29741265-xltj4"] Jul 19 15:45:00.151902 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:45:00.151881 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29741265-xltj4" Jul 19 15:45:00.281813 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:45:00.281769 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29741265-xltj4"] Jul 19 15:45:00.284157 ip-10-0-138-220 kubenswrapper[2558]: W0719 15:45:00.284129 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5a835529_23b9_4a16_b0bf_d3c287fbf0df.slice/crio-5157558ac610076d5613d047613ac755841c7196d2ee7f435c3746c99ebdc7e7 WatchSource:0}: Error finding container 5157558ac610076d5613d047613ac755841c7196d2ee7f435c3746c99ebdc7e7: Status 404 returned error can't find the container with id 5157558ac610076d5613d047613ac755841c7196d2ee7f435c3746c99ebdc7e7 Jul 19 15:45:01.279750 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:45:01.279714 2558 generic.go:356] "Generic (PLEG): container finished" podID="5a835529-23b9-4a16-b0bf-d3c287fbf0df" containerID="f3c4a1e34388b969d2a46ec0bf4fd982268408f856eb9705e3cc5d7cee22f506" exitCode=0 Jul 19 15:45:01.280150 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:45:01.279769 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29741265-xltj4" event={"ID":"5a835529-23b9-4a16-b0bf-d3c287fbf0df","Type":"ContainerDied","Data":"f3c4a1e34388b969d2a46ec0bf4fd982268408f856eb9705e3cc5d7cee22f506"} Jul 19 15:45:01.280150 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:45:01.279809 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29741265-xltj4" event={"ID":"5a835529-23b9-4a16-b0bf-d3c287fbf0df","Type":"ContainerStarted","Data":"5157558ac610076d5613d047613ac755841c7196d2ee7f435c3746c99ebdc7e7"} Jul 19 15:45:02.407012 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:45:02.406986 2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29741265-xltj4" Jul 19 15:45:03.288727 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:45:03.288696 2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29741265-xltj4" Jul 19 15:45:03.288905 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:45:03.288694 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29741265-xltj4" event={"ID":"5a835529-23b9-4a16-b0bf-d3c287fbf0df","Type":"ContainerDied","Data":"5157558ac610076d5613d047613ac755841c7196d2ee7f435c3746c99ebdc7e7"} Jul 19 15:45:03.288905 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:45:03.288819 2558 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5157558ac610076d5613d047613ac755841c7196d2ee7f435c3746c99ebdc7e7" Jul 19 15:45:04.687227 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:45:04.687194 2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-c17947-openshift-default-5748bf899-hxgq8"] Jul 19 15:45:04.687635 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:45:04.687404 2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-c17947-openshift-default-5748bf899-hxgq8" podUID="51bae637-42b6-4b76-89d7-5406a7b62fcd" containerName="istio-proxy" containerID="cri-o://c89370003d3ad9557990f64b47d042e8e664ba2ecfa3da93e5deaac195d30ed6" gracePeriod=30 Jul 19 15:45:09.937373 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:45:09.937342 2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-c17947-openshift-default-5748bf899-hxgq8" Jul 19 15:45:10.037151 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:45:10.037062 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/51bae637-42b6-4b76-89d7-5406a7b62fcd-credential-socket\") pod \"51bae637-42b6-4b76-89d7-5406a7b62fcd\" (UID: \"51bae637-42b6-4b76-89d7-5406a7b62fcd\") " Jul 19 15:45:10.037151 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:45:10.037099 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/51bae637-42b6-4b76-89d7-5406a7b62fcd-workload-certs\") pod \"51bae637-42b6-4b76-89d7-5406a7b62fcd\" (UID: \"51bae637-42b6-4b76-89d7-5406a7b62fcd\") " Jul 19 15:45:10.037151 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:45:10.037130 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/51bae637-42b6-4b76-89d7-5406a7b62fcd-workload-socket\") pod \"51bae637-42b6-4b76-89d7-5406a7b62fcd\" (UID: \"51bae637-42b6-4b76-89d7-5406a7b62fcd\") " Jul 19 15:45:10.037421 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:45:10.037155 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/51bae637-42b6-4b76-89d7-5406a7b62fcd-istio-data\") pod \"51bae637-42b6-4b76-89d7-5406a7b62fcd\" (UID: \"51bae637-42b6-4b76-89d7-5406a7b62fcd\") " Jul 19 15:45:10.037421 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:45:10.037211 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-68qw6\" (UniqueName: \"kubernetes.io/projected/51bae637-42b6-4b76-89d7-5406a7b62fcd-kube-api-access-68qw6\") pod \"51bae637-42b6-4b76-89d7-5406a7b62fcd\" (UID: \"51bae637-42b6-4b76-89d7-5406a7b62fcd\") " Jul 19 15:45:10.037421 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:45:10.037244 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/51bae637-42b6-4b76-89d7-5406a7b62fcd-istio-podinfo\") pod \"51bae637-42b6-4b76-89d7-5406a7b62fcd\" (UID: \"51bae637-42b6-4b76-89d7-5406a7b62fcd\") " Jul 19 15:45:10.037421 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:45:10.037274 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/51bae637-42b6-4b76-89d7-5406a7b62fcd-istio-envoy\") pod \"51bae637-42b6-4b76-89d7-5406a7b62fcd\" (UID: \"51bae637-42b6-4b76-89d7-5406a7b62fcd\") " Jul 19 15:45:10.037421 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:45:10.037298 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/51bae637-42b6-4b76-89d7-5406a7b62fcd-istiod-ca-cert\") pod \"51bae637-42b6-4b76-89d7-5406a7b62fcd\" (UID: \"51bae637-42b6-4b76-89d7-5406a7b62fcd\") " Jul 19 15:45:10.037421 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:45:10.037369 2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/51bae637-42b6-4b76-89d7-5406a7b62fcd-istio-token\") pod \"51bae637-42b6-4b76-89d7-5406a7b62fcd\" (UID: \"51bae637-42b6-4b76-89d7-5406a7b62fcd\") " Jul 19 15:45:10.037723 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:45:10.037398 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/51bae637-42b6-4b76-89d7-5406a7b62fcd-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "51bae637-42b6-4b76-89d7-5406a7b62fcd" (UID: "51bae637-42b6-4b76-89d7-5406a7b62fcd"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 19 15:45:10.037723 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:45:10.037422 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/51bae637-42b6-4b76-89d7-5406a7b62fcd-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "51bae637-42b6-4b76-89d7-5406a7b62fcd" (UID: "51bae637-42b6-4b76-89d7-5406a7b62fcd"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 19 15:45:10.037723 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:45:10.037439 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/51bae637-42b6-4b76-89d7-5406a7b62fcd-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "51bae637-42b6-4b76-89d7-5406a7b62fcd" (UID: "51bae637-42b6-4b76-89d7-5406a7b62fcd"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 19 15:45:10.037723 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:45:10.037659 2558 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/51bae637-42b6-4b76-89d7-5406a7b62fcd-credential-socket\") on node \"ip-10-0-138-220.ec2.internal\" DevicePath \"\"" Jul 19 15:45:10.037723 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:45:10.037677 2558 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/51bae637-42b6-4b76-89d7-5406a7b62fcd-workload-certs\") on node \"ip-10-0-138-220.ec2.internal\" DevicePath \"\"" Jul 19 15:45:10.037723 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:45:10.037700 2558 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/51bae637-42b6-4b76-89d7-5406a7b62fcd-workload-socket\") on node \"ip-10-0-138-220.ec2.internal\" DevicePath \"\"" Jul 19 15:45:10.037723 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:45:10.037673 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/51bae637-42b6-4b76-89d7-5406a7b62fcd-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "51bae637-42b6-4b76-89d7-5406a7b62fcd" (UID: "51bae637-42b6-4b76-89d7-5406a7b62fcd"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 19 15:45:10.038416 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:45:10.038241 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/51bae637-42b6-4b76-89d7-5406a7b62fcd-istio-data" (OuterVolumeSpecName: "istio-data") pod "51bae637-42b6-4b76-89d7-5406a7b62fcd" (UID: "51bae637-42b6-4b76-89d7-5406a7b62fcd"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 19 15:45:10.039722 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:45:10.039682 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/51bae637-42b6-4b76-89d7-5406a7b62fcd-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "51bae637-42b6-4b76-89d7-5406a7b62fcd" (UID: "51bae637-42b6-4b76-89d7-5406a7b62fcd"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 19 15:45:10.039722 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:45:10.039691 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/51bae637-42b6-4b76-89d7-5406a7b62fcd-istio-token" (OuterVolumeSpecName: "istio-token") pod "51bae637-42b6-4b76-89d7-5406a7b62fcd" (UID: "51bae637-42b6-4b76-89d7-5406a7b62fcd"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 19 15:45:10.039862 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:45:10.039830 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/51bae637-42b6-4b76-89d7-5406a7b62fcd-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "51bae637-42b6-4b76-89d7-5406a7b62fcd" (UID: "51bae637-42b6-4b76-89d7-5406a7b62fcd"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 19 15:45:10.039862 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:45:10.039835 2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/51bae637-42b6-4b76-89d7-5406a7b62fcd-kube-api-access-68qw6" (OuterVolumeSpecName: "kube-api-access-68qw6") pod "51bae637-42b6-4b76-89d7-5406a7b62fcd" (UID: "51bae637-42b6-4b76-89d7-5406a7b62fcd"). InnerVolumeSpecName "kube-api-access-68qw6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 19 15:45:10.138536 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:45:10.138507 2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-68qw6\" (UniqueName: \"kubernetes.io/projected/51bae637-42b6-4b76-89d7-5406a7b62fcd-kube-api-access-68qw6\") on node \"ip-10-0-138-220.ec2.internal\" DevicePath \"\"" Jul 19 15:45:10.138536 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:45:10.138532 2558 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/51bae637-42b6-4b76-89d7-5406a7b62fcd-istio-podinfo\") on node \"ip-10-0-138-220.ec2.internal\" DevicePath \"\"" Jul 19 15:45:10.138536 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:45:10.138543 2558 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/51bae637-42b6-4b76-89d7-5406a7b62fcd-istio-envoy\") on node \"ip-10-0-138-220.ec2.internal\" DevicePath \"\"" Jul 19 15:45:10.138784 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:45:10.138552 2558 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/51bae637-42b6-4b76-89d7-5406a7b62fcd-istiod-ca-cert\") on node \"ip-10-0-138-220.ec2.internal\" DevicePath \"\"" Jul 19 15:45:10.138784 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:45:10.138560 2558 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/51bae637-42b6-4b76-89d7-5406a7b62fcd-istio-token\") on node \"ip-10-0-138-220.ec2.internal\" DevicePath \"\"" Jul 19 15:45:10.138784 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:45:10.138567 2558 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/51bae637-42b6-4b76-89d7-5406a7b62fcd-istio-data\") on node \"ip-10-0-138-220.ec2.internal\" DevicePath \"\"" Jul 19 15:45:10.323918 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:45:10.323828 2558 generic.go:356] "Generic (PLEG): container finished" podID="51bae637-42b6-4b76-89d7-5406a7b62fcd" containerID="c89370003d3ad9557990f64b47d042e8e664ba2ecfa3da93e5deaac195d30ed6" exitCode=0 Jul 19 15:45:10.323918 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:45:10.323872 2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-c17947-openshift-default-5748bf899-hxgq8" Jul 19 15:45:10.323918 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:45:10.323905 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-c17947-openshift-default-5748bf899-hxgq8" event={"ID":"51bae637-42b6-4b76-89d7-5406a7b62fcd","Type":"ContainerDied","Data":"c89370003d3ad9557990f64b47d042e8e664ba2ecfa3da93e5deaac195d30ed6"} Jul 19 15:45:10.324219 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:45:10.323953 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-c17947-openshift-default-5748bf899-hxgq8" event={"ID":"51bae637-42b6-4b76-89d7-5406a7b62fcd","Type":"ContainerDied","Data":"63f812d5048061d020be557a3694699b7becd522423b0291d3e629f9991c726d"} Jul 19 15:45:10.324219 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:45:10.323976 2558 scope.go:117] "RemoveContainer" containerID="c89370003d3ad9557990f64b47d042e8e664ba2ecfa3da93e5deaac195d30ed6" Jul 19 15:45:10.333166 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:45:10.333145 2558 scope.go:117] "RemoveContainer" containerID="ae6937ca774a5b56758194155835eb038c1a9bec6e9dc05a84d2ec2918ae8aa2" Jul 19 15:45:10.340398 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:45:10.340382 2558 scope.go:117] "RemoveContainer" containerID="c89370003d3ad9557990f64b47d042e8e664ba2ecfa3da93e5deaac195d30ed6" Jul 19 15:45:10.340676 ip-10-0-138-220 kubenswrapper[2558]: E0719 15:45:10.340656 2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c89370003d3ad9557990f64b47d042e8e664ba2ecfa3da93e5deaac195d30ed6\": container with ID starting with c89370003d3ad9557990f64b47d042e8e664ba2ecfa3da93e5deaac195d30ed6 not found: ID does not exist" containerID="c89370003d3ad9557990f64b47d042e8e664ba2ecfa3da93e5deaac195d30ed6" Jul 19 15:45:10.340762 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:45:10.340681 2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c89370003d3ad9557990f64b47d042e8e664ba2ecfa3da93e5deaac195d30ed6"} err="failed to get container status \"c89370003d3ad9557990f64b47d042e8e664ba2ecfa3da93e5deaac195d30ed6\": rpc error: code = NotFound desc = could not find container \"c89370003d3ad9557990f64b47d042e8e664ba2ecfa3da93e5deaac195d30ed6\": container with ID starting with c89370003d3ad9557990f64b47d042e8e664ba2ecfa3da93e5deaac195d30ed6 not found: ID does not exist" Jul 19 15:45:10.340762 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:45:10.340699 2558 scope.go:117] "RemoveContainer" containerID="ae6937ca774a5b56758194155835eb038c1a9bec6e9dc05a84d2ec2918ae8aa2" Jul 19 15:45:10.340954 ip-10-0-138-220 kubenswrapper[2558]: E0719 15:45:10.340934 2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ae6937ca774a5b56758194155835eb038c1a9bec6e9dc05a84d2ec2918ae8aa2\": container with ID starting with ae6937ca774a5b56758194155835eb038c1a9bec6e9dc05a84d2ec2918ae8aa2 not found: ID does not exist" containerID="ae6937ca774a5b56758194155835eb038c1a9bec6e9dc05a84d2ec2918ae8aa2" Jul 19 15:45:10.341010 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:45:10.340965 2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ae6937ca774a5b56758194155835eb038c1a9bec6e9dc05a84d2ec2918ae8aa2"} err="failed to get container status \"ae6937ca774a5b56758194155835eb038c1a9bec6e9dc05a84d2ec2918ae8aa2\": rpc error: code = NotFound desc = could not find container \"ae6937ca774a5b56758194155835eb038c1a9bec6e9dc05a84d2ec2918ae8aa2\": container with ID starting with ae6937ca774a5b56758194155835eb038c1a9bec6e9dc05a84d2ec2918ae8aa2 not found: ID does not exist" Jul 19 15:45:10.344155 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:45:10.344135 2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-c17947-openshift-default-5748bf899-hxgq8"] Jul 19 15:45:10.347790 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:45:10.347771 2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-c17947-openshift-default-5748bf899-hxgq8"] Jul 19 15:45:10.933750 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:45:10.933721 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-6cb557cbc4-rqxpv_d50f665d-6f6e-436f-b1bc-94bd6aa10e6d/manager/0.log" Jul 19 15:45:11.416993 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:45:11.416972 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-74d6d868c8-sbwgw_080b867a-6665-459f-9a44-8b35cf3dafe0/copy-manifests/0.log" Jul 19 15:45:11.423803 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:45:11.423783 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-74d6d868c8-sbwgw_080b867a-6665-459f-9a44-8b35cf3dafe0/manager/0.log" Jul 19 15:45:11.564999 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:45:11.564968 2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="51bae637-42b6-4b76-89d7-5406a7b62fcd" path="/var/lib/kubelet/pods/51bae637-42b6-4b76-89d7-5406a7b62fcd/volumes" Jul 19 15:45:12.018267 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:45:12.018238 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29741235-h4dwb_e7d2d946-1495-4295-8b66-b64469d617bc/cleanup/0.log" Jul 19 15:45:12.141480 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:45:12.141453 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29741250-j875z_bce4499d-3ed2-4f05-8a56-a28653e37a2a/cleanup/0.log" Jul 19 15:45:12.252446 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:45:12.252422 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29741265-xltj4_5a835529-23b9-4a16-b0bf-d3c287fbf0df/cleanup/0.log" Jul 19 15:45:14.346914 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:45:14.346886 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-54d6d988d5-d4lll_8f25f7b4-20d5-483b-9d94-e01ed527dcb0/limitador/0.log" Jul 19 15:45:14.930465 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:45:14.930436 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-94bb8fbfd-jbnb7_2928718a-b22e-4315-b01a-4f2cabed02fa/discovery/0.log" Jul 19 15:45:24.664812 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:45:24.664775 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-kj72j_5994a848-5c3c-4d6c-898c-bb659e1a41b2/global-pull-secret-syncer/0.log" Jul 19 15:45:24.813716 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:45:24.813682 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-t8n4w_5f0a3171-e26a-47ef-b546-078b7c259d3e/konnectivity-agent/0.log" Jul 19 15:45:24.862385 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:45:24.862356 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-138-220.ec2.internal_e02b018e3a25db1c9e92391d89932378/haproxy/0.log" Jul 19 15:45:30.371767 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:45:30.371731 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-54d6d988d5-d4lll_8f25f7b4-20d5-483b-9d94-e01ed527dcb0/limitador/0.log" Jul 19 15:45:32.453733 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:45:32.453703 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-pwvsv_8281d5f9-51cc-46b4-a611-06ae3c073d41/node-exporter/0.log" Jul 19 15:45:32.476627 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:45:32.476605 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-pwvsv_8281d5f9-51cc-46b4-a611-06ae3c073d41/kube-rbac-proxy/0.log" Jul 19 15:45:32.494904 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:45:32.494882 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-pwvsv_8281d5f9-51cc-46b4-a611-06ae3c073d41/init-textfile/0.log" Jul 19 15:45:35.087692 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:45:35.087653 2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-h7n64/perf-node-gather-daemonset-c7vmf"] Jul 19 15:45:35.088094 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:45:35.087925 2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="51bae637-42b6-4b76-89d7-5406a7b62fcd" containerName="istio-proxy" Jul 19 15:45:35.088094 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:45:35.087937 2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="51bae637-42b6-4b76-89d7-5406a7b62fcd" containerName="istio-proxy" Jul 19 15:45:35.088094 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:45:35.087948 2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5a835529-23b9-4a16-b0bf-d3c287fbf0df" containerName="cleanup" Jul 19 15:45:35.088094 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:45:35.087954 2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="5a835529-23b9-4a16-b0bf-d3c287fbf0df" containerName="cleanup" Jul 19 15:45:35.088094 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:45:35.087997 2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="65ec992a-d1ff-4cfb-a12f-514137d3a4c1" containerName="istio-proxy" Jul 19 15:45:35.088094 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:45:35.088005 2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="5a835529-23b9-4a16-b0bf-d3c287fbf0df" containerName="cleanup" Jul 19 15:45:35.088094 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:45:35.088012 2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="51bae637-42b6-4b76-89d7-5406a7b62fcd" containerName="istio-proxy" Jul 19 15:45:35.088094 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:45:35.088035 2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="51bae637-42b6-4b76-89d7-5406a7b62fcd" containerName="istio-proxy" Jul 19 15:45:35.091169 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:45:35.091148 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-h7n64/perf-node-gather-daemonset-c7vmf" Jul 19 15:45:35.093410 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:45:35.093391 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-h7n64\"/\"kube-root-ca.crt\"" Jul 19 15:45:35.093410 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:45:35.093400 2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-h7n64\"/\"openshift-service-ca.crt\"" Jul 19 15:45:35.094007 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:45:35.093987 2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-h7n64\"/\"default-dockercfg-7hpkc\"" Jul 19 15:45:35.099852 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:45:35.099831 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-h7n64/perf-node-gather-daemonset-c7vmf"] Jul 19 15:45:35.242187 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:45:35.242153 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/964cbc93-c92c-4435-a1a6-1fd2918727b4-proc\") pod \"perf-node-gather-daemonset-c7vmf\" (UID: \"964cbc93-c92c-4435-a1a6-1fd2918727b4\") " pod="openshift-must-gather-h7n64/perf-node-gather-daemonset-c7vmf" Jul 19 15:45:35.242187 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:45:35.242188 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/964cbc93-c92c-4435-a1a6-1fd2918727b4-sys\") pod \"perf-node-gather-daemonset-c7vmf\" (UID: \"964cbc93-c92c-4435-a1a6-1fd2918727b4\") " pod="openshift-must-gather-h7n64/perf-node-gather-daemonset-c7vmf" Jul 19 15:45:35.242425 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:45:35.242263 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8zgl4\" (UniqueName: \"kubernetes.io/projected/964cbc93-c92c-4435-a1a6-1fd2918727b4-kube-api-access-8zgl4\") pod \"perf-node-gather-daemonset-c7vmf\" (UID: \"964cbc93-c92c-4435-a1a6-1fd2918727b4\") " pod="openshift-must-gather-h7n64/perf-node-gather-daemonset-c7vmf" Jul 19 15:45:35.242425 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:45:35.242306 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/964cbc93-c92c-4435-a1a6-1fd2918727b4-podres\") pod \"perf-node-gather-daemonset-c7vmf\" (UID: \"964cbc93-c92c-4435-a1a6-1fd2918727b4\") " pod="openshift-must-gather-h7n64/perf-node-gather-daemonset-c7vmf" Jul 19 15:45:35.242425 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:45:35.242345 2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/964cbc93-c92c-4435-a1a6-1fd2918727b4-lib-modules\") pod \"perf-node-gather-daemonset-c7vmf\" (UID: \"964cbc93-c92c-4435-a1a6-1fd2918727b4\") " pod="openshift-must-gather-h7n64/perf-node-gather-daemonset-c7vmf" Jul 19 15:45:35.342960 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:45:35.342860 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/964cbc93-c92c-4435-a1a6-1fd2918727b4-proc\") pod \"perf-node-gather-daemonset-c7vmf\" (UID: \"964cbc93-c92c-4435-a1a6-1fd2918727b4\") " pod="openshift-must-gather-h7n64/perf-node-gather-daemonset-c7vmf" Jul 19 15:45:35.342960 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:45:35.342899 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/964cbc93-c92c-4435-a1a6-1fd2918727b4-sys\") pod \"perf-node-gather-daemonset-c7vmf\" (UID: \"964cbc93-c92c-4435-a1a6-1fd2918727b4\") " pod="openshift-must-gather-h7n64/perf-node-gather-daemonset-c7vmf" Jul 19 15:45:35.342960 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:45:35.342952 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8zgl4\" (UniqueName: \"kubernetes.io/projected/964cbc93-c92c-4435-a1a6-1fd2918727b4-kube-api-access-8zgl4\") pod \"perf-node-gather-daemonset-c7vmf\" (UID: \"964cbc93-c92c-4435-a1a6-1fd2918727b4\") " pod="openshift-must-gather-h7n64/perf-node-gather-daemonset-c7vmf" Jul 19 15:45:35.343261 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:45:35.342981 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/964cbc93-c92c-4435-a1a6-1fd2918727b4-podres\") pod \"perf-node-gather-daemonset-c7vmf\" (UID: \"964cbc93-c92c-4435-a1a6-1fd2918727b4\") " pod="openshift-must-gather-h7n64/perf-node-gather-daemonset-c7vmf" Jul 19 15:45:35.343261 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:45:35.342996 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/964cbc93-c92c-4435-a1a6-1fd2918727b4-proc\") pod \"perf-node-gather-daemonset-c7vmf\" (UID: \"964cbc93-c92c-4435-a1a6-1fd2918727b4\") " pod="openshift-must-gather-h7n64/perf-node-gather-daemonset-c7vmf" Jul 19 15:45:35.343261 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:45:35.343050 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/964cbc93-c92c-4435-a1a6-1fd2918727b4-sys\") pod \"perf-node-gather-daemonset-c7vmf\" (UID: \"964cbc93-c92c-4435-a1a6-1fd2918727b4\") " pod="openshift-must-gather-h7n64/perf-node-gather-daemonset-c7vmf" Jul 19 15:45:35.343261 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:45:35.343052 2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/964cbc93-c92c-4435-a1a6-1fd2918727b4-lib-modules\") pod \"perf-node-gather-daemonset-c7vmf\" (UID: \"964cbc93-c92c-4435-a1a6-1fd2918727b4\") " pod="openshift-must-gather-h7n64/perf-node-gather-daemonset-c7vmf" Jul 19 15:45:35.343261 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:45:35.343109 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/964cbc93-c92c-4435-a1a6-1fd2918727b4-podres\") pod \"perf-node-gather-daemonset-c7vmf\" (UID: \"964cbc93-c92c-4435-a1a6-1fd2918727b4\") " pod="openshift-must-gather-h7n64/perf-node-gather-daemonset-c7vmf" Jul 19 15:45:35.343261 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:45:35.343145 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/964cbc93-c92c-4435-a1a6-1fd2918727b4-lib-modules\") pod \"perf-node-gather-daemonset-c7vmf\" (UID: \"964cbc93-c92c-4435-a1a6-1fd2918727b4\") " pod="openshift-must-gather-h7n64/perf-node-gather-daemonset-c7vmf" Jul 19 15:45:35.351559 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:45:35.351534 2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8zgl4\" (UniqueName: \"kubernetes.io/projected/964cbc93-c92c-4435-a1a6-1fd2918727b4-kube-api-access-8zgl4\") pod \"perf-node-gather-daemonset-c7vmf\" (UID: \"964cbc93-c92c-4435-a1a6-1fd2918727b4\") " pod="openshift-must-gather-h7n64/perf-node-gather-daemonset-c7vmf" Jul 19 15:45:35.401635 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:45:35.401605 2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-h7n64/perf-node-gather-daemonset-c7vmf" Jul 19 15:45:35.533944 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:45:35.533902 2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-h7n64/perf-node-gather-daemonset-c7vmf"] Jul 19 15:45:35.537952 ip-10-0-138-220 kubenswrapper[2558]: W0719 15:45:35.537917 2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod964cbc93_c92c_4435_a1a6_1fd2918727b4.slice/crio-8e7160a405854bff0128259168de548cd571927054e9575601a734808294a5a3 WatchSource:0}: Error finding container 8e7160a405854bff0128259168de548cd571927054e9575601a734808294a5a3: Status 404 returned error can't find the container with id 8e7160a405854bff0128259168de548cd571927054e9575601a734808294a5a3 Jul 19 15:45:36.437703 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:45:36.437666 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-h7n64/perf-node-gather-daemonset-c7vmf" event={"ID":"964cbc93-c92c-4435-a1a6-1fd2918727b4","Type":"ContainerStarted","Data":"0ddb021350148f305f11e3f4f0fc1a93e50c7ef2b0ab9ccf629d7f6052373ec2"} Jul 19 15:45:36.438152 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:45:36.437720 2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-h7n64/perf-node-gather-daemonset-c7vmf" event={"ID":"964cbc93-c92c-4435-a1a6-1fd2918727b4","Type":"ContainerStarted","Data":"8e7160a405854bff0128259168de548cd571927054e9575601a734808294a5a3"} Jul 19 15:45:36.438152 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:45:36.437783 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-h7n64/perf-node-gather-daemonset-c7vmf" Jul 19 15:45:36.452460 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:45:36.452380 2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-h7n64/perf-node-gather-daemonset-c7vmf" podStartSLOduration=1.45236134 podStartE2EDuration="1.45236134s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-19 15:45:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-19 15:45:36.45177636 +0000 UTC m=+2877.373479074" watchObservedRunningTime="2026-07-19 15:45:36.45236134 +0000 UTC m=+2877.374064054" Jul 19 15:45:37.313119 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:45:37.313092 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-kxhq6_021eeff4-26d3-4e17-890e-19e1f81c1173/dns/0.log" Jul 19 15:45:37.332356 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:45:37.332334 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-kxhq6_021eeff4-26d3-4e17-890e-19e1f81c1173/kube-rbac-proxy/0.log" Jul 19 15:45:37.438721 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:45:37.438688 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-qsm4z_8c1b2505-369a-4989-ac89-344f9d9991f9/dns-node-resolver/0.log" Jul 19 15:45:37.957671 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:45:37.957638 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-6f485ff8ff-jrz6w_1e4e44f7-d447-4d70-bf92-656f49c76bb0/registry/0.log" Jul 19 15:45:38.000656 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:45:38.000627 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-tnrrh_80a8f70e-f1a5-408d-bb95-b347a0ec8715/node-ca/0.log" Jul 19 15:45:39.032386 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:45:39.032357 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-94bb8fbfd-jbnb7_2928718a-b22e-4315-b01a-4f2cabed02fa/discovery/0.log" Jul 19 15:45:39.864368 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:45:39.864335 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-7prf4_47e880d5-eff9-4f7b-9c47-b7af7f74dc41/serve-healthcheck-canary/0.log" Jul 19 15:45:40.445054 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:45:40.445010 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-hp977_7fec0b2b-b831-414c-9a64-bf1b67c04fdd/kube-rbac-proxy/0.log" Jul 19 15:45:40.463603 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:45:40.463574 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-hp977_7fec0b2b-b831-414c-9a64-bf1b67c04fdd/exporter/0.log" Jul 19 15:45:40.482464 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:45:40.482441 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-hp977_7fec0b2b-b831-414c-9a64-bf1b67c04fdd/extractor/0.log" Jul 19 15:45:42.451120 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:45:42.451088 2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-h7n64/perf-node-gather-daemonset-c7vmf" Jul 19 15:45:42.881993 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:45:42.881917 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-6cb557cbc4-rqxpv_d50f665d-6f6e-436f-b1bc-94bd6aa10e6d/manager/0.log" Jul 19 15:45:43.098304 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:45:43.098274 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-74d6d868c8-sbwgw_080b867a-6665-459f-9a44-8b35cf3dafe0/manager/0.log" Jul 19 15:45:43.118789 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:45:43.118764 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-74d6d868c8-sbwgw_080b867a-6665-459f-9a44-8b35cf3dafe0/copy-manifests/0.log" Jul 19 15:45:44.549071 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:45:44.549045 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-57c99ccc67-wpkfm_031f67ac-e1a0-42d7-a44c-26e987378ffb/manager/0.log" Jul 19 15:45:44.575088 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:45:44.575065 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_openshift-lws-operator-fd8ccff4c-7gqmr_9fc231d4-04b0-4462-b869-067b8a369a4f/openshift-lws-operator/0.log" Jul 19 15:45:45.990302 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:45:45.990273 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-94bb8fbfd-jbnb7_2928718a-b22e-4315-b01a-4f2cabed02fa/discovery/0.log" Jul 19 15:45:47.470768 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:45:47.470735 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-94bb8fbfd-jbnb7_2928718a-b22e-4315-b01a-4f2cabed02fa/discovery/0.log" Jul 19 15:45:49.802430 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:45:49.802404 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-7cc9c7ffd7-2k6h8_f6b9fc46-d49c-4ef3-98f8-244658f46ea9/migrator/0.log" Jul 19 15:45:49.820767 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:45:49.820738 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-7cc9c7ffd7-2k6h8_f6b9fc46-d49c-4ef3-98f8-244658f46ea9/graceful-termination/0.log" Jul 19 15:45:51.355058 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:45:51.355009 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-vzds8_2571b663-0e56-4668-9831-fcdaf3793a1f/kube-multus-additional-cni-plugins/0.log" Jul 19 15:45:51.374117 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:45:51.374089 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-vzds8_2571b663-0e56-4668-9831-fcdaf3793a1f/egress-router-binary-copy/0.log" Jul 19 15:45:51.395013 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:45:51.394993 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-vzds8_2571b663-0e56-4668-9831-fcdaf3793a1f/cni-plugins/0.log" Jul 19 15:45:51.413865 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:45:51.413845 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-vzds8_2571b663-0e56-4668-9831-fcdaf3793a1f/bond-cni-plugin/0.log" Jul 19 15:45:51.432273 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:45:51.432251 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-vzds8_2571b663-0e56-4668-9831-fcdaf3793a1f/routeoverride-cni/0.log" Jul 19 15:45:51.452361 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:45:51.452343 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-vzds8_2571b663-0e56-4668-9831-fcdaf3793a1f/whereabouts-cni-bincopy/0.log" Jul 19 15:45:51.470912 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:45:51.470891 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-vzds8_2571b663-0e56-4668-9831-fcdaf3793a1f/whereabouts-cni/0.log" Jul 19 15:45:51.533711 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:45:51.533689 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-jhf4r_5a78ce34-491f-4ae5-971b-65d650736dff/kube-multus/0.log" Jul 19 15:45:51.552139 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:45:51.552120 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-g52v5_c9a1c57d-99f6-49a1-bd75-d00fcffc86f1/network-metrics-daemon/0.log" Jul 19 15:45:51.570001 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:45:51.569982 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-g52v5_c9a1c57d-99f6-49a1-bd75-d00fcffc86f1/kube-rbac-proxy/0.log" Jul 19 15:45:52.446656 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:45:52.446620 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-5wfzs_5b2d3a07-86b2-4ba2-8a71-c2d89ada79c6/ovn-controller/0.log" Jul 19 15:45:52.462373 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:45:52.462348 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-5wfzs_5b2d3a07-86b2-4ba2-8a71-c2d89ada79c6/ovn-acl-logging/0.log" Jul 19 15:45:52.462684 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:45:52.462670 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-5wfzs_5b2d3a07-86b2-4ba2-8a71-c2d89ada79c6/ovn-acl-logging/1.log" Jul 19 15:45:52.478862 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:45:52.478841 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-5wfzs_5b2d3a07-86b2-4ba2-8a71-c2d89ada79c6/kube-rbac-proxy-node/0.log" Jul 19 15:45:52.498481 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:45:52.498462 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-5wfzs_5b2d3a07-86b2-4ba2-8a71-c2d89ada79c6/kube-rbac-proxy-ovn-metrics/0.log" Jul 19 15:45:52.517264 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:45:52.517245 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-5wfzs_5b2d3a07-86b2-4ba2-8a71-c2d89ada79c6/northd/0.log" Jul 19 15:45:52.536786 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:45:52.536768 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-5wfzs_5b2d3a07-86b2-4ba2-8a71-c2d89ada79c6/nbdb/0.log" Jul 19 15:45:52.555058 ip-10-0-138-220 kubenswrapper[2558]: I0719 15:45:52.555042 2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-5wfzs_5b2d3a07-86b2-4ba2-8a71-c2d89ada79c6/sbdb/0.log"